When failure is good news
“We tried to break it—and failed.” That was the verdict after CERN’s IT teams unleashed a marathon stress test on the software that orchestrates the lab’s compute workload, bombarding it with an extreme volume of tasks to see where it would crack. It didn’t. While the run wasn’t in full production conditions, the outcome is an early, encouraging sign as CERN prepares for the era of the High-Luminosity Large Hadron Collider (HL‑LHC).
Why push so hard now?
The HL‑LHC, slated to start up in 2030, will deliver far more proton–proton collisions than the current LHC. Annual luminosity is expected to leap from roughly 125 inverse femtobarns per year to 300 or more. With 1 inverse femtobarn corresponding to about 100 million million potential collisions, that jump translates into an enormous surge in data that must be queued, scheduled, executed and stored—reliably and quickly.
The October trial: what was tested
The focus was the software layer that manages CERN’s batch computing workload—the service that takes job requests from physicists, monitors available resources and dispatches tasks to the right machines across a global pool. The goal: probe the limits of the control plane under intense pressure and identify bottlenecks before HL‑LHC scale becomes a daily reality.
Results by the numbers
- Duration: 13 hours of continuous stress.
- Throughput: about 16 800 jobs injected per minute—around 20 times today’s average.
- Volume: more than two million jobs successfully executed during the run.
- Responsiveness: average job handling time remained around 5 minutes, even at this scale.
Crucially, the system held steady: no systemic failures, no runaway latencies, and no collapse of the scheduling layer. For an early-stage exercise, that stability is noteworthy. As one test lead put it, it’s a strong opening milestone on the road to HL‑LHC readiness, not a finish line.
Under the hood: HTCondor at CERN
CERN’s workload management relies on HTCondor, the open‑source high-throughput computing platform developed at the University of Wisconsin–Madison. CERN adopted HTCondor for batch processing in 2016 and has since worked closely with its developers to scale the technology for high‑energy physics demands.
Two core HTCondor services did the heavy lifting in this test:
- Collector daemon: gathers information on jobs and available resources across the pool.
- Negotiator daemon: matches queued jobs to suitable machines according to policy, priority and resource constraints.
Together, these components provide the essentials of a production scheduler: job queueing, policy-driven scheduling, prioritization, resource monitoring and resource management. The stress test validated that, even at extreme injection rates and volumes, this control plane could maintain situational awareness of the pool and keep jobs flowing.
Context matters: a test, not production
While the outcome was robust, the team is clear-eyed about what comes next. The run did not mirror all the complexities of live physics workloads, which can feature diverse job profiles, fluctuating data dependencies and real-time operational constraints. Future rounds will dial up realism—introducing mixed job types, varied input sizes and dynamic resource conditions—to uncover edge cases that a synthetic load might miss.
What’s next on the road to HL‑LHC
Compute is only half the story; storage and data movement are the other half. Upcoming tests will bring CERN’s disk-based storage systems into the loop, exercising end-to-end pipelines from job submission through data staging, execution and archival. The objective is to validate not just raw throughput but the balance between CPU, memory, network and storage that keeps the whole ecosystem performant and predictable.
As the HL‑LHC era approaches, CERN’s strategy is methodical: test early, scale iteratively and partner closely with the open-source community and experiment teams. The latest stress test shows that the scheduling backbone can take a punch. In this case, failure would have been a warning. Not failing is good news—and a green light to push even harder in the next round.
