We replaced ad-hoc evening quantization runs with a real batch scheduler across eleven nodes. The first full-speed run compressed an evening of work into eleven minutes. It ran four jobs directly on the box that coordinates production inference. Production barely blinked.
Before late August 2026, we quantized models into “ladders,” which are one master model cut into many output sizes. We ran them ad-hoc, one job at a time, hand-supervised. A six-rung ladder took a full evening. We had a standard open-source scheduler, Slurm, across eleven compute nodes in four partitions. We added hard per-job memory enforcement via cgroup v2. This meant a batch job physically could not eat a serving box.
One partition included Adhara, a big-memory server that also coordinates the production 262B model at 29 to 30 tokens per second decode. Scheduling batch work on Adhara, beside live production, was the whole test.
I submitted a twelve-rung ladder of a small 9B model as one batch. At T0 plus 11 seconds, eight rungs were already running. Four of them ran concurrently on Adhara beside the live coordinator. The queue drained in about 7.5 minutes. With one redone probe rung, the complete twelve-rung ladder took about 11 minutes wall clock. This was against a prior full evening for six rungs.
Production during the crunch held up. The A/B/A decode readings were 29.4 to 29.8 baseline, 28.3 to 28.5 with four rungs running on the coordinator, and 29.4 to 29.8 after. It returned exactly to baseline. That is a 4 percent dip.
Serving health through the window was clean. We had 357 probes across 7 endpoints with zero failures. The worst response was 83 milliseconds. All 12 rungs were archived with per-rung checksums.
The memory floor was a myth. The old sizing table said a quantize job’s memory tracks its output size. Hard per-job limits disproved it in one night. An 18GB master needs about 9GB regardless of which rung it is cutting. A probe capped at 8GB crunched happily for 5.5 minutes and then died in the write phase. A 9GB cap cruised. Every earlier “elastic” memory reading was soft-throttle behavior. The job slowed under pressure. It was not actual appetite. A hard limit is an instrument. It measures the true floor by refusing to lie.
A later, larger master showed the floor is also sub-linear in master size. Roughly master over ten at the 350GB class. The rule became: probe each new size class, never extrapolate the constant.
The environment-passing trap was the other finding. On this scheduler version, mixing “export everything” with a single added variable on the submission line trips a launch-time environment fetch that fails and requeues the job. It does this forever. It creates a requeue storm with no work done. I set variables in the submission shell instead and let the default export carry them. The cost was one confusing half hour. It is worth a named rule.
We needed a scheduler because the old way consumed evenings and attention. Scripts do not queue. They do not provide per-job accounting. CPU-hours per artifact now land in the lab’s provenance records. The scheduler adds enforcement. The memory caps protect production and measure the truth.
The dip has a fix already scoped. An arm of the lab’s inference gate will pause batch work during interactive inference. This should erase even the 4 percent.
Enforcement is measurement. Hard cgroup limits found the real memory floor on night one. Months of soft limits never did.
Coexistence with production is a designed property, not luck. We used hard caps, honest memory requests, and an A/B/A decode check before, during, and after.
When a batch system misbehaves at launch, suspect the environment hand-off before the job. The requeue storm contained zero information about the actual work.