We needed a full quantization ladder for a new 354GB master model. The job was to produce seven output sizes, ranging from a 188GB Q8 down to a 75GB Q2, totaling about 824GB of artifacts. The constraint was simple: we would run these jobs on Adhara, the big-memory server that coordinates our production 262B model. Production was running decode at 29 to 30 tokens per second, and we needed to ensure it stayed there.
I had fifteen years of scars from homelab experiments where “background” tasks quietly starve the main service. I did not want to repeat that mistake. So we scheduled the quantization jobs under our new scheduler with hard per-job memory limits, running directly beside the live coordinator process. We treated the run as an A/B test. If production latency spiked, we would kill the jobs.
The plan was sound. The execution was not.
The run and the verdict
The quantization ladder ran over a seven-hour window in late August 2026. Each rung took between 32 and 72 minutes. We monitored the production decode rate continuously.
Before the run, production decode sat at 29.4 to 29.8 tokens per second. While four quantization jobs crunched on the same box, the rate held at 29.3 to 29.6. After the run, it returned to 29.4 to 29.8.
Production never noticed. The load was manageable, and the hard memory caps kept the jobs from spilling into shared memory. We had achieved the primary goal: a clean quantization ladder with zero impact on the live service.
The storage trap
Then the middle of the run collapsed.
A rung died at 48% written. Two more jobs failed instantly behind it. The error was “disk full.”
I checked the storage server, the NAS. The working share was supposed to be on the big 24T array. But the platform uses user-share storage virtualization. The working share was actually riding a dedicated 1.5T pool. When the jobs finished writing, the script moved the artifacts “to the archive share” to free up space for the next rung.
On this platform, a share-path move does not guarantee a physical move. The share-level allocator picks the destination device. It picked the source pool. The files were copied back onto the same 1.5T pool. The move freed nothing. The pool filled up, and the jobs choked.
We were stuck. The jobs were dead, the pool was full, and we had no space to restart them.
I looked around the NAS mount points. There was a 1.6TB SSD pool intended for exactly this work. It was mounted and sitting empty.
We moved all 575GB of finished artifacts from the working share to the SSD pool. The move took 13 minutes and 13 seconds. The working pool went from 117GB free to 560GB free. We resubmitted the dead rungs. They completed.
The lesson was immediate and painful. On share-virtualizing NAS platforms, “moved to the archive” and “freed the pool” are different claims. A share-path move can silently reallocate in place. When the point of a move is to free space, use an explicit device path or a destination in another pool.
Two honest surprises
The run revealed two things I had not expected.
First, the memory usage. The old rule of thumb says a quantize job’s memory scales with the master, roughly half. Under soft limits, we had seen elastic behavior that suggested high appetite. But under the hard per-job caps, every rung peaked at 32 to 35GB. That is about master divided by ten.
The hard limit told the truth. The soft limits had hidden the actual floor. And the floor is not a constant. An 18GB master needs about 9GB. The ratio moves with size class. Probe each new size class; do not extrapolate a single constant from one run.
Second, the wall clock. Running politely beside production cost about 5x the elapsed time a claimed exclusive window would have. The tax was not compute. It was 30 to 40% of CPU used for write back-pressure, pushing output over a 1-gigabit network path to the NAS.
Coexistence worked. It was the right call for a first run beside production. But it is now measured. The next big master gets a burst window and a faster storage path.
Trust, but verify the archive
Producer checksums were taken at creation. But the files moved pool-to-pool during recovery. A recorded digest only proves the path it was taken on.
I took a later pass and re-read every archived copy. Seven of seven rungs matched byte-for-byte. The master matched too. Eight of eight artifacts, about 1,238GB, were witnessed on the copies a future build would actually open.
A checksum verifies a copy, not a file. After any move, the digest that matters is the one taken where the file now lives.
We quantized a 354GB model into seven sizes on the box that runs production inference. Production never noticed. The storage layer had opinions, and the wall clock sent a bill. We paid the bill, and we kept the ladder.