All writing
War story · Cluster

Won the eval, lost the cluster

A newer 284B model beat our production 262B on 13 of 13 quality probes. We spent two weeks fitting it. One honest benchmark retired it anyway.

01 Sep 26 · 4 min · by Stephen Robinson

In late July 2026, a new model beat our production 262B mixture-of-experts on every quality probe we threw at it. We spent two weeks engineering the fit. It met our context bar. It decoded faster than the incumbent. Then we ran one honest benchmark and retired it.

The deciding constraint was not the model’s quality. It was the fleet’s scarcest resource.

The challenger earns the work

The side-by-side eval was not close. The challenger answered 13 of 13 structured probes correctly. The incumbent scored 9 of 13.

We logged one caveat honestly at the time. At very deep context, around 84K tokens, a quantized build of the challenger corrupted exact literals when asked to reproduce them verbatim. We flagged it. We did not fully attribute it then. It may have been a sampler artifact. The follow-up was still open when the story ended.

The engineering to fit the model succeeded. The bar was 256K context per slot. We met it at 4.48 bits per weight.

Along the way, we hunted a crash-on-load. It ended when we found a memory-mapping boundary landing mid-layer. The fix was halving the prefill microbatch size. Three data points from that hunt: a 1698 MiB spill to host memory loaded fine. 3738 MiB also loaded fine. 4428 MiB faulted. The spill size is a risk indicator, not a verdict. The boundary has to land inside a layer to kill you.

With a small draft model attached, decode hit 31.1 to 31.6 tokens per second at 0.825 draft acceptance on its best workload class. Faster than production. On prose workloads the same setup ran 13.5 to 26.6 at 0.52 to 0.72 acceptance. Any throughput claim has to name its workload class.

The benchmark that ended it

Prefill had never been measured properly. The earlier bench used a ~20-token prompt. That measures fixed request overhead, not prefill.

We ran a real harness. Cache disabled. Per-run nonce so nothing was a cached prefix. Three prompt sizes from ~2.7K to ~14K tokens. Timings read from the server’s own timing object.

The incumbent ran ~305 tokens per second prefill, flat across prompt length. We measured it live on production with zero downtime.

The challenger ran ~160 tokens per second. About half.

The decisive detail was batch-invariance. The challenger’s prefill was identical at microbatch 256 and 448, ~160 both. The incumbent gains 34% from a bigger microbatch, moving from 168 to 226 in its own ladder. A workload that stops improving with bigger batches is already saturating its GPUs. The challenger’s prefill reads as compute-bound on this hardware. That is an inference from the batch-invariance, not a measured utilization figure.

Why the architecture taxed the wrong resource

The challenger spends extra compute per token. It scores and selects over its compressed attention cache to buy a tiny attention-cache footprint. We measured the cache at 32 to 64 MiB per device.

Our cluster was never cache-constrained. The architecture taxes the resource this fleet is poorest in, compute spread across 15 modest cards, to optimize one it had headroom on.

The cluster served a 262B model across a 15-GPU cluster: five modest boxes of consumer cards on a 40G RDMA fabric, one coordinator. The challenger was a 284B next-generation model. The incumbent stayed in production. We retired the challenger on throughput, not quality.

Honest confound

The challenger was benched with its draft model attached. The incumbent has none. The draft must prefill the prompt too, so some of the gap is draft overhead. It is a 3-layer draft against a 43-layer target, so this should be a small share, not 2x. But it was never isolated. If the succession is revisited, the no-draft prefill run goes first.

The lessons

Measure prefill with a real prompt. A tiny-prompt “prefill” number is request overhead wearing a costume.

Batch-invariance is a free diagnostic. If throughput does not move when the microbatch does, you are compute-bound, and no config knob will save you.

Judge an architecture against your fleet’s scarcest resource. The same model on a few fat GPUs might scale beautifully. On many small ones the compute tax came due. “Better model” is not a portable fact.

More from the journal

Everything here happened on the hardware this site is about. If this one was useful, the rest of the ledger is one click away.