A 212B on the same five boxes: the active-parameter rule, tested

Where we left off

The last post in this thread ended with a rule: on a multi-node cluster, design for active-parameter count, not total size. A mixture-of-experts model keeps the per-token wire cost small, so it stays fast when you split it across machines. A dense model of similar strength craters on the same links.

A rule you wrote after one investigation is a hypothesis. A rule that predicts the next deployment’s numbers before you run it is something better. This month we got to test ours.

The upgrade that shouldn’t fit

Our deep tier ran a 122B MoE with roughly 10B active parameters per token. It lived across five Linux machines on 10-gigabit Ethernet, about 110GB of pooled VRAM, and the hard constraint that shapes everything we do: the smallest cards in that pool are 8GB. Any layer that cannot seat under 8GB does not deploy, no matter what the aggregate says.

The replacement is a 212B with 17B active per token. Nobody serves a model in that weight class on repurposed gaming hardware by loading it straight. This one exists because of expert pruning. The base model is a REAP of a 397B: about half the experts are removed outright, keeping the ones that actually fire. We took that 212B base, ran our own abliteration pass, and then built a custom quant for it the same way we built the last one — attention layers held at high precision, everything else mixed per-layer so that every single layer seats under the 8GB ceiling. The result is an 81GB file that spreads across the same five boxes that served the 122B. No new GPUs. No new nodes.

That is the part worth sitting with. The route to a 212B-class model at home was not more hardware. It was pruning at the expert level, then quantizing with the smallest card in mind.

The prediction

Here is what the two ways of thinking predict for decode speed.

Total-size thinking says we nearly doubled the model, 122B to 212B, so throughput should roughly halve, and the extra sync traffic across five nodes should make it worse than that. Call it disaster.

Active-parameter thinking says total size is mostly irrelevant to the wire and the compute path. What changed per token is 10B active to 17B active, a factor of about 1.7. The 122B decoded at 27.8 tokens per second, so the rule predicts around 16.

We measured the 212B at 18.6 tokens per second, single stream, on the production cluster.

The active-parameter prediction landed within about 15 percent, on the optimistic side — decode slowed by a factor of 1.5, a bit less than the 1.7 the active ratio implied. The total-size prediction was not close. The rule held, and it held in the direction that matters: it told us before we committed the download that the model would be usable, not a science project.

What total size buys

Active parameters set the speed. Total parameters still buy things, and the 212B makes the trade visible.

It is a deeper model in every way that matters for the hard queries we route to the cluster tier. And the deployment carries a million tokens of live context — four slots of 256K each — which is the difference between “summarize this file” and “hold the whole subsystem in your head while we talk about it.” That is what we spent the speed on. A third fewer tokens per second, in exchange for a much bigger brain with a much longer memory, on hardware that did not change.

That is the honest shape of the rule. It is not “bigger MoE is free.” It is: the cost of bigger scales with the active count, so you can budget it like an engineer instead of finding out like a victim.

The slowdown still drew blood

Here is the war story part, because there is always one.

A 33 percent decode slowdown does not show up in benchmarks as a failure. Everything works. Every request completes. The graphs look fine. Where it shows up is in every downstream consumer that quietly encoded the old model’s speed into a pair of config values.

Our PR-review bot runs every pull request in the org through a local model. Its heavy tier gets a 4,000-token output budget and had a fixed 180-second call timeout. Nobody ever paired those numbers on purpose. At the 122B’s 27.8 tokens per second, 4,000 tokens took about 144 seconds, and the pairing worked, invisibly, for weeks. At 18.6 tokens per second, the same budget needs about 215 seconds. Every maximum-length heavy review was now guaranteed to time out, and the bot would have reported “review couldn’t run” while the cluster was healthy and the model was mid-sentence.

We caught it doing arithmetic during the documentation pass, before it fired once in production. The fix was to stop pretending the timeout is a constant: it now scales from the token budget, so the pairing survives the next model swap too.

The lesson generalizes past this bot. A model cutover changes decode rate, and decode rate is an implicit contract with every timeout, cron watchdog, and output cap downstream of the endpoint. None of those consumers error at deploy time. They fail later, at full output length, on the request that mattered. When you swap the model, sweep the cap-and-timeout pairs and redo the arithmetic against the measured speed of the new brain.

The rule, updated

On a multi-node cluster, judge a model by its active-parameter count. Total size is what you get; active size is what you pay, and it prices both the wire and the clock. Expert pruning plus a fit-aware quant is how a 397B-descended model ends up living on five machines full of 8GB gaming cards.

And when the new brain is slower than the old one, the tax collector does not knock on your benchmarks. It knocks on your timeouts.