13 Sep 26
War story · ResearchThe label said abliterated. It was not.
Every mixture-of-experts model we had published as abliterated still refused. The bug was not the architecture and not the code. It was a default we never questioned. Corrected 23 September: the fix did not hold on the 262B.
6 min
13 Sep 26
War story · OpsThe ceiling that was a flag
A 67.5 GB model loaded in 43 seconds. The next file up, 71.7 GB, took a whole server dark for fifteen minutes. We wrote a confident post-mortem saying we needed a bigger box. We were wrong, and the same box later loaded a file 1.41 times its total memory.
6 min
01 Sep 26
War story · OpsThe ladder beside a live brain
We quantized a 354GB model into seven sizes on the box that runs production inference. Production never noticed. The storage layer had opinions, and the wall clock sent a bill.
5 min
01 Sep 26
War story · ClusterTwelve rungs, eleven minutes
An evening of model quantization became eleven minutes the first time a real scheduler ran the ladder. Four of those jobs ran on the production coordinator. Production dipped 4 percent and came back exact.
4 min
01 Sep 26
War story · ClusterWon the eval, lost the cluster
A newer 284B model beat our production 262B on 13 of 13 quality probes. We spent two weeks fitting it. One honest benchmark retired it anyway.
4 min
05 Aug 26
EngineeringBandwidth paid prefill, RDMA paid decode
The cluster got a 40-gigabit RDMA fabric built from secondhand cards, and the winnings split clean down the middle: bandwidth moved one number, RDMA moved the other, and two of the five links faked the upgrade until a probe patch called them on it.
5 min
24 Jul 26
EngineeringAn organ transplant for the agent model
Our 35B agent model shipped without the speculative-decode head its base model carries. A stranger's repo proved the graft could work. The better donor was in our own archive the whole time.
4 min
24 Jul 26
War story · ResearchThe pool that decoded at 0.9 tokens per second
An 88B pooled across two new boxes crawled at 0.90 tokens per second, and every plausible suspect got blamed first. One profiler run named the real killer in six tokens.
4 min
24 Jul 26
War story · OpsThe slot that ate the card
A GPU kept dying: firmware hangs, dark boxes, one 3.5 hour outage. Every fix chased the card. A swap experiment proved all four cards innocent and named the motherboard slot.
5 min
11 Jul 26
War story · LabA server named Adhara
We lost our dog on a Friday afternoon. That night I rebuilt half the lab with my hands, fought a Dell R730 through five install attempts, and named the new machine after the brightest star in a light you can't see.
6 min
10 Jul 26
War story · ResearchA 212B on the same five boxes: the active-parameter rule, tested
We retired the 122B and put a REAP-pruned 212B on the identical cluster without adding a GPU. The rule from last time predicted the decode number almost exactly. The slowdown still found something to break.
7 min
04 Jul 26
War story · ResearchThe coordinator didn't need our biggest CPU
We placed our cluster coordinator for core count and waited for tokenization to become the bottleneck. Then we measured it: 20 percent of one core. The speedup was hiding in the network leg.
5 min
03 Jul 26
War story · ResearchWhy a 122B MoE beat dense on the slow interconnect
Before the 10-gig upgrade, dense models were a trap: the sync tax scales with hidden_dim. The numbers, and the architecture rule we still design around.
6 min
03 Jul 26
War story · OpsThe TX cliff that wasn't the NIC
One host's throughput cratered on every large model. We swapped NICs, cables, drivers, and reproduced it on all of them. The defect was somewhere we weren't looking.
4 min
21 Jun 26
War story · Opsapt history first: when "nothing changed" but everything broke
A validated cluster node started crashing on a workload it had served cleanly for weeks. Three days down the wrong rabbit hole (mmap theory, KV pressure, tensor-split) before checking the one log that actually mattered.
6 min