Writing

What we ship, we write down.

War stories and research notes from building a local-first AI cluster: the dead ends as much as the wins.

16 dispatches · engineering journal
War story · Ops

The cable was fine

A new server's 40-gigabit link fell back to 10 after every boot. We blamed the cable, then autonegotiation, then firmware. The answer was a BIOS setting, and the test that found it was free and took two seconds.

20 Sep 26 · 6 min · by Stephen Robinson
Latest
dispatch
13 Sep 26
War story · Research

The label said abliterated. It was not.

Every mixture-of-experts model we had published as abliterated still refused. The bug was not the architecture and not the code. It was a default we never questioned. Corrected 23 September: the fix did not hold on the 262B.
6 min
13 Sep 26
War story · Ops

The ceiling that was a flag

A 67.5 GB model loaded in 43 seconds. The next file up, 71.7 GB, took a whole server dark for fifteen minutes. We wrote a confident post-mortem saying we needed a bigger box. We were wrong, and the same box later loaded a file 1.41 times its total memory.
6 min
01 Sep 26
War story · Ops

The ladder beside a live brain

We quantized a 354GB model into seven sizes on the box that runs production inference. Production never noticed. The storage layer had opinions, and the wall clock sent a bill.
5 min
01 Sep 26
War story · Cluster

Twelve rungs, eleven minutes

An evening of model quantization became eleven minutes the first time a real scheduler ran the ladder. Four of those jobs ran on the production coordinator. Production dipped 4 percent and came back exact.
4 min
01 Sep 26
War story · Cluster

Won the eval, lost the cluster

A newer 284B model beat our production 262B on 13 of 13 quality probes. We spent two weeks fitting it. One honest benchmark retired it anyway.
4 min
05 Aug 26
Engineering

Bandwidth paid prefill, RDMA paid decode

The cluster got a 40-gigabit RDMA fabric built from secondhand cards, and the winnings split clean down the middle: bandwidth moved one number, RDMA moved the other, and two of the five links faked the upgrade until a probe patch called them on it.
5 min
24 Jul 26
Engineering

An organ transplant for the agent model

Our 35B agent model shipped without the speculative-decode head its base model carries. A stranger's repo proved the graft could work. The better donor was in our own archive the whole time.
4 min
24 Jul 26
War story · Research

The pool that decoded at 0.9 tokens per second

An 88B pooled across two new boxes crawled at 0.90 tokens per second, and every plausible suspect got blamed first. One profiler run named the real killer in six tokens.
4 min
24 Jul 26
War story · Ops

The slot that ate the card

A GPU kept dying: firmware hangs, dark boxes, one 3.5 hour outage. Every fix chased the card. A swap experiment proved all four cards innocent and named the motherboard slot.
5 min
11 Jul 26
War story · Lab

A server named Adhara

We lost our dog on a Friday afternoon. That night I rebuilt half the lab with my hands, fought a Dell R730 through five install attempts, and named the new machine after the brightest star in a light you can't see.
6 min
10 Jul 26
War story · Research

A 212B on the same five boxes: the active-parameter rule, tested

We retired the 122B and put a REAP-pruned 212B on the identical cluster without adding a GPU. The rule from last time predicted the decode number almost exactly. The slowdown still found something to break.
7 min
04 Jul 26
War story · Research

The coordinator didn't need our biggest CPU

We placed our cluster coordinator for core count and waited for tokenization to become the bottleneck. Then we measured it: 20 percent of one core. The speedup was hiding in the network leg.
5 min
03 Jul 26
War story · Research

Why a 122B MoE beat dense on the slow interconnect

Before the 10-gig upgrade, dense models were a trap: the sync tax scales with hidden_dim. The numbers, and the architecture rule we still design around.
6 min
03 Jul 26
War story · Ops

The TX cliff that wasn't the NIC

One host's throughput cratered on every large model. We swapped NICs, cables, drivers, and reproduced it on all of them. The defect was somewhere we weren't looking.
4 min
21 Jun 26
War story · Ops

apt history first: when "nothing changed" but everything broke

A validated cluster node started crashing on a workload it had served cleanly for weeks. Three days down the wrong rabbit hole (mmap theory, KV pressure, tensor-split) before checking the one log that actually mattered.
6 min
On the journal

Every post here is something that actually happened on this hardware, written down while it was still fresh. No roadmap posts, no predictions. If it is in the journal, it either shipped or it broke.