The number on the screen was 11 of 12, and it was ugly.
We had just promoted a new serving profile for the big Flash-Next model on the cluster. The change was a rebalance of how layers sit across the cards, plus a higher-precision draft head for speculative decoding. The head alone bought about 7.5 percent more decode speed on identical output, so the night had gone well. Then a replay came back and put a hole in it.
What the replay said
We use a small review bot on our pull requests. To test a prompt change for it, we replay a fixed set of real pull requests through the model and count how many come back as proper reviews. On the new profile, the two longest prompts (over 10,000 tokens each) came back as a bare “VERDICT: LGTM” in 10 of 12 cells and a bare “VERDICT:” in one more. Seven tokens, no reasoning, nothing to read. The previous serving line had given zero of those on the same two pull requests across 20 cells, and zero of 95 cells across the whole set.
Same build, same weights, same payloads. The only differences were the layer placement and the draft head. That is a very clean regression story, and we believed it. The plan written down at the time was simple: replicate once, and if it held, roll back to the older line.
The replicate
We replicated on the same coordinator process a few minutes later, with no relaunch. Eight cells this time, four runs each of the two long pull requests, prompts of 16,896 and 11,136 tokens.
Zero of eight collapsed. Every one was a real review, between 151 and 283 words, with a rationale. One of them even flagged a genuine concern about a retry loop. We also posted the exact request body straight to the model server, bypassing our own gateway, to rule out the proxy. It came back with 425 tokens of a real review.
So the same line, minutes apart, had produced 11 of 12 broken and then 0 of 8 broken. Nothing about the model had changed. Something about the test had.
The timings gave it away
The replicate logged per-request timings, and one pair of numbers told the whole story. On the second sample of a pull request, the server reported 16,892 prompt tokens served from cache and 4 tokens actually processed. The third and fourth samples looked the same.
llama-server caches the prompt by default. If you send the same prompt again, or one that shares a long prefix with an earlier one, it reuses the key-value state from the first request instead of recomputing it. That is a good feature for serving. It is a bad one for a test that thinks its samples are independent.
Our replay took three samples per pull request and ran the same pull request through three arms that differed only after a shared prefix. Looking back at the original run’s timings, the pattern was plain: one real prefill per pull request, taking 40 to 60 seconds, and then a string of cells finishing in one to three seconds because they were riding the cache.
So the “12 samples” were not 12. They hung off two prefills. If one of those prefills landed in a bad state, every cell that reused it inherited the bad state. Two bad prefills, each reused four to nine times, reads as 11 of 12. The effective sample size was 2, not 12. And the old line’s tidy 0 of 20 had exactly the same structure, so it was weaker evidence than it looked, too.
What we do not know
We do not know why those two prefills went bad. It might be transient numerics. It might be something in the cached state itself. The record says the cause is unverified, and that is where it stays. This is not a story about a fix to a bug. It is a story about not mistaking one event for a rate.
What we do know is what we did not do. We did not roll back a production serving line on the strength of one correlated burst. The new profile stayed. The cost of being wrong was also bounded: a bare approval from the bot posts with a marker that tells the merge poller there is no rationale, so no auto-merge window opens, and the canary warns on IRC. Nothing could have merged on a seven-token review.
The collateral damage
The same mistake had already leaked into another result. That replay was measuring a new cue for the review bot, a list of definitions removed by a diff. Its headline number was a lift in catches on one pull request, two of three with the cue against zero of three without. Three samples, sharing a cache. That result is weaker than it looks, and the pull request that carries it now says so, with a note to re-run the gate with independent prefills before the cue rolls out to the fleet.
Lessons
- Repeat samples of one prompt against a caching server are not independent. They share the first request’s prefill. Count prefills, not requests.
- Force independence on purpose: send
cache_prompt: false, or put a unique nonce on the first line of every cell so no prefix is shared. - Log the cached token count next to the prompt token count for every cell. If the cached count is close to the prompt length, that sample is a copy.
- Before you act on a surprising quality drop, above all a rollback of something in production, replicate with fresh prefills first. It cost us a few minutes. The rollback would have cost the night.
- The same mechanism makes repeated-prompt prefill benchmarks measure cache hits. We had written that one down before. It applies to quality tests just as well.