We needed speed. The lab’s agent lane was running a 35B mixture-of-experts model that had won our tool-use benchmark at 80 percent, but it was slow. A rival model decoded at 24.8 tokens per second versus our 19.5 because it ran speculative decode. The rival had a multi-token-prediction head. Ours did not.
The fine-tune we were using was a public release. Its base model shipped with that head, but the fine-tune stripped it out. The base file had 753 tensors. Ours had 733. We were missing 20 tensors and one metadata key.
A repo surfaced on Hugging Face where a third party had already grafted the stock base head onto our fine-tune body. They claimed a 28 percent speedup on different AMD hardware. It was an existence proof. We could copy their weights and go home.
We did not copy their weights.
We ran a feasibility pass. Header range-reads, about 11 MB per file, showed the donor file was literally our file plus the head. The architecture dims were identical everywhere that matters. The plan inverted. We did not take the stranger’s bytes. The lab’s own archive already held a head-preserving requant of the base model, built earlier in our pipeline. It was the exact same head from the same origin, but in our quant types. The donor’s head used a quant family that had measured 3.2x slower on these AMD cards. Zero external tensors. Cleaner provenance.
I also learned something about the model body itself. The work item assumed the model had been through weight surgery. The build log showed it was a plain requant of stock weights. The head-mismatch risk shrank accordingly.
The graft was simple surgery. I copied 20 tensors. I bumped one metadata integer, the block count, from 40 to 41. I added one metadata key, the MTP layer count, set to 1. It was a pure tensor copy. No quant math.
I set a gate. The output tensor count must be 753. Every one of the 20 head tensors had to verify name, shape, type, and sha256 exact against a manifest hashed from the source file.
The bench ran on the same box, same serving flags, greedy decoding.
The baseline was 26.8 to 28.1 tokens per second depending on prompt shape.
With the graft and speculative decode on, short prompts went up 16 percent, from 31.5 to 32.5. Medium prompts went up 30 percent, from 34.1 to 36.1. Long prompts went up 10 percent, from 29.6 to 29.7. Structured tool-call output, the lane’s actual workload, went up 48 percent, from 39.3 to 41.7 at 87 percent draft acceptance.
We deployed to both lane boxes after human review. We saw 37.7 and 37.0 tokens per second on the structured probe at 76.7 percent acceptance, versus about 27.6 before. Rollback is two lines: point back at the old file, drop two launch flags.
The honest wrinkle came during verification.
The planned correctness gate was temp-0 output identity versus the ungrafted model. Speculative decode is verified speculation, so greedy output should be byte-identical. The check failed.
Before blaming the graft, we ran the control. The unmodified baseline server returned three different outputs for three identical greedy requests. The serving stack itself is not run-to-run deterministic on these cards. Batch numerics flip near-tie token choices. Byte-identity was never achievable here. Any graft-versus-baseline diff was unattributable.
Correctness instead rests on the mechanism. Speculative decode only accepts a drafted token when it equals the main model’s own next-token choice at that step. The emitted text is always something the main model would produce. Acceptance sat in the expected band. Structured outputs did come back byte-identical across all legs.
When an identity check fails, control the baseline before blaming the change.
One more measurement trap. The first post-deploy probe read 26.7 tokens per second and nearly failed the bar. The box’s background volunteer-compute load only parks itself when inference arrives through the routed front door. The probe had gone in the side door. One routed request parked it. The identical probe then read 37.7.
We judge a verification gate by its mechanism. Draft-and-verify makes correctness structural. The byte-diff it replaced was weaker and, on this stack, impossible.
Before importing a stranger’s weights, check your own archive. The third-party repo was worth its existence proof. The actual donor was in-house, in our own quant types, with clean provenance.