In mid-July I lit a private 10-gigabit link between two freshly built AMD boxes: six older gaming cards total, three per box, running Vulkan. The link’s first job was the first pooled serve of an 88B mixture-of-experts model, split across both boxes with the llama.cpp RPC pattern.
It crawled. Decode sat at 0.90 tokens per second. Prefill was 2.8. The GPUs on both boxes idled through decode, CPU load was low, all 49 layers were offloaded, and the graph split was clean at 7 splits. The link itself had just benched 9.4 Gbps on iperf. Time was vanishing somewhere, and nothing on the usual dashboards owned it.
So everything plausible got blamed. The brand-new fabric. RPC overhead. The old AMD cards. The model’s exotic architecture.
The control
I had a control that acquitted the infrastructure. A 35B mixture-of-experts model pooled over the identical topology, build, and flags decoded at 22.42 tokens per second. One test split “the pool is broken” from “this model is broken.” The fabric, the RPC layer, and the six-GPU cross-box split all walked in one stroke.
The problem lived inside the 88B.
The theory that died overnight
The 88B is a newer hybrid architecture with gated-delta recurrent layers, and the pool was running a build from May. The obvious theory: the old build lacked fast kernels for the new architecture’s recurrent ops. It was plausible, it fit everything visible, and it went in the logbook as the likely root cause.
It died overnight. I deployed a current release build with RPC support on both boxes and reran the identical test: 0.90 tokens per second again, 1113.53 ms per token against the old build’s 1114 to 1116. Build age changed nothing.
Six tokens with the profiler
I needed op-level proof, so I set GGML_VK_PERF_LOGGER=1 to enable the per-op profiler built into the Vulkan backend and ran a single 6-token generation.
It named the killer immediately. MUL_MAT_ID, the expert matmul at the heart of every mixture-of-experts model, was running the IQ2_S expert tensors at 15,456 microseconds per call, about 6.5 GFLOP/s. The IQ3_S tensors ran at 13,711 microseconds, 7.3 GFLOP/s. The same cards run q6_K matrix-vector kernels at over 1,400 GFLOP/s. This was a scalar fallback path, roughly a 200x kernel cliff.
The recurrent layers everyone had blamed ran at 30 microseconds. Innocent. About 27 of those slow expert matmuls per token accounted for essentially the whole 1.1 seconds. Wire traffic during decode was about 9 MB per token, roughly 8 ms at 10 gigabit. The network was never in the frame.
The mechanism
This quant of the 88B carried its expert tensors as IQ2_S and IQ3_S to hit a size target. On these AMD cards under Vulkan, those types have no fast expert-matmul kernel and fall back to scalar code. The 35B control used IQ4_XS, which does have a fast path, which is exactly why the control ran fine.
I requantized the expert tensors to K-quants, specifically Q2_K, leaving attention and the other dense tensors at q6_K. The quantize step took 8 minutes on the build host and produced a 30 GB file at 2.91 bits per weight.
The result was 7.01 tokens per second, a 7.8x recovery. Prefill jumped to 9.94. Output was coherent.
And 7 is not a residual defect, it is this model’s real ceiling on this pool. The 88B activates about 10B parameters per token against the 35B control’s 3B; scale the control’s 22.4 by 3/10 and you land at about 7. Landing clean on the active-parameter prediction is itself the mechanism confirmation. A surviving scalar fallback would have left 1 or 2 tokens per second on the board, not 7.
The rules
Bench a known-good model over the identical topology before blaming the infrastructure. One control A/B split pool-broken from model-broken in minutes, and it saved me from tearing apart a network stack that was never guilty.
Get op-level proof before a root cause goes in the logbook. The kernel-age theory sat in canon for a day. The profiler settled it in one six-token generation. Cheap to run, expensive to skip.
No IQ2 or IQ3-class expert tensors on the AMD Vulkan pool. K-quant experts only.