In July I strung a 40-gigabit fabric under the cluster. The payoff arrived in two halves that had nothing to do with each other. Prompt processing got its raise from raw bandwidth. Token generation got its raise from RDMA. And for the first stretch, two of the five links quietly declined to participate.
Some context on why the wire matters this much. Six boxes serve a 262B mixture-of-experts model through the llama.cpp RPC backend: a coordinator named Adhara and five NVIDIA workers, the model split across fifteen GPUs. Prefill, the prompt-crunching phase, ships big batches of work across the fabric and is happy as long as the pipe is wide. Decode is the opposite animal. Every generated token round-trips activations across every split point in the model, so the payload is small but the trip count is enormous, and the cost that compounds is per-hop latency, not throughput. One fabric, two workloads, two different definitions of fast.
The old fabric was the 10-gigabit SFP+ backbone I built in April out of secondhand Intel X520s and a cheap switch. It was a huge step up from gigabit, and by mid-July it was the ceiling. The 262B decoded at 18.2 to 18.6 tokens per second and prefilled around 226, and neither number was going anywhere on that wire.
The bet
The upgrade was a stack of ConnectX-3 QSFP cards, the InfiniBand-era silicon that datacenters have been shedding by the pallet for years. I want to be precise about what this fabric is, because the gear invites exaggeration: the cards can speak InfiniBand, but this fabric runs Ethernet mode with RoCEv2 on top. RDMA over Converged Ethernet. Calling it an InfiniBand fabric would be dressing it up for the photo.
There’s a 40-gigabit switch in the rack, a secondhand Brocade ICX-6610, but this fabric doesn’t run through it. It’s hub and spokes: three cards in Adhara with five ports lit between them, one direct DAC cable to each worker, each leg on its own tiny subnet. The coordinator is the center of the star, which is fine, because the coordinator is where every activation was headed anyway.
The first result came from bandwidth alone. Before RDMA entered the picture, just moving the RPC traffic from the 10-gig backbone to plain TCP over the 40-gig legs lifted prefill from 226 to 366 tokens per second. No exotic protocol, no kernel bypass, just a wider pipe feeding the batch. Prefill is a bandwidth customer, and it paid up immediately.
Three honest legs
Then came the first RDMA night, and the fabric’s first lie.
RDMA activated on three of the five legs. The other two probed as unavailable and fell back to TCP, which is exactly what a well-behaved fallback should do, and that is the problem. The cluster kept serving. Nothing errored. Three legs ran RDMA and two ran TCP, and from the outside the only symptom was a decode number, 22.7 to 23.3 tokens per second, that was better than the old fabric but short of what five legs should buy.
The pattern in the failures gave it away: both dead legs terminated on the second port of a dual-port card in the hub. The probe code walked the RDMA devices and asked each one about port 1, always and only port 1. A leg landing on port 2 of a card that already had a leg on port 1 was invisible to it. On a spoke box with one card and one port, the probe was right by accident. On a hub with five ports across three cards, it was wrong twice.
I proved it outside the serving stack first, with a small standalone probe program that asked both ports and activated both dead legs on the spot. Then the same fix went into the RPC backend’s probe, and the A/B stopped being hypothetical.
One verification lesson made the knowledge base permanently. The coordinator’s own log never showed the RDMA activation lines, because launching it detached with output redirected put stdio into block buffering, and the lines sat in a buffer for the whole window. The log said nothing while the kernel was happily running five RDMA connections. The truth lives in rdma resource show qp: five queue pairs in RTS state, one per leg, the two port-2 legs sitting exactly where the standalone probe predicted. Trust the kernel’s table, not the application’s log.
Five legs, and the split verdict
The production window landed the patched build everywhere, activated RDMA on all five legs, and bundled one unrelated fix, a switch to f16 KV cache that closed a latent out-of-memory trap at deep context fill. The window also swapped in a slightly leaner quant of the same 262B; the two bench indistinguishable on decode, so the speed ledger belongs to the fabric.
Decode: 29.2 to 29.9 tokens per second, up 60 percent over the old fabric. Sixty percent, on the same GPUs, the same model, the same active parameters, purely by making every per-token hop cheaper. RDMA is a latency product, and decode is a latency customer.
Prefill: 354.5 tokens per second, against 366 on plain TCP over the same 40-gig legs. Call it parity with a rounding error. Past the point where the wire keeps the GPUs fed, prefill is compute-bound, and no protocol trick buys another token. RDMA did nothing for prefill because bandwidth had already finished that job.
I keep a mental spreadsheet of upgrades and which number they actually moved. This one gets two lines. Bandwidth paid prefill. RDMA paid decode. Neither touched the other’s column.
The rules
Prefill and decode are different workloads with different bottlenecks, so bench both, every time. A fabric upgrade can pay one and stiff the other, and a prefill-only benchmark after a topology change will tell you a comforting lie about decode.
“Link up” is not “path fast.” Every graceful fallback that keeps a cluster serving is also a mask over a degradation. If a fast path can silently downgrade, verify the fast path at the kernel, not in the application’s own log, and especially not in a log subject to stdio buffering.
Whatever probes your hardware, make sure it asks about the second port. Dual-port cards are the default in the secondhand bin, and code that grew up on single-port test rigs will be confidently blind to half of them.
The whole fabric is built from cards that were retired from real datacenters years ago. Forty gigabits with RDMA is a solved, almost free problem on the used market. The price is that nobody ships you the port-2 bug fix. You find that one yourself.