Why a 122B MoE beat dense on the slow interconnect

The trap of total size

I used to look at a model’s total parameter count and assume it was the primary indicator of quality. That instinct works fine when you have a single machine with fast interconnects. You load the weights, the GPU crunches them, and you get output. The bottleneck is the silicon.

But my lab runs models too large for one card. We split them across a cluster of repurposed gaming hardware. The nodes talk over the network using an RPC pattern similar to llama.cpp. Before we upgraded to 10-gigabit Ethernet, that link was a single gigabit pipe.

On a slow link, the network becomes the floor on throughput. The GPU math is fast enough; the wire is not.

The trap was assuming that a dense model of a given size was always better than a mixture-of-experts (MoE) model of similar total size. I thought dense was “more model per parameter.” On a single box, that is true. Across a slow network, it is a liability.

The investigation

I wanted to see how different architectures performed when forced to span multiple nodes. I set up a pool of six older AMD gaming cards spread across two machines. They ran on Vulkan. The interconnect was standard gigabit Ethernet.

I started with the intuitive choice: a dense model. I picked a dense 8B model (llama3.1-8B, Q4). It ran at 22.04 tokens per second. That felt decent. It was fast because the model fit comfortably within the network’s ability to move the necessary data.

Then I looked at a 30B MoE model (Heretic 30B with about 3B active parameters per token, Q2_K). I expected it to be slower because it was larger on disk. I was wrong. It ran at 17.22 tokens per second.

I thought the quantization was hurting it. I re-ran the same 30B MoE at Q6_K. The result was 17.85 tokens per second. The precision barely moved the number. The bottleneck was not the on-card math or the weight precision. It was the network sync floor.

Finally, I tested a dense 32B model (Qwen3-VL-32B, Q8). This was the same size class as the MoE. I expected it to crush the MoE because dense models do more compute per token. It ran at 6.67 tokens per second.

That is about 2.7 times slower than the similarly sized MoE. The dense model put far more data on the wire per token. The wire set the pace, and the dense model lost.

Why dense loses on the wire

When a model is split across nodes, every token requires the nodes to synchronize. The cost of that synchronization scales with how much of the model is active per token. This is the hidden dimension.

A dense model activates all of its weights on every token. What crosses the network per token is not the weights, which stay resident on each node, but the hidden-state activation handed off at each node boundary, and that payload is sized by the model’s hidden dimension. A dense 32B model in this class has a much larger hidden dimension and does full-width compute at every layer, so both its per-token sync payload and its compute are bigger. The network saturates on that larger activation, and the pace falls to whatever the wire can move.

An MoE model activates only a few small experts per token. Even though the full model is 30B on disk, only 3B are active per token. The amount of data crossing the wire is small. The network stays clear. The GPUs keep crunching.

The numbers proved it. The dense 32B model was slower than the MoE because the dense model’s sync tax was too high for the gigabit link. The MoE’s active parameter count kept the wire cost low.

The rule we kept

This was the moment I stopped judging models by their total size. The thing that matters on a slow interconnect is the active-parameter count per token, not the model’s total size on disk.

We design our deep, multi-node models around active parameters. This is why our production model is a large MoE (a 122B-class model with roughly 10B active per token), not a dense model of similar capability. The big MoE keeps the per-token wire cost small. It stays fast even when spread across nodes. A dense model of comparable strength would crater on the same links.

The 10-gigabit upgrade later raised the floor. The network is no longer the primary bottleneck. But the rule still drives our design. When nodes talk over a link that is slow relative to the GPUs, choose the architecture that puts the least on the wire per token.

On a slow interconnect, MoE wins and dense is a trap. Design for active-parameter count, not total size.

The cost asymmetry

Picking dense “for quality” feels safe. It is free to assume. You see the total parameter count, you see the quality metrics, and you think you are making the right choice.

The miss only shows up after you have split the model across nodes and watched it crawl. By then, you have spent weeks tuning the cluster, optimizing the RPC calls, and fighting the network saturation. The model is already deployed. The architecture is already fixed.

The cheap move is to design for active-parameter count up front. It takes five minutes to check the active parameter count of a model before you commit to it. It saves weeks of debugging a bottleneck that was there from the start.

I learned this the hard way. I will not make the same mistake again.