Mixture of experts

What a mixture of experts is

Imagine a team with several specialists. A coordinator chooses a few to handle each task. A mixture-of-experts model works in a similar way. A part called the router chooses a few expert blocks for each token. The model combines their results. The experts are learned parts of the model, not people.

Some training methods add a load-balancing loss to discourage a few experts from receiving all tokens. Some implementations impose a capacity limit per expert and drop or reroute overflow; others process every selected assignment. Dropping or rerouting can change the output.

Some models use shared experts that every token uses, plus routed experts. Others use fine-grained experts with many small blocks. The names and the counts differ, but the serving problem is the same: route, gather, compute, combine.

Picture shared layers and a shelf of experts. Each token uses the shared layers and the experts chosen by its router. The entire shelf must be available, even though a token uses only part of it.

Memory and compute

Total parameters largely determine resident weight memory. Active parameters help estimate per-token compute and weight traffic; batch routing, caching, and implementation also matter. For a model with E experts, a shared part S, and top-k routing:

For example, a model with 120B total parameters and 30B active needs about 60 GB for its 4-bit weight payload, before scales and cache. One token uses about 15 GB of active weights. At 273 GB/s, a weight-only estimate is about 18 tokens/s. The model uses less compute than a dense model with 120B parameters, but still needs space for all its weights.

The router does not by itself reduce the memory needed to keep all experts resident. Any expert may be selected for a later token. A model that does not fit must shard across devices or offload experts, and both add latency.

Routing and balance

A trained router can favor a few experts. That is load imbalance. The hot expert becomes a queue, while the other experts idle. Tail latency rises even when average utilization looks fine.

Imbalance can leave one device busy while others wait. Reusing a hot expert within a batch can save weight reads, so routing skew alone does not tell you how much bandwidth is wasted. Fine-grained routing and a balancing loss reduce the problem, but they do not remove it.

Expert parallelism splits experts across devices. Each token then needs an all-to-all exchange: send the token to the device that holds its expert, then send the result back. That link can become the bottleneck. Tensor parallelism has different traffic, and the two are often combined.

Serving a sparse model

Sparse serving needs grouped matrix multiplies, a routing kernel, and a combine kernel. Small batches waste the wide matrix units, because each expert receives few tokens. Large batches improve utilization but raise cache and queue pressure.

Measure the tail, not only the mean. A hot expert can make p95 much worse than p50. If the implementation can drop assignments, check the drop rate too. A dropped assignment is a quality change, not only a speed change.

Quantization interacts with routing. A 4-bit expert format can cut weight traffic, but the router and combine steps add overhead that does not shrink. Profile the full path before you claim a speedup.

When MoE is a bad fit

• Memory-limited devices. If the full expert set does not fit, sharding or offload adds latency.
• Small batches. Each expert gets few tokens, so the matrix units are underused.
• Strict tail latency. A hot expert or an all-to-all hop can break the target.
• Simple quality needs. A smaller dense model can be easier to serve and to evaluate.

MoE is not a free quality upgrade. The router can make mistakes, and the model can fail on rare inputs that route poorly. Test the routing distribution on your own traffic.