Sparse Expert Model

A sparse expert model is a neural network that contains many expert components but activates only a subset for a given input. In LLMs, the most common form is a sparsely activated Mixture of Experts (MoE).

The key idea is to increase total model capacity without using every parameter on every token. A routing or gating mechanism selects which experts process each token or example. This can make a model much larger in total model parameters while keeping per-token compute closer to a smaller dense model.

Sparse expert models can improve scaling efficiency, specialization, and throughput. They also introduce engineering challenges:

• load balancing across experts;
• routing instability during training;
• communication overhead across devices;
• expert collapse or unused experts;
• more complex fine-tuning; and
• harder debugging and observability.

The word sparse refers to activation, not necessarily storage. The full model may contain many parameters, but only a sparse subset participates in each forward pass.

DeepMind discusses sparse MoE architectures and their tradeoffs in From Sparse to Soft Mixture of Experts.