MoE (Mixture of Experts): why compute doesn't grow as much as the model does
Contents
If you understand MoE only through the common assumption that "a bigger model is always heavier and slower," you can't explain why, among two models of the same quality, one runs far lighter than the other.
Background — why this term now
Recent introductions to large language models often include a line like: "the model has tens to hundreds of billions of total parameters, but inference only uses a fraction of them." The usual assumption has been that scaling up a model raises quality but also raises compute cost (and serving cost) in lockstep — this explanation sounds like it breaks that assumption.
This structure is called MoE, or Mixture of Experts. The idea itself was first proposed in a 1991 paper by Jacobs et al., and took its modern form with Google's 2017 research on "sparsely-gated" MoE layers. In 2024, Mistral AI's Mixtral popularized the structure as an open-weight model, and later that same year, DeepSeek's DeepSeek-V3 pushed the ratio to an extreme, making it common to find models where you have to read "total parameter count" and "actually used parameter count" as two separate numbers.
Key data & current state
| Category | Dense model | MoE (sparse) model |
|---|---|---|
| Parameters used per token of compute | Uses all parameters | Uses only the experts the router selects |
| Memory (VRAM) requirement | Based on total parameters | Based on total parameters — all of them still need to be loaded |
| Compute (FLOPs) | Proportional to total parameter count | Proportional to active parameter count |
- Mixtral: Of roughly 47 billion total parameters, top-2 routing means only about 13 billion are actually used in compute per token (Mistral AI research paper, January 2024).
Interpretation: Storage (memory) needs as much as a 47-billion-parameter model, but compute speed is close to a 13-billion-parameter dense model.
- DeepSeek-V3: Of 671 billion total parameters, the active parameters per token are reported at 37 billion (DeepSeek technical report, December 2024).
Interpretation: The total-to-active ratio is far larger than Mixtral's, making the gap between storage cost and compute cost even more pronounced.
- Switch Transformer: Under the same compute (FLOPs) budget, it reported pretraining about 7x faster than T5-XXL (Google research paper, 2021).
Interpretation: This is a case where MoE's effect of "having more parameters at the same cost" was first demonstrated through training speed rather than quality.
Deep dive
1) What exactly is MoE
Each transformer block has an attention layer and a feed-forward (FFN) layer. Instead of using one single FFN layer, MoE splits it into several smaller FFNs that play the same role — called "experts" — and lets each token pick only a few of them. A small neural network called the router (or gate) decides which experts to use: it scores each expert based on the input, then only actually runs the top-k highest-scoring experts (commonly 1 or 2). Experts that aren't selected do no computation at all for that token. The rest of the model — the attention layers, embeddings, and so on — usually stays a shared, dense structure.
2) A common misunderstanding — why it reads this way
The most common misunderstanding is that "a larger total parameter count always means heavier and slower." That's true for dense models, but not for MoE models. Think of someone who owns an entire massive encyclopedia: to answer one question, they don't reread the whole thing — they open just the relevant few entries. The shelf space needed to store the books (memory) still has to cover the whole set, but the time it takes to find an answer (compute) depends only on how many entries were actually opened. You have to read "total parameters" as tied to storage cost and "active parameters" as tied to compute cost separately to understand why this structure is efficient.
3) What actually matters when judging it
There are a few metrics you shouldn't conflate when evaluating an MoE model.
- Total parameters vs. active parameters: Total parameters determine the cost of loading the model into memory, while active parameters determine the compute cost of processing a single token. These are two separate numbers, and you can't declare a model "lighter" by looking at just one of them.
- Choice of top-k routing: Whether each token uses 1 expert (top-1) or 2 (top-2) is a trade-off between quality and compute. Using fewer experts is lighter on compute but risks more quality loss; using more does the opposite.
- Load balancing: If the router is trained as-is, selections tend to concentrate on a small number of popular experts, leaving the rest undertrained. To prevent this, it's common to train an auxiliary loss alongside the main one that evens out how often each expert gets selected.
4) When it helps and when it doesn't matter
MoE has a clear advantage in large-scale distributed training or in large serving environments where throughput matters, because it lets you have more parameters (more "knowledge capacity") at the same compute budget. On a single GPU or local setup with limited VRAM, though, it can actually be a disadvantage — even if per-token compute is light, you still have to load every expert into memory. Fine-tuning also tends to be trickier than with dense models, since touching the router easily disrupts the overall load balance, often requiring a smaller batch size or a different learning rate setup.
In practice — 4 principles for reading MoE correctly
- Don't judge a model's size or speed from the "total parameter" number alone. You need to check active parameters (what's actually used per token) to properly gauge compute cost.
- Memory (VRAM) requirements are calculated from total parameters, not active parameters. "Light" only describes compute — it doesn't apply to storage.
- Not every model labeled "MoE" has the same structure. Depending on how many experts are used (top-k) and how load balancing is handled, two models that are both called "MoE" can differ quite a bit in actual efficiency and stability.
- If you plan to run an MoE model locally or on a single GPU, check first whether you have enough memory to load all the parameters — not just whether compute is light.
References
Contents
Related posts

Training vs. Inference — When Does AI Actually Get Smarter?
It feels like AI gets smarter the more you talk to it, but it does not actually learn anything new mid-conversation. This article lays out how training and inference differ, and why test-time compute became the AI industry's new buzzword in 2026, with data.

Transformer — The Ancestor of Every AI Today
From GPT to Gemini to Claude, every major AI model in 2026 runs on the Transformer architecture Google published in 2017. This article covers how self-attention works and the line of variations from decoder-only models to MoE and Mamba hybrids.

Parameters — 70B, 405B… What Do These Numbers Mean?
The B after a model's name is not a report card; it is closer to a hardware requirements sheet. Read this number as a performance ranking and your model choice is almost guaranteed to be wrong.

Reasoning models — why thinking AI costs more
Reasoning models generate hidden thinking tokens before answering, and those tokens are billed as output. We compare OpenAI/Anthropic docs and DeepSeek R1 to explain the price gap and how to tune costs.