A '30B' model that runs as fast as a 3B one? That's a Mixture-of-Experts model, and it changes the local-AI math. Here's what MoE means and why it matters for what you can run.
What's the difference between MoE and dense models?
A dense model uses all of its parameters for every token it generates — a 14B dense model does 14B parameters' worth of work each step. A Mixture-of-Experts (MoE) model splits its parameters into 'experts' and activates only a few per token — so a model with, say, 30B total parameters might only use 3B at a time (written like 'Qwen3-30B-A3B', meaning 30B total, Active 3B). The mind-bending result: an MoE model can be huge in total but run as fast as a small one, because only the active experts do work each step. The catch for local AI: you still need enough VRAM to load all the parameters, even the inactive ones. Here's what that means for you.
Why an MoE model can be big AND fast
This is the clever part. Generation speed (tokens per second) depends on how many parameters are actually used per token — the active parameters. In a dense model, that's all of them, so a bigger dense model is slower. In an MoE model, a router picks only a few 'experts' (small sub-networks) to handle each token, so even a very large MoE only does a small amount of work per step. That's why Qwen3-30B-A3B — 30 billion total parameters but only 3 billion active — generates about as fast as a 3B dense model, while being far more capable than one, because it has all those experts to draw on. gpt-oss and DeepSeek-V3 use the same trick, as does the classic Mixtral. So MoE breaks the old rule that 'bigger = slower': you get the capability of a large model at the speed of a small one. It's one of the most important architecture shifts in open models.
MoE models route each token to a few 'experts' — so a huge model does only a little work per token, running fast. · Unsplash
What it means for running AI locally
Here's the practical catch, and it's important for local AI. Even though an MoE model only uses a few experts per token, all the experts have to be loaded into memory — the router might pick any of them at any step, so the whole model sits in VRAM. So Qwen3-30B-A3B runs at 3B-model speed but needs 30B-model VRAM (roughly 18-24GB at 4-bit). That's the trade: MoE gives you the speed of a small model and the capability of a large one, but the memory footprint of the large one. For a local buyer, the implication is that MoE models are fantastic if you have enough VRAM to hold them — you get big-model quality without the slow generation. If you're VRAM-constrained, a dense model of the size that fits your card is simpler and equally valid. There's also a neat trick for the VRAM-limited: because MoE models have few active parameters, they offload to CPU better than dense models — you can run a big MoE partly in system RAM with less speed penalty than you'd expect. So MoE is genuinely great news for local AI: more capability per token of speed, and friendlier to CPU offloading, as long as you can fit (or offload) the full model.
Quick answers
What is a Mixture-of-Experts (MoE) model?
A Mixture-of-Experts model splits its parameters into many 'experts' (sub-networks) and, for each token it generates, a router activates only a few of them rather than the whole model. This means a model with a huge total parameter count (say 30 billion) might only use a small fraction (say 3 billion) per token — written like 'Qwen3-30B-A3B' for 30B total, 3B active. The result is that a large MoE model can generate as fast as a small model, because only the active experts do work each step, while retaining the capability of a much larger model.
What is the difference between MoE and dense models?
A dense model uses all of its parameters for every token, so a bigger dense model is slower and needs more VRAM. A Mixture-of-Experts (MoE) model activates only a few 'experts' per token, so its speed depends on the small number of active parameters, not the total — letting a huge MoE run as fast as a small model. The catch is that all of an MoE's parameters must still be loaded in VRAM, since the router can pick any expert. So MoE gives small-model speed with large-model capability, but large-model memory needs.
Are MoE models better for local AI?
They can be, if you have the VRAM. MoE models like Qwen3-30B-A3B or gpt-oss give you the capability of a large model at the generation speed of a small one, which is excellent — but you still need enough VRAM to load the whole model (all experts), so a 30B MoE needs roughly 30B-model memory. A useful bonus: because MoE models have few active parameters, they offload to CPU/system RAM more gracefully than dense models, so you can run a big MoE partly off-GPU with less speed penalty. For VRAM-rich or CPU-offload setups, MoE is great.
MoE models break the 'bigger = slower' rule — small-model speed, large-model capability, if you can hold them in VRAM. They also offload to CPU better, a gift for VRAM-limited setups. Related: what tokens are and quantization. Source: Aliteq guides.
