The MoE offload trick that makes your model 5× faster — and why it probably won't work for you

The short answer

Adding CPU offload with --n-cpu-moe normally makes an MoE model slower, not faster. The famous 5× speedup only happens when the model didn't fit in VRAM in the first place — at which point you…

Performance against offload depth is a V shape: catastrophic while overcommitted, best at the smallest value that genuinely fits, then a slow decline as you push further.

The flag moves routed expert weights only. Attention, KV cache, the router, shared experts and norms all stay put.

This mostly bites on Windows, where NVIDIA's driver silently falls back to system memory instead of failing. Stock Linux usually just runs out of memory and tells you.

Recent llama.cpp auto-sizes offload by default, so newer builds largely prevent you from landing in the bad regime at all.

Why this particular split is clever

In a mixture-of-experts model the routed experts are most of the weight but only a fraction of the work per token — the router picks a handful and ignores the rest. Attention is the opposite: small,…