The MoE offload trick that makes your model 5× faster — and why it probably won't work for you

There's a result that circulates in local-LLM threads and sounds like nonsense: someone moves more of their model off the GPU and onto the CPU, and it gets nearly five times faster. Not five percent. Five times. Every instinct says that's backwards — CPU RAM is slower than VRAM, so shifting work to it should hurt.

The measurement is real. I traced it to a merged llama.cpp pull request with the full llama-bench output attached. But the explanation people give for it is usually wrong, and if you act on the folklore version you'll make your setup slower rather than faster.

What the flag actually does

--n-cpu-moe N does one specific, narrow thing. For layers 0 through N-1, it overrides the buffer type for tensors matching this pattern:

\.ffn_(up|down|gate|gate_up)_(ch|)exps

Those are the routed expert weights — the big stacks of per-expert feed-forward matrices that make an MoE model large. Nothing else moves. Attention weights stay where -ngl put them. So does the KV cache, the router that picks which experts to use (ffn_gate_inp), any shared experts (_shexp), the layer norms and the embeddings.

You can see the shape of it in our model catalogue: on Qwen3-30B-A3B, the routed experts account for roughly 95% of the parameters, while only about 3.3B are active for any given token. Nearly all the memory, almost none of the per-token reading.

--cpu-moe vs --n-cpu-moe vs -ot

All three flags do the same kind of thing: they tell llama.cpp to keep certain weight tensors in system RAM. They differ only in how much you control. --cpu-moe moves every MoE expert weight. --n-cpu-moe N moves them for the first N layers only. -ot lets you write the pattern yourself.

Under the hood, the first two are shortcuts for the third. --cpu-moe adds one override with the pattern shown above. --n-cpu-moe 3 adds three overrides, one per layer, of the form blk\.0\.ffn_(up|down|gate|gate_up)_(ch|)exps, then blk\.1… and blk\.2…. So these two commands do the same thing:

llama-server -m model.gguf --n-cpu-moe 3
llama-server -m model.gguf -ot "blk\.(0|1|2)\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU"

Which one should you use? Use --n-cpu-moe almost always. It has one knob, and that knob maps straight onto the V-shaped curve below. Use --cpu-moe when the model is far too big for your card and you want the simplest possible start. That is the same as setting --n-cpu-moe to the layer count. Reach for -ot only when you need a pattern the shortcuts can't express, like pinning specific layers to a second GPU. The text after the = must be a buffer type your build knows. If it isn't, llama.cpp prints the list of available buffer types and stops.

If you searched for the phrase "number of layers for which to force MoE weights onto CPU", that is this same N. It counts layers from the start of the model, not experts. N = 0 moves nothing. N equal to the model's layer count is the same as --cpu-moe.

LM Studio added an all-or-nothing version in 0.3.23 (12 August 2025) and replaced it with a per-layer slider for n_cpu_moe in 0.4.0 (28 January 2026). Its release notes describe an advanced model load setting that places all MoE expert weights onto the CPU or the GPU, with GPU as the default. That switch is the --cpu-moe equivalent.

Normally, more offload is slower

This is the part the folklore skips. When the model already fits on your GPU, increasing --n-cpu-moe degrades throughput steadily and predictably. A llama.cpp contributor published a clean sweep of it while working on CUDA graph support for this path:

Monotonic decline, both models. Every layer you push to the CPU is a layer whose expert weights now have to come across PCIe from system RAM instead of out of VRAM. That's the behaviour physics predicts, and it's what you should expect by default.

So where does the 5× come from?

From a different situation entirely — one where the baseline was broken. The measurement comes from the pull request that added `--n-cpu-moe` to llama-bench, where contributor jacekpoplawski ran a 13.70 GiB model on a 12 GB card:

Look at the model size against the card. 13.70 GiB does not fit in 12 GB. That's the entire story.

On Windows, NVIDIA's driver has a feature called System Memory Fallback. When a CUDA allocation won't fit in VRAM, rather than failing it silently backs the allocation with host RAM and pages it over PCIe on demand. Your model "loads". It runs. It runs terribly, because the driver is now shuttling weights across the bus with no idea which ones matter, thrashing on whatever it guesses wrong about.

The 11 t/s baseline isn't GPU inference. It's a driver frantically paging. When you then set --n-cpu-moe 12, you're not adding a penalty — you're taking control of a decision the driver was already making badly. You tell it exactly which tensors live in host memory, everything else fits properly in VRAM, and the thrashing stops. Hence 54 t/s.

The actual shape: a V, not a slope

Put both findings together and the real curve has three regions.

Which reframes the tuning problem completely. You're not looking for "how much can I offload" — you're looking for the least you can get away with. Start at zero, increase until it fits without overcommitting, stop. Every step past that costs you.

Three reasons this may not apply to you

I'd rather tell you when advice is irrelevant than let you apply it blindly, so:

You're probably on a build that fixes this automatically. llama.cpp's --fit is on by default in current source. It auto-sizes offload to what will actually fit. That largely prevents anyone from landing in the overcommit regime in the first place. This advice increasingly matters for older builds, manually forced configurations, and non-CUDA backends. The next section shows exactly how --fit and --n-cpu-moe interact.

On Linux you'd probably have seen an error instead. Stock CUDA on Linux doesn't do the silent host-memory fallback unless you opt in with GGML_CUDA_ENABLE_UNIFIED_MEMORY=1. Without it, an overcommitted model usually just fails to allocate and tells you so. Which is, frankly, the better behaviour — a loud failure beats a quiet 5× slowdown.

Your system RAM speed matters more than the flag does. Once expert weights live in host memory, their read speed is your memory bandwidth. Dual-channel DDR5 is in a different league from quad-channel or server memory, and the same --n-cpu-moe value will behave very differently across those machines. Any specific number someone quotes you is a number from their motherboard, not yours.

How to pick N on a current llama.cpp build

On a current build, try no MoE flag at all first. The default --fit on already does a version of --n-cpu-moe for you. If you then add --n-cpu-moe yourself, --fit stops placing weights and your N is final. So a manual value only helps when it beats the automatic one.

Here is what the fit code does, read in common/fit.cpp on 4 October 2026. It only adjusts settings you did not set yourself. It aims to leave a safety margin of 1,024 MiB free on each GPU (change it with --fit-target). It may shrink the context first, but never below 4,096 tokens (change it with --fit-ctx). For a MoE model it then fills the GPU with the small, hot tensors first, with all expert weights in RAM. After that it moves expert weights back onto the GPU while they still fit. That is the "knee" of the V, found for you.

The catch is in the same file. If you set -ngl, --n-cpu-moe, --cpu-moe or -ot yourself, the fit step logs "failed to fit params to free device memory" and leaves weight placement to you. It can still have trimmed the context before that point. So you have two sane workflows:

A dense model has no expert tensors, so --n-cpu-moe does nothing for it. Since 27 August 2026 (PR #26622) llama.cpp has a sibling for that case: --n-cpu-ffn N (short -ncffn). It keeps the dense feed-forward weights of the first N layers in RAM. The gpt-oss-20b on an 8GB or 12GB card walkthrough shows the MoE version on a real model.

To find a starting value for your own card, use the per-model numbers in our guide to the best MoE models for a 12GB or 16GB GPU, which also shows how much expert data each model reads from RAM per token.

If you want to know whether your model fits before you start guessing at flags, our cost-to-run pages compute the weight and KV-cache requirement per quantisation from each model's own configuration, so you can see which term is pushing you over the edge — and it's more often the KV cache at long context than people expect.

Related guides

• Best MoE models for a 12GB or 16GB GPU: which models are worth offloading, with RAM read per token.
• Run a big MoE model on a small GPU: the step-by-step version of this page.
• llama.cpp MoE offloading: the wider picture, including -ot recipes.
• gpt-oss-20b on an 8GB or 12GB card: the flag applied to one popular model.
• Interactive VRAM calculator — drag the context slider and watch the KV cache fill the card.
• RTX 5090 vs two RTX 3090s — one fast card or two big ones.
• Cheapest way to serve Llama 70B — rent or buy, and on what.
• Is the DGX Spark worth it? — whether a big unified-memory box is right for you.