There's a result that circulates in local-LLM threads and sounds like nonsense: someone moves more of their model off the GPU and onto the CPU, and it gets nearly five times faster. Not five percent. Five times. Every instinct says that's backwards — CPU RAM is slower than VRAM, so shifting work to it should hurt.
The measurement is real. I traced it to a merged llama.cpp pull request with the full llama-bench output attached. But the explanation people give for it is usually wrong, and if you act on the folklore version you'll make your setup slower rather than faster.
What the flag actually does
--n-cpu-moe N does one specific, narrow thing. For layers 0 through N-1, it overrides the buffer type for tensors matching this pattern:
\.ffn_(up|down|gate|gate_up)_(ch|)exps
Those are the routed expert weights — the big stacks of per-expert feed-forward matrices that make an MoE model large. Nothing else moves. Attention weights stay where -ngl put them. So does the KV cache, the router that picks which experts to use (ffn_gate_inp), any shared experts (_shexp), the layer norms and the embeddings.
You can see the shape of it in our model catalogue: on Qwen3-30B-A3B, the routed experts account for roughly 95% of the parameters, while only about 3.3B are active for any given token. Nearly all the memory, almost none of the per-token reading.
Normally, more offload is slower
This is the part the folklore skips. When the model already fits on your GPU, increasing --n-cpu-moe degrades throughput steadily and predictably. A llama.cpp contributor published a clean sweep of it while working on CUDA graph support for this path:
Token generation as offload increases (tg128) — model fits comfortably
GLM-4.5-Air 106B IQ4_XS
ncmoe 8
60.84 t/s
ncmoe 16
48.25 t/s
ncmoe 32
32.84 t/s
ncmoe 64
25.08 t/s
gpt-oss 120B MXFP4
ncmoe 8
95.96 t/s
ncmoe 16
70.42 t/s
ncmoe 32
44.80 t/s
ncmoe 64
40.87 t/s
ncmoe 8
ncmoe 16
ncmoe 32
ncmoe 64
GLM-4.5-Air 106B IQ4_XS
60.84 t/s
48.25 t/s
32.84 t/s
25.08 t/s
gpt-oss 120B MXFP4
95.96 t/s
70.42 t/s
44.80 t/s
40.87 t/s
Monotonic decline, both models. Every layer you push to the CPU is a layer whose expert weights now have to come across PCIe from system RAM instead of out of VRAM. That's the behaviour physics predicts, and it's what you should expect by default.
If more offload is making you faster, something is already wrong. The speedup is a symptom, not a feature.
So where does the 5× come from?
From a different situation entirely — one where the baseline was broken. The measurement comes from the pull request that added `--n-cpu-moe` to llama-bench, where contributor jacekpoplawski ran a 13.70 GiB model on a 12 GB card:
Qwen3-30B-A3B-Instruct-2507 Q3_K_M (13.70 GiB) on a 12 GB RTX 5070, Windows
-ngl 99 alone
Generation (tg128)
11.09 t/s
Prefill (pp512)
378.46 t/s
-ngl 99 --n-cpu-moe 12
Generation (tg128)
54.04 t/s
Prefill (pp512)
1,253.62 t/s
Change
Generation (tg128)
4.87×
Prefill (pp512)
3.31×
Generation (tg128)
Prefill (pp512)
-ngl 99 alone
11.09 t/s
378.46 t/s
-ngl 99 --n-cpu-moe 12
54.04 t/s
1,253.62 t/s
Change
4.87×
3.31×
Look at the model size against the card. 13.70 GiB does not fit in 12 GB. That's the entire story.
On Windows, NVIDIA's driver has a feature called System Memory Fallback. When a CUDA allocation won't fit in VRAM, rather than failing it silently backs the allocation with host RAM and pages it over PCIe on demand. Your model "loads". It runs. It runs terribly, because the driver is now shuttling weights across the bus with no idea which ones matter, thrashing on whatever it guesses wrong about.
The 11 t/s baseline isn't GPU inference. It's a driver frantically paging. When you then set --n-cpu-moe 12, you're not adding a penalty — you're taking control of a decision the driver was already making badly. You tell it exactly which tensors live in host memory, everything else fits properly in VRAM, and the thrashing stops. Hence 54 t/s.
The actual shape: a V, not a slope
Put both findings together and the real curve has three regions.
**Too little offload** — the model overcommits VRAM. On Windows you get driver paging and catastrophic throughput. This is the left arm of the V.
**The knee** — the smallest N where everything genuinely fits in VRAM. This is the fastest configuration, and it's what you're looking for.
**Too much offload** — every extra layer sends more expert weight across PCIe for no benefit. Slow, steady decline. The right arm.
Which reframes the tuning problem completely. You're not looking for "how much can I offload" — you're looking for the least you can get away with. Start at zero, increase until it fits without overcommitting, stop. Every step past that costs you.
Three reasons this may not apply to you
I'd rather tell you when advice is irrelevant than let you apply it blindly, so:
You're probably on a build that fixes this automatically. llama.cpp's --fit behaviour has become default-on, and it auto-sizes offload to what will actually fit. That largely prevents anyone from landing in the overcommit regime in the first place. This advice increasingly matters for older builds, manually forced configurations, and non-CUDA backends.
On Linux you'd probably have seen an error instead. Stock CUDA on Linux doesn't do the silent host-memory fallback unless you opt in with GGML_CUDA_ENABLE_UNIFIED_MEMORY=1. Without it, an overcommitted model usually just fails to allocate and tells you so. Which is, frankly, the better behaviour — a loud failure beats a quiet 5× slowdown.
Your system RAM speed matters more than the flag does. Once expert weights live in host memory, their read speed is your memory bandwidth. Dual-channel DDR5 is in a different league from quad-channel or server memory, and the same --n-cpu-moe value will behave very differently across those machines. Any specific number someone quotes you is a number from their motherboard, not yours.
Quick answers
What value of --n-cpu-moe should I use?
The smallest one where the model fits in VRAM without overcommitting. Start at 0 and increase until it loads cleanly — then stop, because every further step costs throughput. There is no universally good value; it depends on your model, quantisation, context length and card.
Does --n-cpu-moe help dense (non-MoE) models?
No. It targets routed expert tensors specifically, and a dense model doesn't have any. For dense models the equivalent lever is -ngl, which controls how many whole layers live on the GPU.
Why did my throughput get worse after following this advice?
Almost certainly because your model already fitted. In that case you've moved expert weights from VRAM to system RAM for no reason, and you're now reading them over PCIe. Reduce the value, or remove the flag entirely and let a recent llama.cpp size it for you.
Does the KV cache move to the CPU too?
No. The flag only overrides tensors matching the routed-expert pattern. The KV cache stays wherever it was — which matters at long context, because a large KV cache can push you back into overcommit even when the weights fit fine.
Is this the same as CPU offload in vLLM or ktransformers?
The idea is related — keep the hot path on the GPU, push bulky rarely-read weights elsewhere — but the implementations and the flags differ, and numbers are not comparable across them. Measure on your own stack rather than porting a value someone quoted for a different runtime.
If you want to know whether your model fits before you start guessing at flags, our cost-to-run pages compute the weight and KV-cache requirement per quantisation from each model's own configuration, so you can see which term is pushing you over the edge — and it's more often the KV cache at long context than people expect.
