Qwen3-235B-A22B sounds like a model that needs a rack of H100s. It doesn't — not to run it, anyway. It has 235 billion total parameters, but only about 22 billion of them activate for any given token, because it's a mixture-of-experts model, not a dense one. Route the right tensors to the right hardware in llama.cpp and that difference is the whole trick: a 16GB consumer GPU can drive a model that the parameter count alone says shouldn't fit, and most people running local AI have never touched the flags that make it happen.
Why -ngl alone leaves performance on the table
Most people's first exposure to GPU offloading in llama.cpp is -ngl, short for --n-gpu-layers. Set it to 99 and it tries to push effectively everything onto the GPU; set it lower and it offloads that many layers in order, top to bottom. That's a blunt instrument for a mixture-of-experts model, because -ngl doesn't know or care which tensors inside a layer are cheap to compute and which ones are enormous. It can strand small, cheap attention tensors on the GPU right alongside the massive expert weight matrices that are actually eating your VRAM — using up scarce GPU memory on the wrong thing.
That --n-cpu-moe 40 tells llama.cpp to move the expert (FFN) weights from the first 40 layers onto system RAM, while -ngl 99 still tries to keep everything else on the GPU where it's fast. The -c flag sets context length; -b and -ub set the logical and physical batch sizes. Their defaults — 2048 and 512 — are tuned for GPU-only inference and tend to be too small once you're splitting work across CPU and GPU, so most guides for this setup push -ub up toward 1024 or higher to keep the handoff from becoming the bottleneck.
The more surgical version: --override-tensor
--n-cpu-moe is a blunt cutoff — it moves whole layers' worth of experts at once. --override-tensor (-ot) is regex-based and lets you target individual tensor patterns instead, which squeezes more of the model onto the GPU than a flat layer cutoff would.
That pattern sends only the feed-forward expert weight matrices from layer 19 onward to CPU, while attention layers and the experts in the first 18 layers stay on the GPU, where they're cheapest to compute. It's fiddlier to write than --n-cpu-moe, and it's easy to typo a regex into loading nothing — but it's the difference between guessing at a layer count and dialing in exactly how much VRAM headroom you actually have.
1
Confirm the model is actually MoE — check the model card or config.json for a num_experts or moe field before you plan a build around this.
2
Start with a GGUF quant you can partially fit — Q4_K_M is the usual first test.
Set --n-cpu-moe to roughly how many layers' worth of experts your VRAM can't hold, then raise -ub toward 1024 or higher so CPU-GPU handoff doesn't become the bottleneck.
4
Benchmark with llama-bench before you commit to a setup — a five-minute run tells you more than any forum post's numbers, because your RAM speed and core count change the result.
Offloading trades GPU VRAM for system RAM bandwidth, so fast dual-channel memory matters more here than almost anywhere else in a local-AI build. · Unsplash
This is also where the comparison in ollama-vs-vllm-vs-lm-studio-which-to-run-2026 undersells raw llama.cpp for this specific job — Ollama and LM Studio both wrap llama.cpp under the hood, but neither app's interface currently exposes --override-tensor directly, so anyone doing this level of tuning is better off running llama-server themselves. And the RAM you're leaning on for all of this is exactly the resource covered in how-much-ram-storage-local-ai-agents-2026 — fast dual-channel DDR5 matters more for this trick than almost anything else in the build.
Does MoE offloading work in Ollama or LM Studio, or only raw llama.cpp?
The underlying engine in both is llama.cpp, so the capability exists, but neither app's UI currently exposes --override-tensor directly — you'll get more control running llama-server yourself.
Will this let me skip buying a bigger GPU?
For MoE models, often yes — that's the whole point. For dense models, no; there's no shortcut around loading every active parameter somewhere fast.
Does this hurt output quality?
No — quantization affects quality slightly (Q4 versus Q8), but which device runs a given tensor doesn't change the math, only the speed.
What RAM speed do I actually need for this to be worth it?
Dual-channel DDR5 at a decent clock is the realistic minimum; single-channel or older DDR4 kits will bottleneck hard enough that the comparison to a GPU-only setup stops being close.
The honest caveat: this isn't a free lunch, it's a trade. You're spending RAM bandwidth and CPU cycles to buy back VRAM headroom, and on a slow CPU or a single-channel RAM kit, that trade goes bad fast. But for the growing pile of huge open MoE models landing every month, it's the difference between 'needs an $8,000 workstation' and 'runs on the GPU you already own.' Worth ten minutes of flag-reading before you go shopping for best-gpu-for-local-ai-2026 or a prebuilt-vs-diy-local-ai-pc-2026 build you might not need.
