how to run a 30B model on a 12GB GPU: the one llama.cpp flag that makes it possible

Here's a trick that feels like cheating. A modern 30-billion-parameter Mixture-of-Experts model needs around 20GB of VRAM to run the normal way — out of reach for a 12GB card like an RTX 3060. But add one flag to your llama.cpp command, --n-cpu-moe, and that same 3060 will run it. Not at full speed, but genuinely usable. The flag works because of a quirk of how MoE models are built, and once you understand why it works you can tune it precisely for whatever card you have. This is the single highest-leverage setting for running big models on modest hardware, and most people either don't know it exists or set it by guesswork. Let me fix both.

Why this works only for MoE models

To understand --n-cpu-moe you need one fact about Mixture-of-Experts models: most of the model is experts, and only a few experts run per token. A model like Qwen3-30B-A3B has dozens of experts but activates only a handful for any given token. The experts are the bulk of the weights — big — and any individual expert is used infrequently. That combination is what makes offloading them to slower CPU memory a good trade: you free up the most VRAM (because they're big) for the least speed penalty (because each one is rarely needed). The attention layers, by contrast, run on every token, so those stay on the fast GPU.

This is why the trick does nothing for a dense model like a standard 70B: in a dense model every parameter runs on every token, so there's no rarely-used bulk to banish to slow memory without paying for it constantly. MoE is the special case, and it's why the MoE architecture has become such a gift to people running models on consumer hardware. For the deeper mechanics of what the flag does under the hood — and the counterintuitive throughput result — see our breakdown of what --n-cpu-moe actually does.

The command to copy

Here's a working starting point for running Qwen3-30B-A3B (Q4_K_M) on a 12GB card. Adjust the model path and the --n-cpu-moe value to taste:

./llama-server \
  -m Qwen3-30B-A3B-Q4_K_M.gguf \
  --n-cpu-moe 32 \
  -ngl 99 \
  -c 16384 \
  -fa 1

# --n-cpu-moe 32  : offload experts from the top 32 layers to system RAM
# -ngl 99         : put as many layers on the GPU as will fit
# -c 16384        : 16k context (lower this first if you OOM)
# -fa 1           : flash attention on

Starting values by card

The logic is simple: lower --n-cpu-moe keeps more experts on the GPU (faster) but uses more VRAM; higher offloads more (slower) but uses less. Your goal is the lowest number that still starts without running out of memory — that keeps the most work on the fast GPU. A 24GB card usually runs a 30B MoE with no offload at all; you only reach for this flag when the model is bigger than your VRAM.

When the server won't start

The most common failure is an out-of-memory error at startup — the weights fit but the context pushes you over. The community-tested tuning order is worth following exactly: cut context length first, because it's the fastest way to recover VRAM. Drop --ctx-size from 16384 to 8192 and try again. If it still won't start, raise --n-cpu-moe in small steps — 32 → 36 → 40 — until it loads cleanly. Then, once it runs, walk the numbers back toward more speed: nudge --n-cpu-moe down and context up until you find the edge, watching your VRAM usage on each launch. It's trial and error, but a handful of restarts gets you to a good setting.

That's the flag that quietly expands what your hardware can do. If you've been eyeing a big MoE model and assumed your card couldn't handle it, --n-cpu-moe is probably the answer — size the model in our VRAM calculator, start with the values above, and tune from there. And if you want the theory behind why offloading experts is such a good deal, our what --n-cpu-moe actually does piece goes deeper on the mechanics.