gpt-oss-20b on an 8GB or 12GB GPU: yes, with --n-cpu-moe

Most gpt-oss-20b guides, including our own can you run gpt-oss-20b, say 16GB is the minimum. That's true if you want the whole model on the GPU. It isn't true if you're willing to put part of it in system RAM, and for this particular model that's a much better deal than it sounds. Here's why, the exact commands from llama.cpp's own guide, and the numbers to start from on an 8GB or 12GB card.

Why gpt-oss-20b splits so well

gpt-oss-20b is a Mixture-of-Experts model: 24 layers, each with 32 expert networks, of which 4 run for any given token. From the model's own configuration, the experts are about 19.1 billion of its 20.9 billion parameters. The part every token touches (attention, embeddings, routing) is only about 1.8 billion. That's an unusually lopsided split, and it's exactly what llama.cpp's --n-cpu-moe flag exploits. The small always-used part stays on the fast GPU, and the big rarely-used part goes to system RAM.

One note on the numbers: our engine models a Q4_K_M quantisation, while the official GGUF ships in OpenAI's MXFP4 format at 11.27 GiB (per llama.cpp's guide). Our estimate runs slightly high, which errs toward "won't fit", the safe direction.

Does it fit without offloading?

Either way, the answer is no for 8GB and 12GB cards, and tight on 16GB. llama.cpp's guide says 16GB is "just at the edge" and caps context at 32K there. (llama.cpp's figure includes 2.7GB of compute buffers at the large batch size its example commands use, which is why it's higher than ours.) That's where offloading comes in.

The commands (from llama.cpp's own guide)

llama.cpp's official gpt-oss guide has a section for devices with less than 16GB of VRAM. Its example machine is an RTX 2060 8GB:

# llama.cpp official gpt-oss guide (Discussion #15396), accessed 25 Sep 2026
# RTX 2060 8GB — full context:
llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048 --n-cpu-moe 22

# RTX 2060 8GB — 32K context:
llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 32768 --jinja -ub 2048 -b 2048 --n-cpu-moe 16

• -hf ggml-org/gpt-oss-20b-GGUF downloads the official GGUF from Hugging Face.
• --n-cpu-moe N: the llama-server README describes it as "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU". The guide: "finding the optimal value requires some experimentation."
• --cpu-moe keeps all expert weights on the CPU. It's the lowest-VRAM option, useful as a first test.
• --jinja uses the chat template embedded in the GGUF. The guide includes it on every gpt-oss command.
• --ctx-size 0 means "the model's default", 128K for gpt-oss. Lower it to save VRAM.

Starting values for your card

Here's our engine's version of the same calculation: the always-on weights plus KV cache and overhead, then as many layers of experts as fit in 90% of your VRAM. The rest go to system RAM. At 32K on 8GB it lands on --n-cpu-moe 16, the same number llama.cpp's guide uses for its RTX 2060, which is a good sign the maths is sound.

# A 12GB starting point based on the numbers above (flags per the llama.cpp README)
llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 16384 --jinja --n-cpu-moe 6

# Won't start / runs out of memory?  Lower --ctx-size first, then raise --n-cpu-moe by 1–2.
# Plenty of VRAM left?  Lower --n-cpu-moe to keep more experts on the GPU (faster).

On system RAM: the offloaded experts need somewhere to live, on top of your OS and apps. On an 8GB card that's roughly 6–7 GiB of experts, so 16GB of system RAM is the practical floor and 32GB is comfortable. That's my inference from the sizes above, not a figure from the guide. How fast it runs depends on your RAM and PCIe speed. llama.cpp's guide doesn't publish a tokens-per-second figure for its 8GB example, and I'm not going to invent one. Expect it to be slower than the 16GB results, but, as the guide puts it, offloading can still "provide decent performance."

Going deeper

• Tuning --n-cpu-moe in general: our run a big MoE model on a small GPU guide covers the tuning order and multi-GPU tensor placement.
• Why offloading experts can be faster than it sounds: see what --n-cpu-moe actually does.
• Running cost: cost to run gpt-oss-20b prices it on your own card vs renting.
• If you're buying a card for it: 16GB fits it fully. See the best GPU for gpt-oss-20b.