your GPU isn't too small for that 235B model — you're just using it wrong
A handful of llama.cpp flags let a 16GB card run models that shouldn't fit, and most people running local AI have never touched them.
Aliteq
Lena Fischer · AI & Local Compute Editor
Mixture-of-experts (MoE) models like Qwen3-235B-A22B activate a small fraction of their total parameters per token — 22B out of 235B — which is what makes CPU offloading viable at all.
llama.cpp's --n-cpu-moe flag keeps expert (FFN) tensors on CPU RAM while routing and attention stay on the GPU, instead of the all-or-nothing choice --n-gpu-layers forces.
The more surgical --override-tensor (-ot) flag targets specific layers by regex, so you can send only the later layers' expert weights to CPU and keep everything else on the GPU.
This trick barely helps dense models — a 70B-class model activates every parameter on every token, so there's no cheap-to-strand expert tensor to offload.
Quantization stacks with offloading: dropping from Q8_0 to Q4 roughly cuts memory need by another 75%, and testers have run 30B+ MoE models at usable speeds on cards with under 8GB of VRAM.
Aliteq
Read the full story
your GPU isn't too small for that 235B model — you're just using it wrong