How much VRAM do you need to run an AI model? About 0.57 GB per billion parameters, plus context

How much VRAM do you need to run an AI model? At Q4_K_M, the quant most people run, about 0.57 GB per billion parameters for the weights, plus memory for the context (the KV cache) and about 0.8 GB of runtime overhead. So a 9B model needs about 6.5 GB at 8K context, a 27B about 17 GB, and a 70B about 40 GB for the weights alone. In card sizes: 8GB runs 9B-class models, 12GB runs 12B to 14B, 16GB runs 20B models or a 27B at 3-bit, and 24GB runs a 27B at Q4_K_M with 32K of context. We checked the rule against Ollama's real download sizes on November 3, 2026; for the newest hybrid Qwen models the real files are up to about 1 GB bigger, so keep that headroom. Check your exact model and card in the VRAM calculator.

How much VRAM per billion parameters?

It depends on the quant. A model's size is its parameter count (the "B" in Qwen3.8 27B), and each quant stores those parameters at a different number of bits. A GGUF quant is a bit bigger than its name suggests, because the format stores scales alongside the weights and keeps some tensors at higher precision: in our engine Q4_K_M works out to 4.85 bits per weight and Q8_0 to 8.5, calibrated against published GGUF file sizes. That gives these working figures for the weights alone:

Our figures are in the binary gigabytes VRAM is measured in (a "24GB" card has 24,576 MiB). The quantization guide covers how to choose between quants.

What size model can each GPU run?

Here is the rule turned into card sizes, at Q4_K_M with 8K of context unless the row says otherwise. We leave about 10% of the card free, because a model that needs 98% of VRAM loads and then fails the moment a browser or the desktop wants some.

For what each tier runs best, see the guides for 8GB, 12GB, 16GB and 24GB, or the whole VRAM ladder.

How much VRAM do popular models need?

These are current models at Q4_K_M from our VRAM engine (weights, KV cache at fp16, and about 0.8 GB of overhead), next to the size of the Q4_K_M download Ollama lists for the same model, read on November 3, 2026.

Why is the real download bigger than the estimate?

Mostly units, and for some models a few extra tensors. Ollama lists sizes in decimal gigabytes, which read about 7% higher than the binary gigabytes VRAM is counted in: Llama 3.3 70B's 43GB is about 40 GB of VRAM, which matches our estimate. For Qwen3 32B and Qwen3.5 35B-A3B the two also match. For the newer hybrid Qwen models (Qwen3.5 9B, Qwen3.8 27B) the real files are still about 0.6 to 1.1 GB bigger than our weights figure, and gpt-oss-20b ships in OpenAI's own MXFP4 format rather than Q4_K_M. So on those, add about 1 GB of headroom to our totals.

For running cost per model, see cost to run, for example Qwen3.8 27B.

How much VRAM does context length add?

Anywhere from about half a gigabyte to 8 GB at 32K tokens, depending on the model's design. The KV cache is the model's running memory of the conversation, and it grows with every token. Older models keep a cache on every layer; newer hybrid models keep a growing cache on only some layers. Qwen's model card for Qwen3.8 27B, for example, lists 64 layers in a "16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))" layout, so only 16 of them hold a growing cache.

That is why Qwen3.8 27B uses less memory at 32K than the older Qwen3-32B, and why Qwen3.5 9B stays near 7 GB at 32K (tight on an 8GB card once you add the ~1 GB download headroom, but possible) while Qwen3 8B needs about 10 GB. An 8-bit KV cache halves these figures, at a small quality cost.

Do mixture-of-experts models need less VRAM?

No. Total parameters set VRAM; active parameters set speed. OpenAI's model card describes gpt-oss-20b as "21B parameters with 3.6B active parameters", and it still needs room for all 21B: that is the 14GB download above. Qwen3.8 Flash Next is a bigger case. Its model card lists "125B with 6B activated, plus 51B n-gram embedding and 4B MTP", about 180B of weights in all, so at Q4_K_M it needs roughly 100 GB if everything stays in memory.

What the small active count buys is speed, and the option to keep inactive experts in system RAM. Engines like llama.cpp (with its MoE offload, explained in what --n-cpu-moe actually does) keep the busy parts on the GPU and the rest in RAM, so an 8GB or 12GB card can run a model several times its size. The costs are real: those setups usually run 2-bit to 4-bit quants, and anything read from RAM slows each token down. The most common complaint we see is a model that loads fine and then runs far slower than expected, because part of it spilled out of VRAM. If you go this route, size your RAM too: see how much RAM you need for local AI.

What if the model doesn't fit my card?

Five options, cheapest first:

1. Drop one quant. Going from Q4_K_M to IQ4_XS saves about 12% on the weights; Q3_K_M saves about 19%.
2. Shorten the context, or use an 8-bit KV cache. On older models this can free several GB.
3. Pick a hybrid-attention model. Qwen3.5, Qwen3.8 and Gemma 4 need far less memory per token of context.
4. Offload to RAM. Works best for MoE models; costs speed.
5. Rent a bigger card for the job. On November 3, 2026 (15:20 UTC) our price feed showed a 24GB RTX 3090 on Vast.ai from $0.11 an hour on-demand and a 48GB RTX A6000 there from $0.35 an hour on-demand. Prices move hourly; cloud GPU prices shows today's.

Renting is also the honest way to check this article's numbers on your own model before you spend on hardware. If you're choosing a card to buy, start with the best GPU for local AI.

That's the rule: a bit over half the parameter count in GB at Q4_K_M, plus context, plus headroom. Check any model against your card in the VRAM calculator, then pick a model from the VRAM ladder and set it up with our beginner's guide.