Gemma 4 12B locally: the VRAM, the GPU, and one Ollama gotcha

Gemma 4 12B is the model most people mean when they say they want "a good local model that fits my card." It's Google's mid-size entry in the Gemma 4 family, it handles text, images and audio, and it slots neatly onto a mid-range GPU. This guide sticks to what's verifiable: the specs from Google's model card, the VRAM our engine computes, and the exact command to run it.

What Gemma 4 12B actually is

From Google's model card (read 26 Sep 2026): 11.95B parameters, a 256K-token context window, and multimodal input — text, images, and audio (audio is supported on the E2B, E4B and 12B models). Open weights, licensed Apache-2.0 (per the model card on Hugging Face). It's a dense model, so every parameter runs on each token.

One architecture detail matters for memory: its config.json shows Gemma 4 uses mostly sliding-window attention (40 of its 48 layers use a 1,024-token window, with 8 full-attention layers). That keeps the KV cache small even at long context, which is why the VRAM barely moves as you add tokens.

The other two family members have their own guides: Gemma 4 31B (the 24GB-card option) and Gemma 4 26B-A4B (the mixture-of-experts one that runs fast on less).

That sliding-window design has a practical payoff worth spelling out: the 256K context is genuine, but because most layers only look back 1,024 tokens, the KV cache stays small, so the VRAM figures below barely move whether you feed the model 4K tokens or 40K. On most models, long prompts balloon your memory use; here you mostly pay for the weights, not the context, which is exactly what makes a 12B pleasant to run on a mid-range card. The multimodal side is a bonus rather than the point — if you only ever send text, the vision and audio encoders sit idle, and you can still treat this as a compact, capable text model.

How much VRAM Gemma 4 12B needs

Quantisation is the lever. These figures are computed by our VRAM engine at 16K context (see the full table and other context lengths on cost to run Gemma 4 12B):

Q4_K_M is the sweet spot for a 12B model — near-full quality at a fraction of the size. At ~7.8 GB it even fits an 8GB card, though 12GB gives you comfortable headroom for context and the vision encoder.

Which GPU runs it

At Q4_K_M, a 12GB card is the value pick: an RTX 3060 12GB runs it comfortably. Step up to a 16GB card only if you want to serve the higher-quality Q8 or run long multimodal prompts. Our best GPU for Gemma 4 12B page ranks the options by live rental value.

Referral link

No 12GB card? Rent one for pennies

A 12GB RTX 3060 was listing from about $0.036/hr on Vast.ai's spot market (26 Sep 2026) — enough to run Gemma 4 12B at Q4_K_M. Prices move, so check the live figure before you rent.

See live 12GB GPU pricesCompare live prices first →

Referral link — we may earn a commission at no cost to you. Prices on our compare page are the provider's live figures, cheapest first; this never changes the ranking.

How to run it (mind the tag)

The gotcha first: on ollama.com/library, the bare gemma4 tag pulls the small e4b model, not the 12B. Ask for the 12B by name:

ollama run gemma4:12b

That pulls a Q4_K_M build by default, which matches the ~7.8 GB above. For llama.cpp, point llama-server at a GGUF of the model and it behaves the same way. If you want the full quality, gemma4:12b-fp16 (or a Q8 GGUF) needs the ~13–23 GB from the table.

Who should skip it

Skip Gemma 4 12B if you're on an 8GB card and want headroom — a 9B like Qwen3.5 9B leaves more room. Skip it if you specifically need the strongest reasoning on a 24GB card, where Gemma 4 31B or a 32B-class model earns the extra VRAM. And skip the audio features if you only ever send text — you're paying (in download size) for encoders you won't use.

For the wider picture, our best GPU for local AI buyer's guide covers how VRAM maps to model size across the board.