Gemma 4 31B locally: the 24GB-card Gemma

Gemma 4 31B is the model to reach for when a 12B isn't quite enough and you have a 24GB card to feed it. It's the large dense member of the Gemma 4 family — more capable than the 12B, and squarely a 24GB-VRAM proposition.

What Gemma 4 31B actually is

From Google's model card (read 26 Sep 2026): a 256K context, multimodal for text and image (audio is limited to the smaller E2B/E4B/12B models). Open weights, licensed Apache-2.0 (per the model card on Hugging Face). On size, the card headlines 30.7B for the core language model and lists its vision encoder separately; our VRAM engine counts the full model at about 32.7B, which is the figure the computed pages linked below use. Like the rest of the family it uses mostly sliding-window attention (50 of 60 layers), keeping the KV cache modest even at long context.

If 24GB is more than you want to spend, the Gemma 4 26B-A4B mixture-of-experts model gives you similar-era Gemma capability on a 16GB card, and Gemma 4 12B drops to a 12GB card.

The way to think about the 31B is that it's the family's "no compromises on a single card" option. It's dense, so every parameter contributes to every token — you're not trading breadth for speed the way you are with the MoE sibling — and the sliding-window attention keeps the KV cache small enough that the 256K context is usable rather than theoretical. The cost of all that is VRAM: a ~32.7B-parameter model simply needs a 24GB card to live on comfortably at a good quantisation. If you have that card, this is the most capable Gemma you can run locally; if you don't, one of the smaller two is the honest choice, not this one forced through heavy offload.

How much VRAM Gemma 4 31B needs

Computed by our VRAM engine at 16K context (full table on cost to run Gemma 4 31B):

At ~20.5 GB, Q4_K_M leaves a little headroom on a 24GB card for context. Q6 and Q8 push past 24GB, so those are two-card or rented-bigger-card territory.

The quant labels decide everything here, so they're worth a sentence. Q4_K_M compresses each weight to about four bits, which is why a ~32.7B model lands near 20 GB rather than the ~63 GB of the full weights — and it holds quality remarkably well, which is why it's the standard recommendation. On a 24GB card that's the level to run; Q6 and Q8 are sharper still but need more VRAM than a single consumer card has, so they belong on a second GPU or a rented big one. And because Gemma's sliding-window attention keeps the KV cache small, the 256K context barely adds to these figures, so you can use genuinely long prompts on the 31B without watching your VRAM climb toward the ceiling.

Which GPU runs it

A 24GB card at Q4_K_M — an RTX 3090 or 4090 — is the model's natural home. See best GPU for Gemma 4 31B for the live-price ranking, including the cheapest 24GB options.

Referral link

Rent a 24GB card for Gemma 4 31B

A 24GB RTX 3090 was listing from about $0.139/hr on Vast.ai's spot market (26 Sep 2026) — enough for Gemma 4 31B at Q4_K_M. Prices move; check the live figure before you rent.

See live 24GB GPU pricesCompare live prices first →

Referral link — we may earn a commission at no cost to you. Prices on our compare page are the provider's live figures, cheapest first; this never changes the ranking.

How to run it

ollama run gemma4:31b

As with the 12B, the bare gemma4 tag pulls the small e4b model, so name the size. gemma4:31b pulls a Q4_K_M build matching the ~20.5 GB above; for higher quality you'll need a bigger card or a rented one.

Who should skip it

Skip Gemma 4 31B if you don't have 24GB — Gemma 4 26B-A4B (16GB) or Gemma 4 12B (12GB) are the honest alternatives, not this one squeezed with heavy offload. Skip it if you want speed above all: a 30B dense model activates every parameter, so it's slower than an MoE of similar size. And skip Q8/BF16 unless you're renting — they don't fit a single consumer card.

For the family at a glance and the 12B/26B alternatives, start with Gemma 4 12B, then compare cards on best GPU for local AI.