Gemma 4 26B-A4B locally: the Gemma that runs fast on 16GB

Gemma 4 26B-A4B is the family's clever middle option: a mixture-of-experts model that stores about 26B parameters but only activates 3.8B per token. You get closer-to-large capability with the speed of a much smaller model, and — unusually for a 26B — it fits a 16GB card at Q4.

What Gemma 4 26B-A4B actually is

From Google's model card (read 26 Sep 2026): multimodal for text and image, a 256K context, and — per the card's MoE table — 3.8B active parameters out of 8 active experts of 128 (plus 1 shared). Open weights, licensed Apache-2.0 (per the model card on Hugging Face). On total size, the card headlines 25.2B for the core model; our VRAM engine counts the full model at about 26.5B, which is what the computed pages below use. Its config.json shows the MoE structure and Gemma's usual sliding-window attention (25 of 30 layers), so both compute and KV cache stay lean. The "A4B" in the name is those active parameters — 3.8B per token.

The same MoE caveat applies as for any mixture-of-experts model: 3.8B active is about speed, not storage. All ~26.5B parameters load into VRAM (or offload to RAM). The good news is that quantised to Q4, that total still lands around 16 GB.

That combination is what makes this the interesting one in the family. A dense model with genuinely strong capability usually asks for a 24GB card; here you get a ~26B-parameter model that runs on a mainstream 16GB card at Q4, and runs quickly because only 3.8B parameters fire per token. It's the closest thing Gemma 4 has to a "punches above its VRAM" pick. The trade-offs are the ones every MoE carries: the 16GB fit is real but tight, so long multimodal prompts want more headroom, and the storage is a 26B's storage even though the speed is a 4B's speed.

How much VRAM Gemma 4 26B-A4B needs

Computed by our VRAM engine at 16K context (full table on cost to run Gemma 4 26B-A4B):

At ~16.1 GB, Q4_K_M is a tight-but-real fit for a 16GB card (leave context modest). A 24GB card runs it with comfortable headroom, and if your GPU is smaller, MoE offload will fit it on 12GB at the cost of speed — see the MoE offload guide.

Because it's a mixture of experts, the same offload trick that helps the big Qwen applies here: if 16GB feels tight, push some experts to system RAM and the model still runs, just more slowly. But the nicer story is that you rarely need to. At Q4 this ~26B model already fits a mainstream 16GB card, which is the entire reason to reach for it over a dense model of similar strength — you get the capability without the 24GB entry ticket, and the 3.8B active parameters keep it quick while you're at it.

Which GPU runs it

A 16GB card is the value pick at Q4 — an RTX 4060 Ti 16GB or 5060 Ti 16GB. A 24GB card gives room for Q6 and long context. See best GPU for Gemma 4 26B-A4B for the ranked live prices.

Referral link

Rent a 16GB or 24GB card

A 16GB RTX A4000 was listing from about $0.072/hr, and a 24GB RTX 3090 from about $0.139/hr, on Vast.ai's spot market (26 Sep 2026). Prices move; check the live figure before you rent.

See live GPU pricesCompare live prices first →

Referral link — we may earn a commission at no cost to you. Prices on our compare page are the provider's live figures, cheapest first; this never changes the ranking.

How to run it

ollama run gemma4:26b         # the 26B-A4B MoE
# smaller GPU? offload experts with llama.cpp:
llama-server -hf <gemma4-26b-a4b-GGUF> --n-cpu-moe 16

As with the other Gemmas, name the size — the bare gemma4 tag pulls the small e4b. gemma4:26b gives the MoE model; --n-cpu-moe offloads idle experts to RAM if your card is under 16GB.

Who should skip it

Skip Gemma 4 26B-A4B if you have a 24GB card and want maximum quality — a dense Gemma 4 31B activates every parameter for a similar VRAM budget. Skip it if you're on a 12GB card and want simplicity; Gemma 4 12B runs there without offload. And treat the 16GB Q4 fit as real but tight — if you run long multimodal prompts, plan for a 24GB card.

For the dense alternatives, compare with Gemma 4 31B and Gemma 4 12B; for the offload mechanics, see the MoE offload guide.