a free model on your 24GB GPU codes within 4 points of Claude Opus. meet Qwen3.6-27B

on a single 24GB card in 2026 there's no one 'best' — it's a clean three-way split: Qwen3.6-27B for coding, Gemma 4 for vision, gpt-oss-20b for raw speed. and a used RTX 3090 still competes, because this is a memory game, not a compute one.

TensorJul 19Updated Aug 57 min readWeb story

Ollama, the local-model runtime used to run open-weight LLMs on consumer GPUs

Start with the number that should reframe how you think about local AI: Qwen3.6-27B, a free open-weight model that fits on a single 24GB graphics card, scores 77.2 on SWE-bench Verified — within about four points of Claude Opus 4.6 (80.8), a frontier model you rent by the token. That's genuinely remarkable. It's also not the whole story, because there is no single 'best' 24GB model in 2026. There's a clean three-way split, and picking right depends entirely on what you're doing.

77.2 SWE-bench

Qwen3.6-27B

coding king; ~15-17GB at Q4

76.9 MMMU-Pro

Gemma 4 31B

vision + general; ~18-21GB at Q4

~225 tok/s

gpt-oss-20b

raw speed; ~12GB, long context

fit 24GB

All three

at Q4 — this is a VRAM game

The three-way split, decided for you

Coding & agents → Qwen3.6-27B (Q4_K_M, ~15-17GB). Best-in-class open coding model, leaves generous room for a big context window. The catch: it leans on chain-of-thought reasoning tokens, so it 'thinks' before it answers — accurate, not instant.

Vision & general use → Gemma 4 31B (Q4_K_M, ~18-21GB). Real multimodal image understanding and a higher general-composite score, but at ~18-21GB of weights your context headroom on a 24GB card gets tight — plan on reduced context or KV-cache quantization.

Speed & long documents → gpt-oss-20b (~12GB). OpenAI's open MoE runs about 225 tok/s on an RTX 4090 and leaves the most room for context. It won't match Qwen's coding ceiling, but for fast general work it's the one that feels instant.

VRAM footprint at Q4 (weights only — leave headroom for context)

gpt-oss-20b~12 GB

most context headroom on 24GB

Qwen3.6-27B~15-17 GB

comfortable on 24GB

Gemma 4 31B~18-21 GB

tight — watch your context length

The hardware truth: 3090 vs 4090 vs 5080 barely matters for what fits

Here's the part the GPU-upgrade crowd hates to hear. What model fits is decided almost entirely by VRAM capacity, not by which generation of card you own. A five-year-old used RTX 3090 has the same 24GB as a 4090 or a 5080, so it runs the exact same models. The newer cards win on tokens-per-second, not on capability. If your goal is running these models at all, a used 3090 is the best dollar-per-gigabyte-of-VRAM buy in local AI — and it's why 'just get more VRAM' beats 'get a faster GPU' almost every time in this space.

Two things that quietly change your results

First, the gpt-oss '128k context trap': the model advertises a 128k context window, but actually setting it that high on a memory-limited card pushes the KV cache into system RAM over PCIe, and speed collapses — one tester watched it fall to ~9 tok/s. Keep context to what your VRAM genuinely holds (around 32k is comfortable on 24GB) and it stays fast. Second, your inference engine now matters as much as your model: Ollama's July 2026 v0.32.0 update added multi-token prediction that runs Gemma 4 roughly 90% faster on average with no config change and identical output. The same hardware does more work than it did a month ago — check you're on the current build before blaming your card.

The honest bottom line: stop hunting for the one 'best' local model. Match the model to the job — Qwen to code, Gemma to see, gpt-oss to fly — make sure it fits your 24GB with room for context, and keep your runtime current. That trio, on a card you may already own, is a genuinely capable local-AI setup in 2026.

For a primary-source reference, see Google's official Gemma documentation: Google — Gemma.

local llm qwen3.6-27b gemma 4 gpt-oss 24gb gpu ollama rtx 3090

Found this useful? Share it

Sources

Local AI & Automation Editor

Tensor

I'm US-based, I run more models at home than I'll admit to, and I've quantized more than I've finished reading about. I write about running AI on your own hardware and, lately, about what it costs a company to do the same — tokens per day, GPUs per month, and the GDPR questions nobody's sales deck answers.

Work out the hardware

Best local LLM for 16GB

Exactly which models fit a 16GB card — and the tier to skip.

RTX 5090 vs two RTX 3090s

One fast card or two big ones — what the benchmarks really show.

RTX 5080 vs RTX 4090

Why 16GB loses to 24GB for local AI, newer card or not.

VRAM calculator

See exactly which models fit your card, at any context length.

The Aliteq brief

The tech worth knowing — hardware, AI, gaming, deals. No spam, unsubscribe anytime.

Keep reading

Flat illustration of a person at a PC running an AI assistant with a graphics card in the case, on a teal background

The Best GPU for Running Qwen3 Locally in 2026 (by Model Size)

Qwen3 isn't one model — it's a family from 4B to 235B, and the right GPU depends entirely on which you run. My VRAM-first map of every variant, the MoE trap that trips people up, and the used card I'd hand most people.

Tensor · Sep 22 · 8 min

Realistic painterly illustration of a large glowing processor die radiating soft light on a dark surface, deep indigo-violet with a warm coral highlight

Gemma 4 31B locally: the 24GB-card Gemma

Gemma 4 31B is the big dense Gemma — the one that asks for a 24GB card and rewards it. The VRAM per quant, the GPU that fits, and how to run it without pulling the wrong tag.

Tensor · 6d ago · 7 min

Isometric diorama of a friendly glowing AI chip seated on a mid-range graphics card, near-black background, indigo-violet with a coral accent

Gemma 4 12B locally: the VRAM, the GPU, and one Ollama gotcha

Google's Gemma 4 12B is the multimodal open model that fits a mid-range card — if you pull the right Ollama tag. The honest VRAM math, the cheapest GPU that runs it, and how to start.

Tensor · 6d ago · 7 min

Editorial infographic of a solid dense 27B model block fitting neatly inside a 24GB graphics card, near-black background, indigo-violet with a coral accent

Qwen3.6 27B locally: the all-dense 24GB Qwen

Qwen3.6's dense 27B is the no-fuss 24GB-card option — every parameter fires, nothing to offload, a real 256K context. The VRAM math, the GPU, and how it differs from the 35B-A3B MoE.

Tensor · 6d ago · 7 min