Start with the number that should reframe how you think about local AI: Qwen3.6-27B, a free open-weight model that fits on a single 24GB graphics card, scores 77.2 on SWE-bench Verified — within about four points of Claude Opus 4.6 (80.8), a frontier model you rent by the token. That's genuinely remarkable. It's also not the whole story, because there is no single 'best' 24GB model in 2026. There's a clean three-way split, and picking right depends entirely on what you're doing.
77.2 SWE-bench
Qwen3.6-27B
coding king; ~15-17GB at Q4
76.9 MMMU-Pro
Gemma 4 31B
vision + general; ~18-21GB at Q4
~225 tok/s
gpt-oss-20b
raw speed; ~12GB, long context
fit 24GB
All three
at Q4 — this is a VRAM game
The three-way split, decided for you
Coding & agents → Qwen3.6-27B (Q4_K_M, ~15-17GB). Best-in-class open coding model, leaves generous room for a big context window. The catch: it leans on chain-of-thought reasoning tokens, so it 'thinks' before it answers — accurate, not instant.
Vision & general use → Gemma 4 31B (Q4_K_M, ~18-21GB). Real multimodal image understanding and a higher general-composite score, but at ~18-21GB of weights your context headroom on a 24GB card gets tight — plan on reduced context or KV-cache quantization.
Speed & long documents → gpt-oss-20b (~12GB). OpenAI's open MoE runs about 225 tok/s on an RTX 4090 and leaves the most room for context. It won't match Qwen's coding ceiling, but for fast general work it's the one that feels instant.
VRAM footprint at Q4 (weights only — leave headroom for context)
most context headroom on 24GB
comfortable on 24GB
tight — watch your context length
The hardware truth: 3090 vs 4090 vs 5080 barely matters for what fits
Here's the part the GPU-upgrade crowd hates to hear. What model *fits* is decided almost entirely by VRAM capacity, not by which generation of card you own. A five-year-old used RTX 3090 has the same 24GB as a 4090 or a 5080, so it runs the exact same models. The newer cards win on tokens-per-second, not on capability. If your goal is running these models at all, a used 3090 is the best dollar-per-gigabyte-of-VRAM buy in local AI — and it's why 'just get more VRAM' beats 'get a faster GPU' almost every time in this space.
Two things that quietly change your results
First, the gpt-oss '128k context trap': the model advertises a 128k context window, but actually setting it that high on a memory-limited card pushes the KV cache into system RAM over PCIe, and speed collapses — one tester watched it fall to ~9 tok/s. Keep context to what your VRAM genuinely holds (around 32k is comfortable on 24GB) and it stays fast. Second, your inference engine now matters as much as your model: Ollama's July 2026 v0.32.0 update added multi-token prediction that runs Gemma 4 roughly 90% faster on average with no config change and identical output. The same hardware does more work than it did a month ago — check you're on the current build before blaming your card.
The honest bottom line: stop hunting for the one 'best' local model. Match the model to the job — Qwen to code, Gemma to see, gpt-oss to fly — make sure it fits your 24GB with room for context, and keep your runtime current. That trio, on a card you may already own, is a genuinely capable local-AI setup in 2026.
