What is the best local LLM for a 32GB RTX 5090? Qwen3.6 35B-A3B, and four more that fit

The best local LLM for a 32GB RTX 5090 is Qwen3.6 35B-A3B for most people: at the Q5_K_M quant with the 128K tokens of context Qwen itself recommends, it needs about 27.1 GB of the card's 32 GB by our VRAM engine. Want one dense model for everything? Qwen3.8 27B at Q6_K. Working with images or scans? Gemma 4 31B at Q5_K_M. Need raw speed? gpt-oss-20b, which llama.cpp's own benchmark clocks at about 282 tokens a second on a 5090. A 70B model still doesn't fit. Checked against each model's card and file sizes on November 3, 2026.

This page answers one question, which card owners ask in every local-AI forum: I have 32GB, what should I actually run? We read every model's card and config on Hugging Face (first on October 4, again on November 3, 2026), ran the numbers through our VRAM engine and checked them against the real GGUF file sizes. We have not loaded these models on a 5090 ourselves. Except where a source measured something, every figure below is arithmetic on published specs. If your card is smaller, start with our 24GB guide, 16GB guide, 12GB guide or 8GB guide.

Which local LLMs fit a 32GB RTX 5090?

Five models, each best at one job. The RTX 5090 has 32 GB of GDDR7, per NVIDIA's spec table, against 24 GB on the RTX 4090. That 8 GB is what decides the quant and the context below.

How do we work out what fits?

VRAM is three things added up: the model's weights, the context cache and a runtime overhead. Weights depend on the quant, which is how many bits each number is squeezed into. The cache grows with every token of conversation. The overhead is a flat 0.78 GB in our engine.

weights  = parameters x bits_per_weight / 8
cache    = 2 x cache layers x KV heads x head dimension x tokens x 2 bytes
total    = weights + cache + 0.78 GB overhead

Qwen3.6 35B-A3B, Q5_K_M, 128K tokens:
23.8 + 2.5 + 0.78 = 27.1 GB  (85% of a 32GB card)

Many 2026 models only keep a growing cache in some layers, so we count those layers alone. Gemma 4 also keeps a small fixed cache in its sliding-window layers: about 0.8 GB for the 31B and 0.2 GB for the 26B-A4B. We add it. You can run the same math on any model, any card, with our VRAM calculator.

What is the best LLM for coding on an RTX 5090?

Qwen3.6 35B-A3B, and it is the model a 32GB card suits best. It is a Mixture-of-Experts model: 35B parameters stored, about 3B used per token, so it is quick for its size. At Q5_K_M with 128K tokens of context it needs about 27.1 GB, which sits comfortably on a 5090 and does not fit comfortably on a 24GB card.

The 128K figure comes from Qwen, not us. The model card says: "we advise maintaining a context length of at least 128K tokens to preserve thinking capabilities."

On a 24GB card, holding that 128K means dropping to IQ4_XS, a slimmer 4-bit quant. Ollama's qwen3.6:35b-a3b download is listed at 23GB, which leaves a 24GB card almost nothing for context. On a 5090 the same download leaves room to spare, or you can step up to Q5_K_M from Hugging Face (see the Ollama section below).

One honest caveat on the file: Unsloth's UD-Q5_K_M file is 24.64 GB, a little above our 23.8 GB estimate, because their "dynamic" quants keep some layers at higher precision. With that file the 128K total is about 27.9 GB, still under the comfortable line. Add Unsloth's vision file (0.84 GB) and you are at about 28.7 GB, right at the 90 percent line, so trim the context a little if you also feed it images.

Qwen's card reports 73.4 on SWE-bench Verified for this model, run on what it calls an "Internal agent scaffold". That is the vendor's number, not ours. Its cache costs only about 0.6 GB per 32K tokens, because just 10 of its 40 layers keep one. Every quant is on its cost-to-run page.

If you want a coding specialist instead, Qwen3-Coder 30B-A3B fits at Q6_K with about 51K tokens. Its cache costs about five times as much per token, so it runs out of room sooner.

What is the best all-round model for 32GB of VRAM?

Qwen3.8 27B, the best dense pick for 32GB. At Q6_K, a high-quality quant, it needs 24.0 GB with 32K tokens and 26.0 GB with 64K, with room for about 108K. That is the big step up from 24GB, where the same model is comfortable only at Q5_K_M with under 40K tokens.

Dense means every parameter works on every token. You trade speed for steadier quality compared with a Mixture-of-Experts model. Qwen's card calls it "a native vision-language model that understands images and videos". The card reports 61.7 on SWE-bench Pro, up from 53.5 for Qwen3.6 27B; it says the models were "evaluated with the Claude Code harness". Again, the vendor's figures.

The real files agree with our math. Unsloth's UD-Q6_K file is 20.47 GB of weights against our 21.2 GB estimate, so here the engine errs on the safe side. Its cost-to-run page has every quant.

Which model should you run for images and documents?

Gemma 4, when you feed the model pictures, scans or screenshots. On 32GB the dense Gemma 4 31B runs at Q5_K_M with 32K tokens in about 24.7 GB, with room for about 84K tokens of text. Add the 1.12 GB vision file and you still have room for about 71K. On a 24GB card it drops to IQ4_XS.

Can it run at Q6_K? For text, yes: about 27.9 GB at 32K tokens, 87 percent of the card, with a ceiling around 43K tokens. Add the 1.12 GB vision file and it reaches about 29.0 GB, roughly 91 percent, just past our comfortable line. That is why we pick Q5_K_M for image work. (Our October version said Q6_K was "a step too far" at 29.0 GB; that figure used a parameter count about 1.4B too high. Hugging Face lists the model at 31.27B.) Google's card scores the 31B above the 26B-A4B on its image tests, for example 76.9 versus 73.8 percent on MMMU Pro.

The Gemma 4 26B-A4B is the other route. It is a Mixture-of-Experts model with 3.8B active parameters, and on 32GB it fits at Q8_0, which our engine calls effectively lossless: about 27.1 GB at 32K tokens, with text context up to about 119K. With the vision file loaded, plan on about 60K. It is also the one 32GB pick you can pull straight from Ollama at that quality: gemma4:26b-a4b-it-q8_0 is listed at 28GB.

For setup help, read how to run Gemma 4 31B locally.

Which local LLM is fastest on an RTX 5090?

gpt-oss-20b, and it is the only pick here with a published speed figure on this exact card. llama.cpp's gpt-oss guide benchmarks it on an RTX 5090 at about 282 tokens a second for generation (its tg128 test) and about 9,800 tokens a second for reading a 2,048-token prompt. Those are llama.cpp's measurements, not ours.

The same guide lists its memory: 14.9 GB at 8K tokens, 15.5 GB at 32K and 17.9 GB at the full 128K. OpenAI ships the model in one format, MXFP4, and its card says it runs "within 16GB of memory."

Here is the honest catch: you do not need a 5090 for this one. 17.9 GB fits a 24GB card with room to spare. Buy 32GB for the first three picks, then keep gpt-oss-20b loaded beside them as the quick model. Our best GPU for gpt-oss page covers it in full.

What does 32GB buy over a 24GB card?

Quality and memory, not a bigger model class. The same models run on a 24GB card, just at slimmer quants and with less context. The gain is largest for the newer hybrid models, which spend so little memory per token that the spare gigabytes turn into 100K tokens or more.

Read the table as a choice, not a rule. Gemma 4 31B shows the trade best: on 32GB you can take Q6_K with about 43K tokens, or stay at Q5_K_M and get nearly twice the context. Context figures are text only; images take a share. If you are unsure which quant to trust, our quantization guide explains the ladder.

Why do the older 32B models feel cramped on 32GB?

Because their cache is huge. A 32GB card sounds made for "32B" models, but the 2025 dense ones are a poor fit. Qwen3 32B keeps a cache in all 64 layers, so 32K tokens costs it 8 GB. On a 5090 it is comfortable only at Q4_K_M, and only up to about 38K tokens.

Compare Qwen3.8 27B: 2 GB for the same 32K tokens, a quarter of the older model's bill. That is why we would not buy a 5090 to run a 2025-era 32B model. The newer models are the ones that make the extra memory pay. Our context window explainer covers why the cache grows at all.

How do you run these in Ollama on a 5090?

Most of Ollama's default tags are 4-bit builds, and its next step up is usually Q8_0, which is too big for the 27B to 35B models on 32GB. On November 3, 2026 Ollama listed qwen3.6:35b-a3b at 23GB and qwen3.6:35b-a3b-q8_0 at 39GB, qwen3.8:27b at 18GB and qwen3.8:27b-q8_0 at 30GB, gemma4:31b at 19 to 20GB and gemma4:31b-it-q8_0 at 34GB. Only gemma4:26b-a4b-it-q8_0 (28GB) is a top-quality build that fits.

For the Q5_K_M and Q6_K picks, pull the GGUF from Hugging Face through Ollama. Hugging Face's docs give the pattern ollama run hf.co/{username}/{repository}:{quantization}, for example:

ollama run hf.co/unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q5_K_M
ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q6_K

Then check the context length Ollama is using for the model and raise it if needed: Qwen3.6 wants at least 128K.

What still does not fit in 32GB?

The 70B class does not fit a single 32GB card at a sensible quant. Llama 3.3 70B needs 39.8 GB for weights alone at Q4_K_M, and 43.1 GB with a short 8K context. Even Q3_K_M weights, at 32.1 GB, fill the whole card before any context.

You can squeeze a 70B in at 2-bit quants, but our engine's note on Q2_K is blunt: "Almost always better to run a smaller model at Q4_K_M instead." We agree. Bigger Mixture-of-Experts models are further out: Qwen3-Next 80B needs about 45 GB of weights at Q4_K_M, and OpenAI says gpt-oss-120b fits "a single 80GB GPU."

There is one middle road. llama.cpp can keep a big MoE model's experts in system RAM and the rest on the card. Its gpt-oss guide says: "You can get for example about 30 t/s at zero context on a 5090 with --n-cpu-moe 21." The 120b model is about 61 GB, so you need enough system RAM to hold the rest; our MoE guide puts it at 64 GB. How the flag works: what --n-cpu-moe actually does. Otherwise, see two cards versus one, or rent.

Should you buy an RTX 5090 for this, or rent one first?

Rent first if you are not sure. You need 32GB of VRAM only if the first three picks are your daily work, and an hour on a rented 5090 tells you whether Qwen3.6 or Qwen3.8 does your job before you spend on a card. Whether a 5090 is worth buying for you is a separate question, which we answer in is the RTX 5090 worth it for local AI and the broader best GPU for local AI guide.

On November 3, 2026 (20:20 UTC) our tracker showed single RTX 5090 rentals on Vast.ai from $0.37 an hour on the spot market (median $0.56 across 59 offers) and from $0.40 an hour on demand (median $0.71). Runpod listed the RTX 5090 at $1.19 an hour on demand. Prices move by the hour: /cloud-gpu shows today's figures, and our rent vs buy breakdown shows where owning wins.

What do these numbers not tell you?

They do not rank model quality. The benchmark figures above are each vendor's own, and we did not run any. They tell you speed for gpt-oss-20b only, because llama.cpp measured it; for the other four, tokens per second depends on your engine, card and quant, and we did not measure it.

They also assume an fp16 cache and one user. An 8-bit cache roughly halves the cache figures. Each extra live conversation adds its own cache. Images add a separate vision file, about 0.8 to 1.1 GB in the Unsloth sets we checked. Leave headroom rather than aiming for 100 percent.