run Mistral Small 24B locally in 2026: VRAM, which GPU, and how fast

Mistral Small 24B fits a single 24GB card at Q4 (~14GB). VRAM by quant, which GPU, realistic speeds by runtime, and the cheapest way to run it — own or rent.

TensorSep 24Updated Sep 247 min readWeb story

Flat illustration of a person running a local AI chat assistant on a laptop with a single graphics card, on a teal background

This post contains affiliate links. If you buy through them, Aliteq may earn a commission — at no extra cost to you. Prices verified at publish time.

Mistral Small 24B is one of the best models you can run on a single consumer GPU: a 24-billion-parameter dense model that punches well above its size, and at 4-bit quantisation it fits a 24GB card with room to spare. Here's exactly what it needs — VRAM by quant, which GPU, what speed to expect — and the cheapest way to run it, whether you own a card or rent one. Part of our local-AI hardware guides.

Mistral Small 24B (v3.2) is a 24B dense model. At Q4_K_M it needs about 14–15GB of VRAM (a ~14GB GGUF), so it fits a 16GB card tightly and runs comfortably on a 24GB card like an RTX 3090 or 4090 with headroom for a long context. Published community benchmarks compiled by Markaicode (mid-2026) put an RTX 4090 at about 38 tokens/second with llama.cpp at Q4_K_M, or about 85 with vLLM and AWQ INT4 quantisation. Q8 needs ~25GB and full BF16 about 55GB. If you don't own a capable card, you can rent a 24GB GPU from about $0.14/hour.

Q4_K_M: ~14–15GB — fits a 16GB card (tight), comfortable on 24GB.
Q8_0: ~25GB — needs a 24GB card at minimum, better on 32GB+.
Full BF16: ~55GB — an A100 80GB or two 24GB cards.
Speed on a 4090: ~38 tok/s (llama.cpp, Q4_K_M) or ~85 tok/s (vLLM, AWQ INT4), per Markaicode's compilation of community benchmarks.
No capable GPU? Rent a 24GB card from ~$0.14/hr and run it in the cloud.

VRAM by quantisation

Mistral Small 24B — VRAM by quant

Q4_K_M

VRAM (approx): ~14–15GB
Fits: 16GB card (tight), 24GB (comfortable)
Trade-off: Best value; tiny quality cost

Q6_K

VRAM (approx): ~19–20GB
Fits: 24GB card
Trade-off: Higher quality

Q8_0

VRAM (approx): ~25GB
Fits: 24GB (tight) / 32GB
Trade-off: Near-lossless

BF16 (full)

VRAM (approx): ~55GB
Fits: A100 80GB / 2×24GB
Trade-off: No quality loss, rarely needed

	VRAM (approx)	Fits	Trade-off
Q4_K_M	~14–15GB	16GB card (tight), 24GB (comfortable)	Best value; tiny quality cost
Q6_K	~19–20GB	24GB card	Higher quality
Q8_0	~25GB	24GB (tight) / 32GB	Near-lossless
BF16 (full)	~55GB	A100 80GB / 2×24GB	No quality loss, rarely needed

No 24GB card? Rent one from pennies an hourReferral link

Rent a 24GB card

Which GPU to run it on

For the best value, a 24GB card — a used RTX 3090 or an RTX 4090 — runs Q4_K_M with about 3.6GB of headroom for context, at about 38 tokens/second with llama.cpp, or roughly 85 with vLLM and AWQ INT4 (Markaicode, citing community benchmarks). A 16GB card (like a 5060 Ti 16GB) fits Q4 but with little room for long context. For Q8 or a big context window you'll want 32GB+ or a data-centre card. See the model-specific pick on our best GPU for Mistral Small 24B page and match any model to a card with the cost-to-run tool.

Or run it in the cloud

If you don't own a 24GB card, renting one is the cheapest way to try Mistral Small 24B at full speed: a 24GB GPU is about $0.14–0.34/hour as of 24 Sep 2026, so an evening of use costs a dollar or two and you buy nothing. It's also the honest way to test whether the model is worth a hardware purchase before you commit — rent first, then decide. Our how to run Mistral locally guide covers the software side.

Referral link

Run Mistral Small 24B on a rented 24GB card

A 24GB card runs it at Q4 comfortably — from about $0.14/hr on Vast.ai spot (24 Sep 2026). Pay pennies for an evening; buy nothing.

See cheap GPUs to run it

Compare live prices first →

Referral link — we may earn a commission at no cost to you. Prices on our compare page are the provider's live figures, cheapest first; this never changes the ranking.

Frequently asked

How much VRAM does Mistral Small 24B need?

About 14–15GB at Q4_K_M (a ~14GB GGUF), so it fits a 16GB card tightly and runs comfortably on a 24GB card with headroom for context. Q8 needs ~25GB and full BF16 about 55GB. For most people, Q4_K_M on a 24GB card is the sweet spot.

Can an RTX 4090 run Mistral Small 24B?

Yes, easily — a 24GB RTX 4090 runs Q4_K_M with roughly 3.6GB of headroom for context, at about 38 tokens/second with llama.cpp at Q4_K_M, or about 85 with vLLM and AWQ INT4, per community benchmarks compiled by Markaicode. A used RTX 3090 (also 24GB) runs it too, a bit slower. Both are ideal single-card homes for this model.

What's the cheapest way to run Mistral Small 24B if I don't have a good GPU?

Rent a 24GB card in the cloud — from about $0.14/hour on Vast.ai spot or $0.34/hour on Runpod as of 24 September 2026. You pay only while it runs, so trying the model costs a dollar or two, and you can decide whether it's worth buying hardware before you spend.

Referral link

Managed pods + serverless

Prefer a stable, managed box? Runpod runs a 24GB card at about $0.34/hr on-demand (24 Sep 2026).

Rent on-demand

Mistral Small 24B is the rare model that's genuinely great on one consumer card: Q4 on a 24GB GPU is all most people need. Own a card or rent one cheaply; see the best GPU for Mistral Small 24B and how to run Mistral locally.

cloud gpu mistral small 24b local ai vram gpu ai

Found this useful? Share it

Sources

Markaicode — Mistral Small 3.2 on RTX 4090 (community benchmarks, mid-2026)

Local AI & Automation Editor

Tensor

I'm US-based, I run more models at home than I'll admit to, and I've quantized more than I've finished reading about. I write about running AI on your own hardware and, lately, about what it costs a company to do the same — tokens per day, GPUs per month, and the GDPR questions nobody's sales deck answers.

Work out the hardware

VRAM calculator

See exactly which models fit your card, at any context length.

What it costs to run

Live rent-vs-buy economics for running a model on real hardware.

Best local LLM for 16GB

Exactly which models fit a 16GB card — and the tier to skip.

The Aliteq brief

The tech worth knowing — hardware, AI, gaming, deals. No spam, unsubscribe anytime.

Keep reading

Flat illustration of a person choosing which GLM model version fits their hardware, on a teal background

how to run GLM locally in 2026: which version, which GPU

GLM ties Kimi K3 at the top of the open-weight rankings, but 'run it locally' depends on which GLM — from the 106B Air that fits one card to the 744B GLM-5 that needs a 256GB Mac or the cloud. The honest map, with real numbers.

Tensor · Sep 24 · 8 min

Editorial infographic of three model badges converging on a single 24GB graphics card with a fork in the path, near-black background, indigo-violet with a coral accent

Which 24GB-class model should you run? Mistral Small 24B vs Qwen3.6 27B vs Gemma 4 31B

You've got a 24GB card and three excellent dense models that fit it. Here's how Mistral Small 24B, Qwen3.6 27B and Gemma 4 31B differ on VRAM, context and modality — and a clear pick for each kind of user.

Tensor · 6d ago · 7 min

Flat illustration of a relaxed person on a laptop with a small spark of AI, wondering if they can run AI without a graphics card

can you run your own AI without buying a graphics card? (yes — here's how)

You don't need a $1,000 graphics card to start using AI yourself. There are two honest paths — free on your own laptop, or a rented GPU for pennies an hour. Here's which is yours.

Tensor · Sep 23 · 6 min

Isometric diorama of a friendly glowing AI chip seated on a mid-range graphics card, near-black background, indigo-violet with a coral accent

Gemma 4 12B locally: the VRAM, the GPU, and one Ollama gotcha

Google's Gemma 4 12B is the multimodal open model that fits a mid-range card — if you pull the right Ollama tag. The honest VRAM math, the cheapest GPU that runs it, and how to start.

Tensor · 6d ago · 7 min