run Mistral Small 24B locally in 2026: VRAM, which GPU, and how fast

Mistral Small 24B is one of the best models you can run on a single consumer GPU: a 24-billion-parameter dense model that punches well above its size, and at 4-bit quantisation it fits a 24GB card with room to spare. Here's exactly what it needs — VRAM by quant, which GPU, what speed to expect — and the cheapest way to run it, whether you own a card or rent one. Part of our local-AI hardware guides.

VRAM by quantisation

Which GPU to run it on

For the best value, a 24GB card — a used RTX 3090 or an RTX 4090 — runs Q4_K_M with about 3.6GB of headroom for context, at about 38 tokens/second with llama.cpp, or roughly 85 with vLLM and AWQ INT4 (Markaicode, citing community benchmarks). A 16GB card (like a 5060 Ti 16GB) fits Q4 but with little room for long context. For Q8 or a big context window you'll want 32GB+ or a data-centre card. See the model-specific pick on our best GPU for Mistral Small 24B page and match any model to a card with the cost-to-run tool.

Or run it in the cloud

If you don't own a 24GB card, renting one is the cheapest way to try Mistral Small 24B at full speed: a 24GB GPU is about $0.07–0.27/hour as of Nov 12, 2026, so an evening of use costs a dollar or two and you buy nothing. It's also the honest way to test whether the model is worth a hardware purchase before you commit — rent first, then decide. Our how to run Mistral locally guide covers the software side.

Mistral Small 24B is the rare model that's genuinely great on one consumer card: Q4 on a 24GB GPU is all most people need. Own a card or rent one cheaply; see the best GPU for Mistral Small 24B and how to run Mistral locally.