If your local model feels sluggish, it's usually one of a few fixable things. Here's how to speed up local LLM inference, from the one change that matters most to the fine-tuning.
The single biggest thing: make sure the whole model fits in VRAM. If any of it spills to system RAM, generation slows to a crawl — this is the #1 cause of slow local AI, and fixing it (by using a smaller model or more aggressive quantization) can take you from a few tokens per second to dozens. After that, the wins are: run 4-bit quantization, enable flash attention, and for maximum speed on NVIDIA, switch to a faster engine like [EXL2](/gguf-vs-exl2-quantization-explained-2026) or vLLM. Here's the practical, ordered checklist to boost your [tokens per second](/what-are-tokens-in-ai-explained-2026).
The one fix that matters most
Before any tweak, check the thing that dominates everything else: is the entire model loaded into VRAM? When a model fits fully on the GPU, it runs fast. When it doesn't, part of it lives in system RAM and the GPU has to shuttle data across the slow PCIe bus for every token — and speed falls off a cliff, from dozens of tokens per second to single digits. So if your model feels sluggish, the first question isn't 'what setting do I change', it's 'does this model actually fit my [VRAM](/how-much-vram-do-you-need-to-run-ai-models-2026)?' The fix is to either run a smaller model, use more aggressive [quantization](/what-is-quantization-llm-explained-2026) (drop from Q5/Q8 to Q4 to shrink it), or shorten your context (which frees VRAM). Getting the model fully on the GPU is worth more than every other speed trick combined — check what fits in the VRAM calculator first.
The #1 speed fix: get the whole model into VRAM. Spilling to system RAM is what makes local AI crawl. · Unsplash
The rest of the speed toolkit
Once the model fits, these stack on top. Quantization: running at 4-bit (Q4_K_M) is both smaller and faster than higher precision, with near-zero quality loss — it's the default for a reason. Flash attention: a more efficient way to compute attention that speeds up generation and reduces memory use; most tools (Ollama, LM Studio, llama.cpp) support enabling it, and it's usually a simple flag or setting. A faster engine: llama.cpp/Ollama are the easy, universal choice, but if you have an NVIDIA card and want maximum speed, [EXL2 via ExLlamaV2](/gguf-vs-exl2-quantization-explained-2026) is meaningfully faster, and vLLM is the throughput king for serving. Shorter context: since a long context window uses more memory and compute per token, keeping context tight helps speed. And the honest hardware truth: if you've done all this and still want more, the answer is a [faster GPU with enough VRAM](/best-gpu-for-local-ai-2026) — memory bandwidth and compute set the ceiling, and a used RTX 3090 or newer card raises it.
Quick answers
Why is my local AI so slow?
The most common reason is that the model doesn't fully fit in your GPU's VRAM. When part of a model spills into system RAM, the GPU must shuttle data across the slow PCIe bus for every token, and generation speed collapses from dozens of tokens per second to single digits. Fix it by running a smaller model, using more aggressive quantization (Q4 instead of Q5/Q8) to shrink it, or shortening your context to free VRAM. Getting the entire model onto the GPU is by far the biggest speed factor — check what fits before changing other settings.
How do I increase tokens per second on a local LLM?
In order of impact: first, make sure the whole model fits in VRAM (the biggest factor). Then use 4-bit quantization (Q4_K_M), which is smaller and faster with minimal quality loss. Enable flash attention, a more efficient attention algorithm most tools support. For maximum speed on NVIDIA, switch from llama.cpp to EXL2 (via ExLlamaV2) or use vLLM for serving. Keep your context length tight, since long context slows generation. If you've done all that, a faster GPU with more VRAM is the hardware upgrade that raises the ceiling.
Does quantization make local AI faster?
Yes. Quantization (like 4-bit Q4_K_M) shrinks the model, so more of it fits in VRAM and less data moves around, which speeds up inference — while keeping quality nearly identical to the full-precision model. It's both a memory and a speed win, and it's the standard for local AI. If a model is slow because it barely fits or spills to system RAM, dropping to a more aggressive quantization (e.g. from Q5 or Q8 to Q4) often provides a large speed boost by getting the whole model onto the GPU.
The local-AI speed formula: fit the model in VRAM first, then quantize to Q4, enable flash attention, and use a fast engine. Check what fits in the VRAM calculator, understand quantization and formats, and if hardware is the limit, see the best GPU for local AI.
