how to make your local AI faster — the settings that actually boost tokens per second

If your local model feels sluggish, it's usually one of a few fixable things. Here's how to speed up local LLM inference, from the one change that… Aliteq Ravi Malhotra · Hardware Editor

1. Fit the whole model in VRAM — the biggest factor. Spilling to system RAM is what makes local AI slow.

The speed checklist (in order of impact)

2. Use 4-bit quantization (Q4_K_M) — smaller model, more fits in VRAM, faster.

The speed checklist (in order of impact)

3. Enable flash attention — a more efficient attention algorithm, faster and lower memory.

The speed checklist (in order of impact)

4. Use a faster engine — EXL2 (NVIDIA) or vLLM beat llama.cpp on raw speed.

The speed checklist (in order of impact)

5. Shorten your context — long context uses more memory and slows generation.

The speed checklist (in order of impact)

6. Or just get more VRAM — a faster GPU with enough memory is the hardware fix.

The speed checklist (in order of impact)

Read the full story

Aliteq how to make your local AI faster — the settings that actually boost tokens per second Read the full story on Aliteq