kv cache

1 article · newest first

Hand-drawn editorial illustration of a hand pressing down the lid of a suitcase packed so full of paper cards that a fan of coral cards spills out, on a near-black background in indigo-violet and off-white

Out of memory at long context? Shrink the KV cache with q8_0 (llama.cpp and Ollama)

The model fits. Then you paste in a long document and the GPU runs out of memory. The culprit is the KV cache, it has a formula you can do on a napkin, and two settings cut it by roughly half or three quarters.

Aliteq Team · 14h ago · 9 min