Why a model that “fits” sometimes doesn't
Three things take up memory when you run a language model locally. The weightsare the model itself, and their size is set by how many parameters it has and how aggressively it's quantised — that number doesn't change no matter how you use the model. The KV cacheis the model's working memory for the conversation so far, and it grows linearly with context length. The overhead is the fixed cost of the runtime and its buffers.
The weights get all the attention because they're the big number, but they're also the predictable one. The KV cache is the term that surprises people. At a short context it's a rounding error next to the weights. At 32k or 128k tokens it can rival the weights or exceed them — which is why a model you ran happily at 2k context suddenly won't load at 32k on the same card.
The default above shows Qwen3 32B at Q4_K_M on a NVIDIA GeForce RTX 4090: about 18.5 GB of weights and 2.5 GB of KV cache at 16k context, which fits in 24GB. Change any control and the numbers recompute from the model's own configuration file.
What the tokens/sec estimate means
When a configuration fits and we have a verified memory-bandwidth figure for the card, the tool shows a rough generation speed. It's an estimate from first principles, not a benchmark: generating a token means reading the active weights out of memory once, so throughput is set by memory bandwidth. Real speed depends on your runtime, batch size and kernels, and we don't run our own hardware tests — see our editorial policy. For mixture-of-experts models we only show a figure when we can account for the active-parameter count, because scoring an MoE model on its total size would badly understate it.
Once you've found a configuration that works, the cost-to-run pages show the full quantisation ladder for each model and what renting the hardware costs by the hour.