Your local model types fast, so why does a long prompt sit there in silence first? That wait is prefill, the model reading your prompt, and it is a different speed from decode, the writing you watch. Here is how long each GPU makes you wait, why a 4090 reads prompts about twice as fast as a 3090, and what it costs to rent each one. Checked Nov 21, 2026.
Some links on this page are referral links: we may earn a commission, at no cost to you.
Share
Prefill is the model reading your prompt. Decode is the model writing its answer. They are two different speeds, and the spec sheets and most "tokens per second" numbers only show the second one. A GPU that writes 50 words a second can still sit silent for 20 seconds first, because before it writes anything it has to read every token you pasted in: the code, the document, the chat history. On hardware-corner.net's llama.cpp tables for Qwen3 14B at 4-bit, a 32,000-token prompt waits roughly 20 seconds on an RTX 3090, 10 on a 4090 and 8 on a 5090 before the first word appears. Long prompts are a prefill problem, and that changes which GPU is worth paying for.
What happens between Enter and the first word
You send a long prompt
Code, a document, the whole chat so far: 32,000 tokens
Prefill reads it
Every token goes through every layer in big batches and fills the KV cache. You see nothing yet
First word appears
Time to first token. About 20 s on a 3090, 10 on a 4090, 8 on a 5090 at 32K (Qwen3 14B Q4_K)
Decode writes the answer
One token at a time, each one reading all the weights and the whole KV cache
You send a long prompt
Code, a document, the whole chat so far: 32,000 tokens
Prefill reads it
Every token goes through every layer in big batches and fills the KV cache. You see nothing yet
First word appears
Time to first token. About 20 s on a 3090, 10 on a 4090, 8 on a 5090 at 32K (Qwen3 14B Q4_K)
Decode writes the answer
One token at a time, each one reading all the weights and the whole KV cache
Prefill reads every prompt token in large batches (llama.cpp default logical batch 2,048, physical 512). Decode then runs once per output token. Source: llama.cpp server and llama-bench READMEs, read Nov 21, 2026.
What is the difference between prefill and decode?
Prefill (llama.cpp calls it prompt processing, "pp", and Ollama prints it as "prompt eval rate") is the model reading your input. All the prompt's tokens are known up front, so the GPU can push thousands of them through each layer at once. That is matrix math at full tilt, so it is limited by how much compute the card has.
Decode (llama.cpp's "tg", text generation, and Ollama's "eval rate") is the model writing. It can only produce one token at a time, because each new token depends on the one before. For every single token the GPU has to read all of the model's weights from memory again, plus the cache of everything said so far. Very little math per byte read, so decode is limited by memory bandwidth.
That is why the two speeds don't move together. NVIDIA's whitepapers list 936 GB/sec of memory bandwidth on the RTX 3090 and 1,008 GB/sec on the RTX 4090, only 8% apart, so they write at similar speeds. The 4090 has far more compute, so it reads prompts about twice as fast.
Prompt reading speed (prefill), Qwen3 14B Q4_K at 16K context, tokens per second (hardware-corner.net, March 2026 tables)
RTX 3090 24 GB1,679 tok/s
RTX 4090 24 GB3,452 tok/s
RTX 5090 32 GB4,473 tok/s
Writing speed (decode), same model and context, tokens per second
RTX 3090 24 GB52.1 tok/s
RTX 4090 24 GB69.8 tok/s
RTX 5090 32 GB102.7 tok/s
Why does my GPU take so long to start answering?
Because your prompt is long and the wait grows faster than the prompt. Two things stack up. First, there are simply more tokens to read. Second, each token gets slower to read the deeper into the context it sits, because attention has to look back over everything before it. On hardware-corner.net's RTX 3090 table, Qwen3 14B reads 2,459 tokens a second at 4K of context, 1,679 at 16K, 1,176 at 32K and 734 at 64K.
We turned those rates into a wait time by adding up the time for each slice of the prompt at the speed measured for that depth. That is our arithmetic, not a measurement, and real runs vary with the build and settings, but it lines up with what people post.
Roughly how long until the first word, Qwen3 14B Q4_K (our arithmetic on hardware-corner.net's prefill tables)
4K tokens (a long chat message)
Prompt length
1.7 s
RTX 3090
0.8 s
RTX 4090
0.6 s
16K tokens (a few source files)
Prompt length
7.8 s
RTX 3090
3.7 s
RTX 4090
2.9 s
32K tokens (a big file or a long session)
Prompt length
19.7 s
RTX 3090
9.6 s
RTX 4090
7.6 s
64K tokens (a small codebase)
Prompt length
56 s
RTX 3090
28 s
RTX 4090
23 s
128K tokens
Prompt length
not in the table
RTX 3090
not in the table
RTX 4090
78 s
Prompt length
RTX 3090
RTX 4090
RTX 5090
4K tokens (a long chat message)
1.7 s
0.8 s
0.6 s
16K tokens (a few source files)
7.8 s
3.7 s
2.9 s
32K tokens (a big file or a long session)
19.7 s
9.6 s
7.6 s
64K tokens (a small codebase)
56 s
28 s
23 s
128K tokens
not in the table
not in the table
78 s
A bigger model makes it worse. Qwen3 32B at Q4_K reads at 768 tokens a second on a 3090 at 16K, 1,707 on a 4090 and 2,077 on a 5090, so a 16K prompt waits roughly 17, 8 and 6.5 seconds.
@kyzoroX posted the same effect from their own RTX 5090 on Nov 20, 2026: with Qwen3.8-27B, a 6K-token prompt started answering in 2.0 seconds and a 63K-token prompt in 26.4, while the writing speed barely moved (59 vs 51 tokens a second). Their 4K-prompt reads were 1,296 tokens a second on a 3090, 3,004 on a 4090 and 3,830 on a 5090, a 2.3x gap between the two older cards. That is their test, not ours, and it is close to hardware-corner.net's 1.9 to 2.25x.
Is a 4090 worth twice the rent of a 3090 for long prompts?
For long prompts, yes, roughly at cost. On Vast on Nov 21, 2026 (15:20 UTC, single-GPU on-demand offers), an RTX 3090 rented from $0.09 an hour, typical $0.21, across 139 offers, and an RTX 4090 from $0.28, typical $0.47, across 205. That is about 2.2 times the typical rent for 1.9 to 2.25 times the prefill speed, so reading a million prompt tokens costs about the same on either card. The 4090 just gets it done in half the time.
For writing, the 3090 is the cheaper card per word. The 4090 writes only 14% faster on Qwen3 32B (34.4 vs 30.3 tokens a second at 16K), so a million output tokens cost roughly twice as much on it.
Rental cost per million tokens, Qwen3 32B Q4_K at 16K context (hardware-corner.net speeds x typical Vast price, Nov 21, 2026)
RTX 3090 24 GB
Card
$0.21/hr (from $0.09)
Typical rent
about $0.08
Reading 1M prompt tokens
about $1.93
RTX 4090 24 GB
Card
$0.47/hr (from $0.28)
Typical rent
about $0.08
Reading 1M prompt tokens
about $3.80
RTX 5090 32 GB
Card
$0.62/hr (from $0.27)
Typical rent
about $0.08
Reading 1M prompt tokens
about $3.38
Card
Typical rent
Reading 1M prompt tokens
Writing 1M answer tokens
RTX 3090 24 GB
$0.21/hr (from $0.09)
about $0.08
about $1.93
RTX 4090 24 GB
$0.47/hr (from $0.28)
about $0.08
about $3.80
RTX 5090 32 GB
$0.62/hr (from $0.27)
about $0.08
about $3.38
So the honest rule: if you paste long code, documents or RAG results, the faster prefill card saves you minutes a day for about the same money per token. If you mostly chat with short prompts, the 3090 is the value pick. Our RTX 4090 vs RTX 3090 comparison has the buying prices, and rent or buy a GPU for AI has the break-even hours.
Why does my model get slower and slower in a long session?
Because the context keeps growing, and both speeds fall as it does. Decode slows too: every new token reads the whole KV cache, so on the 3090 table Qwen3 14B writes 70 tokens a second at 4K, 38.6 at 32K and 25.4 at 64K. And when the session gets so long that the weights plus the cache no longer fit on the card, the runtime starts using system RAM and speed falls off a cliff.
@InkyPyrus_PubCo described exactly that on Nov 21, 2026: Qwen3.8 27B on an RTX 3090 going from 32 to 3 or 4 tokens a second after a few hours of coding. Our KV cache guide shows the memory math for that model and the two settings that halve the cache.
RTX 3090 writing speed as the context fills, Qwen3 14B Q4_K, tokens per second (hardware-corner.net)
4K context70.0 tok/s
16K context52.1 tok/s
32K context38.6 tok/s
64K context25.4 tok/s
How do I make prefill faster?
Let the cache do its job. The llama.cpp server re-uses the KV cache from the previous request by default (cache_prompt is true), so turn two only reads the new tokens. Keep the start of the prompt the same between turns: system prompt and files first, the new question last.
Watch for tools that rewrite the start of the context. Compaction, summaries and sliding windows change the beginning of the prompt, so the whole thing gets read again. That is a full prefill every time it happens.
Send less. Paste the function, not the repo. A 16K prompt waits less than half as long as a 32K one on the same card (7.8 vs 19.7 seconds on a 3090), because the deep tokens are the slow ones.
Keep everything on the GPU. Prefill on layers that spilled to system RAM runs at CPU speed. Check with ollama ps: anything other than 100% GPU means part of the model is on the CPU.
Batch size helps on the GPU. llama.cpp reads prompts in batches (-b 2048 logical, -ub 512 physical by default). A larger -ub can speed up prefill if you have the VRAM for the bigger buffers; test it with llama-bench.
Shop on prefill if your prompts are long. For coding agents and document work, a card with more compute (4090, 5090) cuts the wait in half. More memory bandwidth alone mostly speeds up writing.
Prefill, also called prompt processing, is the step where the model reads your entire prompt and builds its KV cache before writing anything. It handles all prompt tokens in large batches, so it is limited by the GPU's compute. It decides how long you wait before the first word.
What is decode in an LLM?
Decode, also called token generation, is the model writing its answer one token at a time. Each token reads all the model's weights and the KV cache from memory, so decode is limited mostly by memory bandwidth. It is the tokens-per-second figure most benchmarks quote.
Why does my local LLM take so long to start answering?
Because it has to read your whole prompt first, and reading gets slower the deeper the context goes. On hardware-corner.net's tables, Qwen3 14B at Q4_K reads 2,459 tokens a second at 4K on an RTX 3090 but 1,176 at 32K, so a 32K-token prompt waits roughly 20 seconds on a 3090, 10 on a 4090 and 8 on a 5090.
How much faster is an RTX 4090 than a 3090 at prompt processing?
About 1.9 to 2.25 times on hardware-corner.net's llama.cpp tables (Qwen3 14B and 32B at Q4_K), against a 14 to 34% gain in writing speed. On Vast on Nov 21, 2026 a 4090 rented from $0.28 an hour, typical $0.47, and a 3090 from $0.09, typical $0.21.
Does a quantized KV cache speed up prefill?
Its documented benefit is lower memory use, which lets a longer context stay on the GPU. Staying on the GPU is what keeps prefill fast; we found no primary source promising a speed gain on its own.
The short version: decode is the speed you see, prefill is the wait you feel. If your prompts are long, compare cards on prompt processing, keep the start of your context stable so the cache gets re-used, and test on a rented card before you buy. Live hourly prices for every card are on our cloud GPU price page.