"Cheapest" flips completely on how often you run it and how many people hit it at once — and in Europe, the electricity bill alone can cost most of what renting does.
"Cheapest" is the wrong question until you answer two others: how often will this thing actually be running, and how many people are hitting it at once? Change either and the cheapest option changes with it. A setup that's a bargain for one person tinkering in the evenings is a terrible way to serve four concurrent users, and the rig that wins for a 24/7 endpoint is the wrong thing to buy if you use it an hour a day.
First, what actually has to fit
A 70B model at Q4_K_M is roughly 40 GB of weights. Add a 16k-token KV cache and overhead and you're past what any single consumer card holds. That leaves three real shapes: two consumer cards (2×3090, 2×4090), one big professional or datacenter card (A6000 48 GB, H100 80 GB), or rented equivalents. You can see the exact memory breakdown per quantisation on our Llama 3.3 70B cost-to-run page — weights, KV cache and overhead separated, so you can see which term is the one pushing you over a card's limit.
The one honest measured number
I'll be straight about the evidence here, because most articles aren't. There is exactly one primary-source measured 70B throughput figure I'd stand behind: two RTX 3090s on llama.cpp, Llama Q4_0, at 179 tokens/sec prefill and 21 tokens/sec generation.
I also can't give you verified throughput for 2×4090, a single A6000, an H100, or a Mac Studio at this workload. Those are genuine gaps in the public measured data, not numbers I'm going to estimate and dress up as measurements.
Rent or buy: the number that surprised me
The reflex is that owning hardware must be cheaper than renting it forever. For a 24/7 endpoint in Europe, that reflex is wrong more often than you'd think — because of the electricity.
Two 3090s plus the rest of a system pull enough power that, at Denmark's household rate of €0.3312/kWh (Eurostat, H2 2025), running it around the clock costs on the order of €205 a month in electricity alone. Renting the same two 3090s on RunPod's Community Cloud is roughly $0.22/hr each. Do the arithmetic and the power bill is about two-thirds of the rental cost before you've spent a cent on the actual cards.
In high-electricity Europe, the wall socket alone can cost most of what renting the whole machine costs. Buying to save money assumes cheap power you might not have.
Fold in the purchase price, the risk of a five-year-old card failing, resale depreciation, and the hours you'll spend maintaining it, and the break-even against renting stretches past 16 months for a 24/7 workload. Buying still wins if you run the hardware hard and your power is cheap. It loses badly if the box mostly sits idle — at which point you're paying for depreciation and standby power to serve almost nothing. This is duty cycle, and it decides the whole question. You can check live rental rates against our tracked GPU pricing before committing either way.
The cloud reliability trap
If you rent, there's a distinction that matters far more for a served endpoint than for a batch job: interruptible versus guaranteed.
Pros
+ On-demand / guaranteed instances: your process keeps running. Correct for anything a user or app talks to live.
+ RunPod Community Cloud: cheaper than Secure Cloud, and its instances are on-demand — the trade is trust/compliance of the host, not preemption risk.
Cons
− Vast.ai interruptible: a higher bidder hard-preempts you and kills the running process, with no documented warning window. Fine for a batch run you can restart; disqualifying for an endpoint someone is waiting on.
− RunPod container disk is wiped on stop/restart — if your 40 GB of weights live there, every restart is a full re-download.
− RunPod volume storage costs MORE while stopped ($0.20/GB/mo) than running ($0.10). Stopping a pod to 'save money' on a 100 GB volume doubles the storage bill.
One correction to a common assumption: RunPod doesn't appear to offer spot/interruptible pods at all — its documented model is on-demand plus prepaid savings plans. So on RunPod the reliability axis is Community versus Secure Cloud, not spot versus on-demand. That's a different decision than the one people expect to make.
The config defaults that silently ruin a 16k deployment
These are the ones that cost people a weekend, because nothing errors — the setup just quietly does something other than what you asked.
**llama.cpp's `--ctx-size` is a total budget shared across all parallel slots, not per-user.** `-c 16384 -np 4` gives each of four users ~4k, not 16k each. If you need 16k *per* concurrent user, you multiply. (Confirmed by the llama.cpp maintainer.)
**Ollama defaults to a 4,096-token context.** Someone who 'set up 70B at 16k' through stock Ollama and got great throughput was almost certainly measuring 4k. Its memory scales as `OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH`.
**Ollama unloads the model after 5 idle minutes** (`OLLAMA_KEEP_ALIVE`). On a 40 GB 70B, the next request after that eats a full reload. Set it to `-1` for a served endpoint.
**Multi-GPU without NVLink wants pipeline, not tensor, parallelism** — vLLM's own docs steer non-NVLink builds that way, and the 4090 has no NVLink at all. The wrong choice here is a silent throughput tax.
So, cheapest — for whom
Verdict
Two answers, because it's two different machines
Single user, occasional use: two used 3090s on llama.cpp, or just rent by the hour — at low duty cycle, renting almost always wins and you skip the maintenance entirely. A real served endpoint with concurrency: a single 80 GB card on vLLM is the right shape, and whether you rent or buy comes down to duty cycle and your electricity price. If your power is Northern-European-expensive and the box won't run most of the day, rent.
Best for: Most people asking this are one person who wants a private 70B occasionally — for which the honest answer is usually 'rent it, and don't buy hardware to solve a problem you have for two hours a week'.
Quick answers
Can a single 24 GB card serve Llama 70B at 16k?
No. A 70B at Q4 is around 40 GB of weights before any KV cache, so 24 GB isn't close. You need two consumer cards or one 48–80 GB card. Our cost-to-run page for the model shows the exact requirement per quantisation.
Is llama.cpp or vLLM cheaper for serving 70B?
They're different tools. llama.cpp is the pragmatic choice for a single user on consumer hardware. vLLM is built for concurrency — it's dramatically more efficient at serving many simultaneous requests, and pointless overhead if you're the only user. 'Cheapest' depends on whether you're serving one person or many.
Why would renting beat buying if I use it a lot?
It often doesn't, if you use it a lot — that's exactly when buying wins. The break-even is duty cycle. At 24/7 in high-electricity Europe the power bill alone approaches the rental cost, which pushes break-even past a year or more once you add purchase, failure and resale. At low usage, renting wins outright.
Is Vast.ai's cheap interruptible pricing usable for an API?
For a live endpoint, no. Interruptible instances are hard-preempted by a higher bidder with no documented warning, and your process is killed. That's fine for a batch job you can restart and disqualifying for anything a user is waiting on. Use guaranteed instances for served workloads.
I got amazing tok/s from Ollama at '16k' — why doesn't this match?
Check your actual context window. Ollama defaults to 4,096 tokens, so a setup you believe is running 16k may be running 4k unless you explicitly raised OLLAMA_CONTEXT_LENGTH. Throughput at a real 16k of filled context is meaningfully lower.
The single most useful thing you can do before spending anything is work out what actually has to fit in memory at your real context length. Our cost-to-run pages compute that from each model's own configuration, so you can see whether you're buying for the weights or for the KV cache — and it's more often the KV cache than people expect.
