Because AI inference is memory-bandwidth-bound, the last chunk of your GPU's power budget buys almost no extra tokens. Cap the wattage and you run cooler, quieter, and cheaper for basically free.
This is one of the best free wins in local AI, and almost nobody does it: cap your GPU's power limit. Because AI inference is memory-bandwidth-bound — the chip spends its time reading weights from memory, not maxing out its compute — the top slice of the power budget buys almost no extra tokens. So you can slash the wattage for a trivial speed loss: an RTX 3090 capped at 280W saves about 70W for under 1% slowdown, and an RTX 4090 limited to 350W keeps ~90% of its performance while cutting power roughly 40%. The payoff: your GPU runs cooler, quieter, cheaper, and longer-lived — for basically no downside. It's a one-command change. Here's how and why.
How to do it (one command)
It's genuinely simple. On Linux, nvidia-smi sets a persistent power cap in watts; on Windows, MSI Afterburner's power-limit slider (or a small undervolt) does the same. Start near the sweet spots above and watch your tokens/second — you'll see the speed barely move while the temperature and fan noise drop noticeably.
# Linux — cap the GPU power limit (watts). Persists until reboot.
sudo nvidia-smi -pm 1 # enable persistence mode
sudo nvidia-smi -pl 280 # RTX 3090: 280W sweet spot (~1% loss)
# RTX 4090: try sudo nvidia-smi -pl 350 (~90% perf, ~40% less power)
nvidia-smi # confirm the new limit + watch temps
# Windows: use MSI Afterburner's Power Limit slider to the same target.
Then run your usual model and compare tokens/second before and after. On an RTX 3090 you'll typically see well under a 1% difference at 280W — but a much cooler, quieter card. If you want to push further, drop toward 250W and accept ~5-10% slower generation in exchange for even lower heat and power. There's no risk to the hardware; you're only reducing what it draws.
Cap the watts and the card runs cooler and quieter — while inference speed barely changes. · Unsplash
Why it works — and why it's basically free
The reason this works comes down to what actually limits AI generation speed. A GPU's power budget is mostly there to feed its compute cores at full tilt — which matters for gaming and training. But inference is different: the card spends most of each token waiting on memory bandwidth, reading the model's weights, not saturating its compute. An RTX 3090 already has ~1000 GB/s of memory bandwidth, and that number doesn't change when you cap the power — so the part of the workload that sets your speed is untouched, while the compute cores (which were partly idle anyway) simply run at a slightly lower clock. That's why you lose ~1% of speed for ~20% less power on a 3090: you're trimming watts the inference wasn't really using. The same bandwidth-bound principle explains a lot of local-AI hardware — it's why a 4090 isn't proportionally faster than a 3090 for inference, and why fast RAM matters so much. Practically, power-limiting is close to a free lunch: do it if you run local AI for more than a few minutes at a time, and enjoy a cooler, quieter, cheaper rig. Combined with a lower electricity bill, it's one of the highest-value five-minute tweaks in local AI.
Quick answers
Does power-limiting a GPU slow down AI inference?
Barely. Because local AI inference is memory-bandwidth-bound rather than compute-bound, capping the power limit trims watts the workload wasn't really using. An RTX 3090 limited to 280W (from a 350W stock limit) loses under 1% of its inference speed while saving about 70W, and even at 250-280W the loss is only around 5-10%. An RTX 4090 capped at 350W (from up to 575W) keeps roughly 90% of its performance while using about 40% less power. In exchange for that tiny speed drop you get a cooler, quieter, cheaper, longer-lived card — which is why power-limiting is one of the best free wins in local AI.
How do you power-limit a GPU for local AI?
On Linux, use nvidia-smi: run 'sudo nvidia-smi -pm 1' to enable persistence mode, then 'sudo nvidia-smi -pl 280' to set a 280-watt cap (a good RTX 3090 sweet spot); for an RTX 4090 try 350W. On Windows, use MSI Afterburner's power-limit slider set to the same target, or apply a small undervolt for a similar effect. After setting it, run your usual model and compare tokens per second — you'll see the speed barely change while temperatures and fan noise drop. The change is safe because you're only reducing what the card draws, and it persists until reboot (or permanently via a startup script).
What is the best power limit for an RTX 3090 running LLMs?
Around 280W is the widely-cited sweet spot for an RTX 3090 doing LLM inference — it saves roughly 70W versus the 350W stock limit for under 1% speed loss. If you want to prioritise low heat and power even more, dropping to 250-280W costs only about 5-10% performance while running significantly cooler. Below that, you start giving up more speed for diminishing thermal gains. The reason the 3090 tolerates this so well is that inference is bandwidth-bound, and its ~1000 GB/s memory bandwidth is unaffected by the power cap — so you lose almost nothing that actually determines token speed.
Power-limiting is one of the highest-value five-minute tweaks in local AI: ~20% fewer watts for ~1% less speed, and a cooler, quieter, cheaper rig. It pairs naturally with speeding up inference and a lower electricity bill. Sources: Jean Brito, PromptQuorum.
