The new 4-bit format baked into RTX 50-series and Blackwell chips runs AI models at up to 3x the inference throughput of FP8, at nearly the same accuracy. Here's how it works and why it's exclusive to your newest GPU.
Every generation, the trick to running AI models faster is to represent their numbers in fewer bits — and NVIDIA's newest move is NVFP4, a 4-bit floating-point format built into the Blackwell architecture that powers RTX 50-series cards. The pitch is genuinely impressive: NVIDIA reports up to 3x the inference throughput of FP8 at close to the same accuracy, plus roughly half the memory traffic. If you've seen the query 'sm120 nvfp4' and wondered what it means, SM120 is the compute capability of desktop Blackwell chips, and NVFP4 is the format they can run natively. Here's how it actually works, and the catch: it only runs on Blackwell.
How you get accuracy out of only 4 bits
The obvious problem with 4-bit numbers is that there aren't many of them — NVFP4's base encoding (E2M1) represents values in roughly the range of -6 to +6, which is far too coarse to store a neural network's weights directly. NVIDIA's solution is scaling, done cleverly. Values are grouped into blocks of 16, and each block shares a high-precision FP8 scaling factor; on top of that sits a per-tensor FP32 scale. So instead of one crude scale for a whole tensor, every group of 16 numbers gets its own fine-grained adjustment. That's what lets 4-bit values track the real distribution of weights closely enough to preserve accuracy within about 1% of FP8 — the format is 4-bit, but the scaling around it is high-precision, and that combination is the whole innovation.
NVFP4 vs FP8 on Blackwell — NVIDIA's reported figures
Inference throughput~3x FP8
Pretraining speed~1.9x FP8
Memory per element~half FP8
Accuracy loss<1%
Why it only works on Blackwell (and what SM120 means)
The speed only materialises if the hardware can do the math in 4-bit directly, and that requires Tensor Cores designed for NVFP4 — which is exclusive to Blackwell. When both a model's weights and activations are quantized to NVFP4 (called W4A4), the dominant matrix multiplications run right on Blackwell's Tensor Cores at up to a 4x throughput ceiling over BF16, at a quarter of the memory traffic per element. Older GPUs simply lack the hardware path, so they can't accelerate NVFP4 no matter how the model is quantized. SM120 is the compute-capability identifier for desktop and workstation Blackwell — the RTX 50-series — which is why you'll see 'sm120 nvfp4' in the context of running these models on consumer cards, typically served through vLLM or TensorRT-LLM.
Fewer bits means less memory and less compute per token — which is why 4-bit formats like NVFP4 are where efficient inference is heading. · Unsplash
What it means for local AI
For someone running models at home, NVFP4 is one of the real reasons to care about Blackwell beyond raw specs. It means an RTX 50-series card can run NVFP4-quantized models faster and with less memory pressure than the same card on older formats — and NVFP4 checkpoints of frontier models (including DeepSeek-R1) are already published for Blackwell deployment. The catch is ecosystem maturity: NVFP4 inference ships in TensorRT-LLM and works in vLLM, but it's newer and more specialized than the GGUF format that llama.cpp uses everywhere. If you have a Blackwell card and want maximum efficiency, NVFP4 is worth learning; if you're on older hardware, it's simply not available to you, and FP8 or 4-bit GGUF remains your path.
The questions people actually ask
What is NVFP4 in simple terms?
It's NVIDIA's way of storing an AI model's numbers in just 4 bits each — a quarter the size of the usual 16-bit — while keeping accuracy high by giving every small group of numbers its own precise scaling factor. On Blackwell GPUs that support it in hardware, this lets models run up to about 3x faster than FP8 and use roughly half the memory traffic, at under 1% accuracy loss. It's a compression format designed specifically for fast, efficient inference.
Which GPUs support NVFP4?
Only NVIDIA Blackwell GPUs — that includes the RTX 50-series (desktop Blackwell, identified as SM120) and datacenter Blackwell parts. The acceleration comes from Tensor Cores built to handle NVFP4 natively, which earlier architectures like Ada (RTX 40-series) and Ampere (RTX 30-series) don't have. On those older cards you can't get NVFP4's speedup; you'd use FP8 or a 4-bit GGUF quantization instead, which run more slowly but work everywhere.
Is NVFP4 better than FP8?
For throughput and memory, yes on Blackwell — NVIDIA reports up to ~3x the inference speed and about half the memory traffic, at accuracy within roughly 1% of FP8. The tradeoff is that it's 4-bit, so there's a small accuracy cost, and it only runs fast on Blackwell hardware. FP8 is more widely supported across GPU generations. If you have a Blackwell card and the tooling, NVFP4 is the more efficient choice; otherwise FP8 remains the practical option.
Can I use NVFP4 with llama.cpp?
Not really — NVFP4 is primarily deployed through NVIDIA's own stack (TensorRT-LLM) and vLLM, not llama.cpp, which centres on the GGUF format and its own quantizations. If you're a llama.cpp user, you'd stick with GGUF 4-bit quants, which run on a much wider range of hardware including CPUs and Macs. NVFP4 is the choice when you're on Blackwell and using a serving framework that supports it and want the maximum efficiency it offers.
NVFP4 is a genuinely clever piece of engineering — 4-bit numbers made accurate by high-precision scaling, run natively on Blackwell for up to 3x FP8 throughput. Its limitation is also its design: it needs Blackwell to fly. Sources: NVIDIA Research and the Edge AI + Vision analysis. If you're weighing a Blackwell card partly for this, our RTX 5080 vs 4090 and is-the-5090-worth-it pieces cover whether the rest of the card justifies it.
