Your GPU has spent a surprising amount of its working life doing nothing. Not because it's slow — because the data it needs has to go through the CPU and system RAM first, and that hop has quietly been the bottleneck behind a lot of AI training and inference jobs. On August 4, at the Future of Memory and Storage conference in Santa Clara, Nvidia open-sourced the fix: cuFile, an API that lets a GPU read from and write to storage directly, and it's making the whole stack a public, shared standard instead of a black box only Nvidia controlled.
The problem this is actually solving
The technical term is GPU starvation, and it's exactly what it sounds like: a GPU that's paid for, powered on, and idle, because storage bandwidth can't keep up with how fast it can process data. It gets worse the bigger the cluster — a rack of GPUs stalling in sync while the CPU shuffles data from disk into RAM and only then hands it off is a much bigger problem than one gaming PC stuttering. Nvidia specifically calls out mixture-of-experts inference and retrieval-augmented generation as the workloads hitting this hardest, since both demand massive, concurrent, unpredictable data access patterns rather than one clean sequential read.
Aug 4, 2026
Announced
FMS conference, Santa Clara
4
Founding maintainers
Google, Intel, Meta, Nvidia
40+
Storage-Next vendors
including DDN, Kioxia, Micron
3.21x
Vera CPU throughput gain
vs. x86, in a compression/encryption pipeline
How the data actually moves
cuFile builds on GPUDirect Storage, a Nvidia technology that's existed in some form since CUDA Toolkit 11.4 in July 2021 — this isn't a from-scratch invention, it's the moment Nvidia handed the keys over. The mechanism itself is straightforward once you see it laid out.
Data sits on an NVMe drive, not yet touched by the CPU
2
cuFile initiates direct memory access (DMA) between the drive and the GPU
3
Data moves straight into GPU high-bandwidth memory — no detour through system RAM or CPU processing
4
Hundreds of thousands of GPU threads start working on it in microseconds, not milliseconds
That's the part worth sitting with: microsecond access, not millisecond. On paper that sounds like a rounding error until you multiply it across a training run that's constantly re-reading massive datasets, or an inference cluster serving concurrent requests against a model too large to keep fully resident in VRAM.
cuFile moves data from NVMe storage directly into GPU memory via DMA, cutting the CPU and system RAM out of the path. · Unsplash
Why open-sourcing it matters more than shipping it
Nvidia could have kept improving cuFile as a proprietary component and let storage vendors keep guessing at how to build against it. Instead, the code now lives on GitHub under a new Accelerated IO Special Interest Group, with Google, Intel and Meta as co-maintainers rather than just users. That's a meaningfully different arrangement — it means storage vendors like Kioxia and Micron can build hardware and firmware that targets a documented, shared standard instead of reverse-engineering Nvidia's assumptions. Nvidia also introduced SCADA, a framework that keeps this fast path secure by splitting privileged storage-access configuration from the application code actually requesting data, so opening the door to GPU-direct storage doesn't also open a door to every process on the box.
My honest take: this reads like a data-center and hyperscaler story first, and it is — nobody's home NVMe drive is starving a single RTX card for data in a normal desktop workload. But the direction matters for anyone running models locally too. The same bottleneck shows up in miniature whenever you're loading a model larger than your VRAM and streaming layers off disk, which is exactly the situation behind running a 70B model on a budget or figuring out how much RAM local AI actually needs. Standardized, faster GPU-to-storage paths trickling down from the data center is a real, if slow, tailwind for that.
It also fits a pattern Nvidia's been running all year of pushing infrastructure plays well beyond gaming GPUs — the same week it's been showing off physics simulation for training surgical robots, it's also rewriting how AI clusters talk to storage. Compute is not the company's only lever anymore; it's trying to own the whole path data takes to reach a GPU.
Quick questions
Is cuFile only useful for data centers?
Practically, yes, for now — the biggest gains are in large GPU clusters running MoE inference or RAG at scale. The same underlying idea, faster GPU-to-storage paths, matters for anyone streaming a model larger than their VRAM, but consumer tooling built on this is still ahead.
What is 'GPU starvation'?
It's when a GPU sits idle because the data it needs hasn't arrived yet — storage and CPU-mediated data transfer can't keep pace with how fast the GPU could actually process it.
Does this replace GPUDirect Storage?
No — cuFile is a core component of GPUDirect Storage. What's new is that the API and stack are now open source under a shared industry group, not that the underlying technology is brand new.
Will this affect gaming GPUs?
Not directly. This is aimed at AI training and inference infrastructure, not game loading times or frame rates.
Worth watching over the next few months: whether Kioxia, Micron and the other Storage-Next vendors ship hardware that actually targets cuFile out of the box, versus this staying a hyperscaler-only advantage for another generation. Open-sourcing the API is the easy part — getting forty vendors to build against it in a coordinated way is the part that decides whether this becomes a real standard or just a good GitHub repo.
