Apparently, this might actually be possible?:
Short answer: yes — at least in the sense of making 13B-class full-parameter fine-tuning fit on a single consumer GPU. The harder question is whether it would be practical enough to use, because once NVMe becomes part of the training memory hierarchy, the main limit shifts from capacity to how much data must move per step, through which path, and how much of that movement can be hidden behind GPU computation.
Also, there are already surprisingly close precedents for almost every part of this idea:
- DeepSpeed ZeRO-3 can offload model parameters and optimizer state to NVMe.
- DeepNVMe provides asynchronous tensor I/O and can benchmark GPU↔NVMe transfer paths.
- ZeRO-Offload already demonstrated up to 13B-class training on a single GPU using CPU memory.
- LoHan demonstrated full fine-tuning of very large models using a consumer RTX 4090, CPU RAM and SSD offloading.
- SlideFormer now explicitly treats GPU VRAM, CPU RAM and optional NVMe as a heterogeneous training-memory hierarchy.
- ASUS has even sold an RTX 4060 Ti with an onboard M.2 slot.
So I think the interesting part of the proposal is not really:
“Can an SSD be attached to a GPU?”
That part is already possible.
The interesting question is:
Can a consumer GPU expose that SSD through a sufficiently good data path, and can the training runtime schedule it well enough that NVMe behaves like a useful third memory tier rather than a giant stall generator?
For a hypothetical card like this, I would mentally separate the design into two parts:
- Physical design: put an M.2 NVMe device on the graphics card.
- Training architecture: make
VRAM -> RAM -> NVMe an efficiently managed memory hierarchy.
The second part is where most of the performance engineering appears to be.
A normal NVMe + DeepSpeed setup is probably the best baseline first
Before custom hardware, the low-cost/high-information test would actually be an ordinary machine with:
That already tests most of the important question:
If one NVMe device is added as the slow capacity tier, is 13B full fine-tuning merely slow, or unusably slow?
DeepSpeed currently supports parameter offload to CPU or NVMe under ZeRO-3:
"offload_param": {
"device": "nvme",
"nvme_path": "/local_nvme"
}
and optimizer-state offload to NVMe as well:
"offload_optimizer": {
"device": "nvme",
"nvme_path": "/local_nvme"
}
One important detail is that optimizer computation still runs on the CPU when optimizer state is offloaded. DeepSpeed also uses CPU-side pinned buffers for asynchronous transfers, so NVMe does not completely replace system RAM.
That means the three-tier picture is actually fairly close to reality:
GPU VRAM
|
| fast working set
v
system RAM
|
| buffering / optimizer / staging
v
NVMe
rather than:
GPU <----> 1 TB of magically slow VRAM
That distinction matters quite a lot.
DeepNVMe also has a useful tool for measuring the actual tensor I/O path rather than trusting the SSD’s box specification:
ds_nvme_tune --nvme_dir /local_nvme --gpu
Their current example uses an SSD advertised around 5.6 GB/s read / 4.3 GB/s write and obtains about 3.69 GB/s read and 3.18 GB/s write for the tested GPU-tensor path.
So I would probably benchmark that before doing anything involving a soldering iron. 
Capacity is probably not the difficult part
As a rough historical reference, Hugging Face’s training-memory anatomy gives about 18 bytes per parameter plus activations for conventional mixed-precision AdamW training.
For 13B parameters, that old-style accounting is roughly:
13 billion parameters × 18 bytes ~= 234 GB
before activation memory and temporary buffers.
The exact number changes with BF16/FP16 details, optimizer choice, master-weight handling, gradient representation, checkpointing, etc., so I would not treat 234 GB as a universal requirement.
But it does show something useful:
a 1 TB SSD is not obviously too small.
The difficult part is that those hundreds of gigabytes are not just parked somewhere. Some of them participate repeatedly in forward/backward/optimizer steps.
That turns the problem into a bandwidth-and-scheduling problem.
Why bandwidth and scheduling matter more than the 1 TB number
There is a useful older DeepSpeed result on an AMD MI100 system.
Microsoft reported that ZeRO-Infinity could train up to 120B parameters on one MI100 using NVMe capacity.
More interesting for this question, they also compared storage bandwidth:
- one NVMe device: about 12 TFLOPs training throughput,
- four NVMe devices: about 24 TFLOPs.
That is not a 16 GB GeForce / 13B benchmark, so it does not directly answer this exact proposed configuration.
But it is a nice demonstration of the basic behavior:
one SSD can work, while aggregate SSD bandwidth can make a very large difference.
A newer storage-vendor experiment provides an even more concrete example. Xinnor tested DeepSpeed with models up to 16B and compared one NVMe against a three-drive RAID0.
For their 16B test:
| Configuration |
Read per step |
Write per step |
Total step time |
| optimizer state → 1 NVMe |
~159 GB |
~318 GB |
~108 s |
| parameters + optimizer → 1 NVMe |
~291 GB |
~397 GB |
~139 s |
| optimizer state → 3-NVMe RAID0 |
— |
— |
~50 s |
| parameters + optimizer → 3-NVMe RAID0 |
— |
— |
~64 s |
That test used an RTX 6000 Pro with 96 GB VRAM, not the hypothetical 16 GB consumer GPU here, and Xinnor is itself a storage vendor, so I would treat it as a useful engineering measurement rather than a universal performance prediction.
Still, it illustrates the central problem very clearly:
once hundreds of GB are moved during one optimizer step, a single SSD can become the critical path.
So “1 TB” answers the capacity question, but something like:
bytes moved per optimizer step
--------------------------------
effective NVMe bandwidth
starts to determine the waiting time.
The good news is that sophisticated runtimes do not necessarily wait synchronously for every operation.
For example, SlideFormer overlaps:
- GPU computation,
- CPU optimizer updates,
- FP32 → BF16 conversion,
- host-to-device parameter transfers,
- device-to-host gradient transfers,
- activation offload/prefetch,
- optional NVMe I/O.
Its recent implementation even splits long transfers/updates into smaller chunks so more of them can overlap.
Likewise, LoHan describes the problem in terms of managing all intra-server tensor movement rather than treating each offload independently.
LoHan is particularly interesting because it targets exactly the general environment discussed here: a commodity server with a consumer GPU and limited main memory. The paper reports full fine-tuning of very large models on an RTX 4090 and a 2.32x throughput improvement over its compared baselines for 13B.
However, the public LoHan/Ratel artifact aggregates multiple SSDs into RAID for I/O bandwidth — its provided setup script is configured for up to twelve NVMe devices.
So it is strong evidence for:
“consumer GPU + SSD-backed full fine-tuning is possible”
but not evidence that:
“one M.2 SSD is automatically fast enough.”
The onboard M.2 part already has a consumer-GPU precedent
ASUS’s Dual GeForce RTX 4060 Ti SSD is almost comically close to the physical part of the proposal.
It has an M.2 2280 slot on the back of the graphics card, and the SSD uses the GPU heatsink/fans for cooling.
But there is a very useful distinction here.
According to ASUS’s motherboard compatibility document, the SSD communicates using lanes split from the PCIe x16 slot.
The motherboard therefore needs PCIe bifurcation such as:
x8 / x8
or:
x8 / x4 / x4
and ASUS even notes that its “PCIe RAID Mode” can split the slot into four x4 links, leaving the GPU itself operating at x4.
So that product proves:
yes, putting a real M.2 NVMe slot on a GeForce board is perfectly plausible.
But it does not make the SSD part of the GPU’s VRAM subsystem.
From the operating system’s point of view, it is still a PCIe storage device.
That gives a useful design question for a hypothetical LLM-training version:
Where do the SSD’s PCIe lanes come from, and what is the actual GPU↔SSD data path?
Where an onboard SSD could become more interesting than an ordinary motherboard SSD
If the SSD is just attached through PCIe bifurcation and normal host I/O, moving the M.2 connector from the motherboard onto the GPU may help with:
- physical packaging,
- cooling,
- motherboard M.2-slot availability,
- possibly topology consistency,
but it may not fundamentally improve LLM training.
The much more interesting version would expose a supported path such as:
NVMe
|
| PCIe peer-to-peer / direct DMA
v
GPU VRAM
instead of always requiring:
NVMe
|
v
CPU/system-memory buffer
|
v
GPU VRAM
This general idea is not hypothetical either.
NVIDIA’s GPUDirect Storage provides direct DMA paths between storage and GPU memory under supported configurations.
There was also an older AMD product, the Radeon Pro SSG, which combined a GPU with onboard NVMe storage. AMD’s surviving Linux setup documentation describes both Direct GMA support and NVMe peer-to-peer support, and the card exposed four NVMe devices.
So “GPU + local flash + special direct-access path” has historical precedent too.
However, I would not assume that current GeForce cards automatically support the same thing.
At the moment, NVIDIA’s NVMe PCI P2PDMA troubleshooting documentation explicitly lists:
- A100
- A40
- L4
- L40S
- H100
- GB100
as supported GPUs for that path.
GeForce is not in that explicit list.
There are also requirements around:
- Linux kernel support,
- filesystem,
- PCIe topology,
- driver configuration,
- NVMe multipathing,
- and whether the platform can actually route peer-to-peer PCIe traffic.
So for a hypothetical RTX 6060 Ti, the interesting product feature would not merely be:
“has an M.2 slot”
but something closer to:
“has an officially supported GPU↔NVMe data path designed for tensor streaming.”
That would be much more novel/useful for training.
If the goal is specifically 13B full fine-tuning, RAM changes the answer quite a bit
There is another reason I would benchmark CPU offload first.
DeepSpeed’s existing ZeRO-Offload tutorial says it can train models up to about 13B on a single GPU by moving optimizer memory and computation to the CPU. Their tutorial demonstrates a 10B GPT-2 on one 32 GB V100.
Later ZeRO-3 Offload pushed the single-GPU capacity further by moving more model state into CPU memory.
So, for 13B specifically:
lots of RAM available
|
v
CPU offload may already be enough
whereas:
RAM is limited
|
v
NVMe becomes much more valuable as a capacity tier
This does not make the onboard-SSD idea unnecessary.
It just changes what the SSD is buying you.
Rather than:
“13B is impossible without the SSD”
the benefit may be:
- requiring much less expensive DRAM,
- fitting larger-than-13B models,
- fitting larger batches/contexts,
- keeping more training state off host RAM,
- creating a standardized local storage tier per GPU.
That seems like a stronger justification for the design.
I would think of the card as a three-tier cache, not a 1 TB VRAM card
Something like:
Tier 0: VRAM
fastest, smallest
current layer / active tensors / hot working set
Tier 1: system RAM
slower, larger
staging buffers / CPU optimizer / warm training state
Tier 2: NVMe
much slower, huge
cold model state / optimizer state / selected activations
Then the runtime’s job becomes:
predict what will be needed next
|
v
prefetch it early enough
|
v
keep GPU compute busy
|
v
evict tensors whose next use is far away
That is essentially why projects such as SlideFormer, LoHan, SSDTrain, and DeepSpeed’s offload family are interesting here.
The SSD itself is not the clever part.
Tensor lifetime + placement + prefetch + overlap are the clever part.
A useful decision tree
For the proposed 13B use case, I would roughly evaluate it like this:
Can 13B full FT fit using VRAM + available system RAM?
|
+-- Yes
| |
| +-- CPU offload is reasonably fast
| -> probably use that as the default path.
| -> NVMe can remain optional overflow capacity.
|
+-- No
|
+-- Put optimizer/model state on one NVMe
|
+-- I/O is mostly hidden by compute
| -> this is probably a viable design.
|
+-- GPU spends most of its time waiting for storage
|
+-- reduce offloaded traffic
+-- keep a larger hot set in RAM
+-- improve prefetch/chunking
+-- increase SSD bandwidth
+-- use multiple SSDs
+-- investigate a direct GPU↔NVMe path
There is another independent branch:
Does the onboard M.2 provide a better data path
than a normal motherboard M.2?
|
+-- No
| -> useful packaging/cooling idea,
| but probably little training-specific advantage.
|
+-- Yes
-> potentially very interesting for LLM training,
especially if the driver/runtime exposes it cleanly.
A very small benchmark could answer most of the practical question
I would not start by building the special graphics card.
A normal NVMe system can already test the critical software behavior.
First, measure the storage path:
ds_report
ds_nvme_tune --nvme_dir /local_nvme --gpu
Then compare the same model/configuration under something like:
| Test |
VRAM |
RAM |
NVMe |
| A |
active tensors |
optimizer/model offload |
none |
| B |
active tensors |
staging/buffers |
optimizer state |
| C |
active tensors |
staging/buffers |
parameters + optimizer |
The interesting measurements are not complicated:
Does it fit?
seconds / optimizer step
tokens / second
GPU utilization
peak system RAM
NVMe read GB / step
NVMe write GB / step
If test B or C keeps the GPU reasonably busy, the concept is already looking good.
If GPU utilization collapses while the NVMe remains saturated, then you have found the main engineering target for the hypothetical hardware/runtime.
At that point, testing more PCIe topology becomes worthwhile:
nvidia-smi topo -m
lspci -tv
and, on an actually supported GDS system:
gdscheck.py -p
This gives a lot more information than starting with PCB changes.
A few secondary issues I would keep in mind
1. NVMe does not eliminate CPU RAM
Current DeepSpeed NVMe offload still has CPU-side machinery.
For example, its current configuration exposes:
- pinned CPU memory,
- parameter buffer pools,
- optimizer buffer pools,
max_in_cpu,- CPU optimizer execution.
So RAM capacity and bandwidth still matter.
A design with:
16 GB VRAM + 16 GB RAM + 1 TB NVMe
could behave very differently from:
16 GB VRAM + 128 GB RAM + 1 TB NVMe
even though both have the same SSD.
2. CPU memory bandwidth may become the next bottleneck
If optimizer updates run on the CPU, more CPU cores do not necessarily solve everything.
Eventually the optimizer becomes limited by DRAM bandwidth.
This is another reason why a good direct-storage path alone does not automatically solve the entire training pipeline.
3. Activations are a separate memory problem
Parameters and optimizer state are persistent, but activation memory depends strongly on:
- sequence length,
- micro-batch size,
- layer dimensions,
- checkpointing strategy.
SSDTrain is interesting because it specifically offloads activations to SSD and schedules I/O around their known lifetimes.
So a 13B model at sequence length 512 and the same model at 32K context can have very different bottlenecks even though the parameter count is identical.
4. SSD endurance could become relevant
Training offload can generate very large sustained write volumes.
That does not automatically mean consumer SSDs are unsuitable, because the amount written depends heavily on the runtime and which tensors are offloaded.
But for a real implementation, I would monitor actual writes rather than guessing from SSD capacity.
On Linux, NVMe SMART statistics make this measurable.
5. Sustained bandwidth matters more than the headline benchmark
A consumer SSD may advertise very high sequential write bandwidth while its SLC cache is available and then fall substantially during long sustained writes.
Training is exactly the kind of workload where sustained behavior matters.
6. Cooling could actually be an advantage of the GPU-mounted design
Putting an SSD near a hot GPU sounds suspicious initially, but ASUS’s implementation deliberately couples the M.2 device to the GPU heatsink with a thermal pad.
So thermally, an onboard SSD is not necessarily worse.
It depends on the actual cooler and airflow.
7. PCIe lane allocation needs to be explicit
The ASUS precedent is useful here.
Their on-card SSD does not get free bandwidth; it uses bifurcated lanes from the main PCIe slot.
For a purpose-built training GPU, it would therefore be useful to specify something like:
GPU: x8
NVMe: x4
remaining/reserved: x4
or whatever the architecture actually supports.
Otherwise “M.2 on the GPU” hides an important part of the design.
One thing that looks similar but is not quite the same
Hugging Face Accelerate can already put model weights on "disk" with device_map="auto" or disk_offload().
That can make a model larger than VRAM/RAM runnable, but the documentation explicitly describes this as big-model inference, not training.
So I would not use:
device_map="auto"
as evidence that full fine-tuning with disk offload is solved.
Full training additionally has to deal with:
- gradients,
- optimizer state,
- saved activations,
- update scheduling,
- repeated bidirectional traffic.
DeepSpeed/LoHan/SlideFormer/SSDTrain are much closer references for this particular idea.
So I think the answer is basically “yes, with an important asterisk”
The pieces already exist independently:
What does not seem to be established by those examples is the exact combination:
16 GB consumer GeForce + exactly one NVMe SSD + approximately 13B + ordinary full-parameter fine-tuning, at a speed most people would call practical.
That would still need measurement.
But I would expect the experiment to fail or succeed primarily based on bandwidth and scheduling, not because 1 TB is insufficient.
So if NVIDIA ever made something like this, I think the really interesting specification would be less:
RTX 6060 Ti — 16 GB + one M.2 slot
and more:
RTX 6060 Ti — 16 GB + local NVMe tier + supported GPU↔NVMe path + training-aware prefetch/offload runtime
That version seems technically much more interesting.
The funny part is that none of those ingredients is particularly science-fiction anymore. They just have not, as far as I can tell, been packaged together as a normal consumer GeForce feature specifically for full LLM training.