What is VRAM?

What the number means

VRAM is memory that sits on the graphics card itself, on its own memory bus, connected to the compute units by that bus rather than by the system's memory controller. That placement is the whole point. System RAM behind a slow controller and VRAM next to the processors is a difference in latency and bandwidth large enough to change what a workload is bound by, and it is why a model that fits in VRAM can be many times faster than the same model running from system memory.

The consequence people hit is that VRAM is a fixed pool per card, allocated by the processes that run on it, and the allocation has to be explicit. A typical step is: the host asks the driver for a block of device memory, copies weights into it, then launches a kernel that reads it. Anything that does not fit is either refused, or, on platforms with unified memory, paged between the card and system memory behind your back. That paging is what makes a model technically runnable and practically unusable, and it is the reason the same code behaves differently on a laptop.

So the practical question is not how much memory a card has on the box, it is how much of it is left for the process you care about after everything else on the card is taken.

There is a second reason the number gets treated as a budget rather than a resource to be maximised. Memory on a GPU is not fungible with time. A workload that reads its weights once per token spends most of its time waiting for the memory system, so the arithmetic that decides its speed has almost nothing to do with the arithmetic units. Adding compute does not help a memory-bound workload, and using more memory efficiently does.

Where the memory actually goes

Three consumers, and the biggest one is usually the one people forget.

The weights. Parameter count times bytes per parameter, and the bytes are a choice: two-byte and one-byte formats roughly halve and quarter the footprint, at a cost in accuracy that depends on the model and the method. Our analysis of brontoguana/krasis describes a runtime that keeps multi-hundred-billion-parameter mixture-of-experts models on VRAM-limited cards by keeping prefill and decode on the GPU while a component it calls HCS moves hot and cold experts between VRAM and system memory. The interesting part is not the total parameter count but the fact that in a sparse model most of the weights are not read during a given token, which is what makes moving them worth doing.

The cache. In autoregressive generation the model keeps the keys and values it has already computed. The cache grows with sequence length, with batch size, and with the number of layers, and for a long conversation on a large model it is frequently larger than the weights. This is the single most effective thing to understand about VRAM, because it is a cost that scales with the thing the user controls: the context window. Grouped-query attention exists to shrink it, and quantising the cache is the other common answer.

The working set. Activations are small during inference and large during training, where gradients and optimiser states sit alongside the weights. pollockjj/ComfyUI-MultiGPU is a GPL-3.0 ComfyUI custom node that offloads UNet, CLIP and VAE layers to system RAM or additional GPU VRAM and distributes model shards across multiple devices, and its README states the repository will be archived on 30 September 2026 and is no longer maintained, which is worth knowing before you build on it.

The general shape of every technique below follows from that list: keep the weights on the card, keep only the hot experts, keep the cache small, and push the rest to system memory across the bus.

When the number decides, and when it does not

VRAM is the binding constraint in a small number of situations, and outside them it is one line item among several.

It decides for local inference. There is no second machine to spill onto, so a model either fits or it does not. SillyTavern's description recommends a 3000-series NVIDIA graphics card with at least 6GB of VRAM for local inference, while noting that actual requirements vary with the model and backend, and the projects that target a specific card size treat it as the design constraint: our analysis of cocktailpeanut/fluxgym says it pairs a Gradio frontend derived from an AI toolkit with the Kohya sd-scripts backend to give FLUX LoRA training a web interface that runs on 12GB, 16GB and 20GB of VRAM.

It decides for image and video diffusion, where several large components have to be resident at once rather than one at a time, which is why the multi-GPU offload nodes exist.

It decides at the small end, where the point is to avoid needing a discrete card at all. Our analysis of H-EmbodVis/TurboVLA records a vision-language-action model reporting 97.7 percent average success on LIBERO at 31.2 ms latency and 0.9GB of inference VRAM, which is the shape of result that makes an edge device plausible.

It stops deciding when the work is on someone else's hardware, when the model is small enough to sit in system memory without collapsing throughput, when a training run is spread across many devices so that the per-device share is small, and when the architecture itself is built to need less. kyegomez/VisionMamba is the last case: our analysis describes it as a Python package reimplementing the Vision Mamba paper's bidirectional state space model in PyTorch, shipping a single class and no training script, and its description claims it is 2.8 times faster than DeiT and saves 86.8 percent of GPU memory when doing batch feature extraction on high-resolution images.

Where the plans go wrong

Five mistakes account for most of the disappointment, and none of them are about reading the wrong specification.

Bandwidth and capacity confused. A card with more memory is not a faster card, and a card with more bandwidth can run a memory-bound model faster while having less capacity. The two numbers are bought together and neither substitutes for the other.

Free memory that is not free. Display output, other processes, the driver's own reservations, and fragmentation all come out of the same pool, so the usable figure is lower than the advertised one. A caching allocator also holds memory it is not currently using, so a monitoring tool reporting a large used figure is not evidence of a leak.

Offload moving the bottleneck. Streaming weights from system memory across the bus works until the bus is the limit, and then the arithmetic units wait on PCIe instead of on VRAM. c0deJedi/nbd-vram takes this idea to its extreme, and our analysis describes it as a Linux daemon that allocates NVIDIA GPU VRAM through CUDA and serves it as a block device over the Network Block Device protocol, giving laptops with soldered memory a fast, low-wear swap tier without writing or maintaining a kernel module. That is a deliberate trade of one bottleneck for another.

An out-of-memory error that is really a sequence-length error. Batch and context are the two variables people forget when a model that worked yesterday fails today.

Treating a number as a summary. An inference footprint is not a training footprint, and a project's headline figure usually describes the friendlier of the two.

How the projects use the budget

Read together, the repositories here are five different answers to a memory problem.

Fit the training run into a fixed card. cocktailpeanut/fluxgym targets LoRA training at 12GB, 16GB and 20GB by pairing a simple web interface with an existing training backend, and our analysis notes its last push was 2026-07-28 and that the project has no GitHub releases, so the commit history is where change is visible. momori777/Artemis takes the same idea to a whole stack, bundling llama.cpp, GPT-SoVITS, ComfyUI, Live2D and a vector store behind chat channels and a local web frontend, and our analysis calls it a Windows-first PowerShell project for people who already own the GPU and the patience, which is the honest framing for anything that schedules several models into one card.

Schedule several devices instead of one. pollockjj/ComfyUI-MultiGPU spreads model shards across devices and manages offload and block swap to DRAM or VRAM, which is what a pipeline with several resident components needs.

Stream what is cold. brontoguana/krasis is the clearest statement of the idea, keeping prefill and decode on the GPU while moving hot and cold experts between VRAM and system memory; our analysis of it notes the install path is short and that the cache build and the licence are the parts to think about first.

Use less. kyegomez/VisionMamba changes the architecture rather than the placement, and installs from PyPI as a single class with no training script, so it is a component to read and depend on rather than a pipeline to operate.

Borrow the card for something else. c0deJedi/nbd-vram treats VRAM as storage rather than as working memory, which is the one approach here that will make a memory-bound workload slower, and is aimed at laptops with soldered memory where the alternative is swapping to an SSD.

And the boundary case. H-EmbodVis/TurboVLA reports a sub-gigabyte inference footprint, which is a reminder that the number people quote for a project is almost always the one that looks best, and always about one task rather than all of them.

In practice

VRAM is a fixed pool on a fixed card, and a model has to fit inside it or be paged, and paged is slow enough to be a different answer rather than a smaller one. Work out the three consumers before choosing hardware: weights at the precision you can accept, the cache at the context length your users will actually reach, and the working set of whatever else is resident. Then decide whether to shrink the demand, as kyegomez/VisionMamba does by changing the architecture, or to move what is cold, as brontoguana/krasis and pollockjj/ComfyUI-MultiGPU do, or to train against a specific card size, as cocktailpeanut/fluxgym does. Measure usable free memory on the machine you will deploy on, because the number on the box is not the number the process gets.