airllm

AirLLM cuts inference memory so a 70B model runs on a single 4GB GPU, with no quantization, distillation, or pruning involved.

AirLLM only ever keeps one layer on the GPU at a time, so the VRAM you need depends on a model's layer size rather than its total size. That is how a 671B model can fit on a hobbyist card, according to the README.

The claims are specific: 70B models on a single 4GB GPU, 405B Llama 3.1 on 8GB, DeepSeek V3 at 671B on about 12GB, and Kimi K3, described as the largest open source model released to date, on under 4GB, because sparse MoE models stream one expert at a time.

A model compression feature based on block wise quantization can speed inference up to 3 times with, the README says, almost ignorable accuracy loss. Because the bottleneck is disk loading, only the weights get quantized, which keeps accuracy easier to maintain.

Notable entries include Kimi K3 support measured at 3.72GB of VRAM on a single RTX 6000 Ada, v3.0 adding FP8 model support, CPU inference in v2.10.1, and an AutoModel feature that detects the model type so no class needs to be passed in.

The README says AirLLM works out of the box with virtually every popular open LLM, covering Llama, Qwen, DeepSeek, Mistral and Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and Yi. A lot of the code is credited to SimJeg's work in a Kaggle competition.

The README is the only source here, and it is a running changelog as much as a manual. Nothing in this review claims the memory figures were reproduced.

• Source 1 · Project repository
• Source 2 · README