VPTQ: vector post-training quantization for very large language models

Vector quantization is the whole idea

The README describes VPTQ as Vector Post-Training Quantization, a post-training quantization method that uses vector quantization to reach high accuracy on large language models at an extremely low bit-width, below two bits. The headline claim is that 70B models, and even the 405B model, can be compressed to 1-2 bits without retraining while maintaining high accuracy, with a better accuracy figure on 1-2 bits, a 405B model under two bits and a 70B model at two bits.

The reason vector quantization is the mechanism rather than a detail is that it changes what gets stored. Quantizing each weight on its own against a fixed scale is a scalar problem, and the information you throw away at a fraction of a bit per weight is a rounding decision you cannot revisit. Grouping weights into vectors and assigning each vector a code from a shared codebook keeps the relationships between weights intact, so the same budget stores a different thing.

The project is a research release with a paper behind it. The README states that the VPTQ paper was accepted for the main track at EMNLP 2024, and it links a technical report, which is also in the repository as VPTQ_tech_report.pdf. The algorithm itself was early-released on a separate algorithm branch with its own tutorial, before the inference code was open sourced in September 2024.

Quantisation cost is a build-time problem, and the build is a CMake extension

The README lists a lightweight quantisation algorithm that costs roughly 17 hours to quantize a 405B Llama-3.1 model, and it claims agile quantisation inference with low decode overhead. Whether that cost is acceptable depends on your machine, and the first thing that cost depends on is the build.

VPTQ is not a pure Python package. The repository has a CMakeLists.txt, a cmake directory, a csrc directory and a third_party directory, and setup.py defines a CMakeExtension that points the setuptools build at a CMake project. It also imports the CUDA home directory from torch's cpp_extension module, which is how the build knows where the toolkit is:

from torch.utils.cpp_extension import CUDA_HOME

The setup script interrogates the local CUDA toolkit at build time and parses the nvcc version out of it:

def get_cuda_bare_metal_version(cuda_dir):
    raw_output = subprocess.check_output(
        [cuda_dir + "/bin/nvcc", "-V"], universal_newlines=True
    )
    output = raw_output.split()
    release_idx = output.index("release") + 1
    bare_metal_version = parse(output[release_idx].split(",")[0])

and uses that version to decide how many threads to run nvcc with, reading the NVCC_THREADS environment variable when it is set:

def nvcc_threads():
    _, bare_metal_version = get_cuda_bare_metal_version(CUDA_HOME)
    if bare_metal_version >= Version("11.2"):
        nvcc_threads = os.getenv("NVCC_THREADS") or (os.cpu_count() // 2)
        return nvcc_threads

So a CUDA toolkit and a working `nvcc` on your PATH are hard requirements, and the README says so in a note before the build instructions.

Installing means pip install vptq, and the source route is a git URL

The README recommends saving time by installing the latest release from PyPI rather than building:

pip install vptq

The package is also on PyPI, and the project offers a Huggingface Demo space, Colab notebooks, and a Gradio web app example among its evaluation entry points.

The section titled Install from PyPI then gives a git URL rather than a package index command, which is a naming slip in the documentation worth noting before you follow it:

pip install git+https://github.com/microsoft/VPTQ.git --no-build-isolation

Building from source is the third route, and it is the one that compiles the extension. The wheel is produced with setup.py and then installed from the dist directory:

   python setup.py build bdist_wheel

   # Install the built wheel
   pip install dist/vptq-{version}.whl  # Replace {version} with the version number

and the build artefacts are cleaned up with:

   python setup.py clean

The `--no-build-isolation` flag matters, since it tells pip to use the environment you already prepared rather than a throwaway one, which is what makes a locally installed CUDA toolkit visible to the build.

Two version floors, and the README and the manifest disagree

The dependency list is where the practical requirements live, and there is a discrepancy worth catching. The README's dependency section asks for the CUDA toolkit, python 3.10+, torch >= 2.3.0, transformers >= 4.44.0, Accelerate >= 0.33.0, flash_attn >= 2.5.0, latest datasets and cmake >= 3.18.0.

The packaging metadata is looser in places and stricter in others. The pyproject.toml declares requires-python as ">=3.8", while the README asks for 3.10 or higher, and the runtime dependencies are pinned to transformers>=4.45, a level above the 4.44.0 the README lists. The dependencies themselves are a wide surface:

dependencies = [
    "torch>=2.3.0",
    "datasets",
    "transformers>=4.45",
    "safetensors",
    "psutil",
    "accelerate",
    "gradio",
    "plotly==5.9.0",
    "pynvml",
    "tqdm",
    "sentence_transformers",
]

That list mixes the model runtime with the demo and evaluation tooling. Gradio, plotly and sentence_transformers are there because the project ships a web app and evaluation examples, and pynvml is there for GPU monitoring. If you are embedding VPTQ in a service, you inherit a fairly heavy dependency set for what is fundamentally a quantisation and inference kernel.

flash_attn is listed as a dependency in the README but does not appear in the pyproject runtime dependencies, which is another sign that the two lists were maintained separately.

Transformers support and inference engine support arrived in 2025

The news list in the README tracks how VPTQ moved from a research branch into the surrounding ecosystem, and it is the most informative part of the document for judging maturity. In December 2024 the VPTQ support was merged into the Huggingface Transformers main branch, and by January 2025 the README states that VPTQ is formally supported by Transformers in its wheel package release since v4.48.0.

That is the single most useful integration point for most users, because it means the quantization can be driven through the Transformers API rather than through VPTQ's own package entry points. The README's evaluation section lists a Huggingface Transformers API example and a Python API example from the VPTQ package as separate paths, and the Transformers route is the one that will keep working as upstream changes.

On the inference side, the README names aphrodite-engine as the first inference engine supporting VPTQ, credited to a contributor, and adds a preliminary support announcement for Deepseek R1 with a tutorial and separate inference code, stating that quantized models let you run Deepseek R1 on A100 GPUs. The support is described as preliminary, and a v0.0.5 release note mentions cmake support and an enhanced build pipeline.

The released community models do not match the headline bit-widths

The claim and the artefacts diverge in a way that a reader should notice. The headline is 1-2 bits for the 70B and 405B models, but the model collections the README links from the community are at different widths. Meta Llama 3.1 405B is listed at 3/4 bits, Mistral Large Instruct 2407 with 123B parameters is listed as a contributed model, Meta Llama 3.1 Nemotron 70B is listed with a note that work on the 4-6 bit versions is continuing, and Meta Llama 3.3 70B comes as a 1-4 bits collection.

So the shipped community artefacts for the largest models sit at 3, 4 and 6 bits rather than at the 1-2 bits the method is described as reaching. The gap is not explained in the README, and it is a reminder that the abstract capability and the downloadable weights are two different things.

The README's table of contents also lists a section titled Limitation of VPTQ, so the project acknowledges there are limits, but the text of that section is not part of the visible documentation here. That is the first place to look when you evaluate this package for a specific model, rather than relying on the summary in the news list.

A 0.0.x package, an MIT licence and a last push in August

The version tells you where this sits. The pyproject.toml names the package `vptq` at version 0.0.5.post1, and the tagged releases are v0.0.5post1 from 2025-03-06, v0.0.5 from 2025-01-18 and v0.0.4 from 2024-11-18. A zero major version means no stability promise, and the post-release suffix on the newest tag reflects the usual packaging fix rather than a feature.

The repository is not archived and the last push was on 2026-08-04, so the branch is being touched, but the gap between that push and the newest published package is long enough that the code you get from the branch and the code you get from PyPI may not be the same. The repository has no separate release notes file to explain the difference.

The licence is MIT, confirmed by the badge in the README and the LICENSE file at the root, and the setup.py header carries the Microsoft Corporation copyright. There is also a SECURITY.md and a SUPPORT.md at the root, plus a .pre-commit-config.yaml and a .clang-format, which say something about the project's internal tooling but nothing about API stability.

The other caveat worth naming is the dependency on the surrounding stack. The README's own example claims a 70B model on an RTX4090 with 24G of memory at roughly two bits in real time, and the Deepseek R1 announcement names A100 GPUs. This is a package that assumes a serious GPU, and the README's evaluation examples assume a Huggingface account and downloadable checkpoints on top of that.

Editorial conclusion

VPTQ fits an engineer who has a very large model and a single GPU, and needs it to fit without a fine-tuning budget, and who is willing to treat the package as a research artefact rather than a supported dependency. It is the wrong tool for a model small enough to quantize by hand with an off-the-shelf toolkit, and for anyone who needs a version of the model they can deploy today, because the tagged release is 0.0.5.post1 from 2025-03-06. Verify first that `nvcc` is on your PATH, that your CUDA version is at least 11.2 for parallel compilation, and which bit-width of community quantisation actually exists for the model you want to run.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. microsoft/VPTQ on GitHub
4. README
5. Releases