bitsandbytes: k-bit Quantization for PyTorch, From 8-bit Optimizers to QLoRA
Accessible large language models via k-bit quantization for PyTorch.
At a glance
- What is it?
- bitsandbytes brings 8-bit optimizers, LLM.int8() inference, and QLoRA 4-bit training to PyTorch, with support for CPU, NVIDIA, AMD, Intel, and Apple hardware. This review covers its mechanisms, setup, limitations, and alternatives.
- Who is it for?
- Adopt bitsandbytes if you are a PyTorch user who needs to run or fine-tune large language models on limited memory, especially on NVIDIA GPUs with SM75+ or on CPU with AVX2. Avoid it if you rely on Gaudi accelerators for 8-bit optimizers (not supported) or if you need stable performance on aarch64 CPU (marked with an asterisk for lack of optimizations).
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 8 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What bitsandbytes Actually Solves
Running or training a large language model on a single GPU often fails because the model weights and optimizer states exceed available memory. bitsandbytes attacks that problem with quantization, which reduces the number of bits used to represent each weight or gradient. The project targets PyTorch users who want to fit bigger models on existing hardware without buying new accelerators. Its three headline features address different points in the workflow. The 8-bit optimizers shrink the memory footprint of optimizer states during training, which is often the dominant cost when using Adam or similar methods. LLM.int8() is for inference, cutting memory roughly in half while aiming for no performance degradation. QLoRA goes further for training, quantizing the base model to 4 bits and adding a small set of trainable low-rank adaptation weights. The library exposes these through specific modules and classes, such as bitsandbytes.nn.Linear8bitLt and bitsandbytes.nn.Linear4bit, plus the bitsandbytes.optim module for optimizers.
The Mechanism: Block-wise, Vector-wise, and Outlier Handling
The quantization methods differ in how they map floating-point numbers to lower precision. The README describes 8-bit optimizers as using block-wise quantization, which means the scale factor is computed per block of values rather than per tensor. That local scaling preserves accuracy better than a single global scale. LLM.int8() takes a different route: it uses vector-wise quantization for most features, quantizing to 8 bits, but it identifies outliers and treats them separately with 16-bit matrix multiplication. That hybrid approach is what keeps output quality close to full precision, because outliers are often responsible for a disproportionate share of the model's behavior. QLoRA quantizes the model to 4 bits and then inserts low-rank adaptation weights that remain trainable. This separation lets you train without updating the full precision weights, which is what saves memory. Understanding these distinctions matters because they explain why the library does not simply round all values to a fixed number of bits.
Setup and System Requirements
The installation path depends on your platform. The README lists Python 3.10 and PyTorch 2.4 as minimum requirements for all platforms. On Linux, you need glibc >= 2.24. For x86-64 CPUs, the minimum is AVX2, with AVX512F and AVX512BF16 as optimized paths. NVIDIA GPUs need SM60 or newer, with SM75 recommended. The project also supports AMD GPUs with specific gfx targets, Intel GPUs including Arc A-Series and Data Center Max, and Apple M1 and newer on macOS 14+. The typical install command is not shown in the README, but the project is on PyPI and provides a continuous-release wheel named continuous-release_main. You would install it with pip, for example pip install bitsandbytes, but the README does not give exact commands. The documentation lives on Hugging Face, with separate guides for Transformers, Diffusers, and PEFT integrations. That integration is a key part of the setup: you often do not call bitsandbytes directly but through a library like Transformers that wraps its quantized layers.
Hardware Support: Broad but Uneven
The accelerator support table is the most informative part of the README, and it reveals real differences in maturity. On Linux x86-64, all three features are marked as supported for CPU, NVIDIA, AMD, and Intel GPUs. But the table also uses a legend with planned, partially supported, and not supported states. Intel Gaudi is a clear example: LLM.int8() is supported, QLoRA 4-bit is only partially supported, and 8-bit optimizers are not supported at all. Apple's Metal backend has 8-bit optimizers marked as planned, meaning they are not yet available. The asterisks on aarch64 CPU and macOS CPU indicate that those features are supported but may lack performance optimizations. That matters if you are on an M-series Mac and plan to train with 8-bit optimizers: it may work, but do not expect the same speed as on a CUDA GPU. The README notes that the table reflects the current development branch, so the stable release 0.50.0 may have a different status. Always check the tag-specific README if you are on a released version.
Where It Fails or Is the Wrong Tool
The most obvious limitation is hardware. If you are on an Intel Gaudi accelerator and want to use 8-bit optimizers, this library is not the right choice because the feature is listed as not supported. Similarly, if you are on Apple Metal and need 8-bit optimizers today, they are only planned, so you would be waiting. Another constraint is the Python and PyTorch floor: Python 3.10 and PyTorch 2.4 mean that projects pinned to older versions cannot use the latest bitsandbytes without upgrading. The README also warns that while they aim for wide backwards compatibility, using the latest PyTorch is recommended. That is a maintenance signal: you may need to move your whole stack forward to stay on supported versions. Performance is another concern. The asterisks on aarch64 CPU and macOS CPU say those features 'may lack performance optimizations.' If you are on an Apple Silicon Mac and expect production-grade throughput for LLM.int8(), the documentation is telling you that you might be disappointed. For users who only need inference and have ample memory, quantization is an unnecessary complexity layer; full precision inference is simpler and avoids any risk of quality loss.
Alternatives: How They Differ in Approach
The most direct alternative is Hugging Face's own quantization integrations, which often build on bitsandbytes but add a layer of abstraction. The README links to Transformers, Diffusers, and PEFT documentation for bitsandbytes, so those are not competitors but wrappers. A real alternative in approach is GPTQ, which is a post-training quantization method that calibrates against a dataset to find optimal scales and zero points. Unlike bitsandbytes, which applies quantization at runtime with a deterministic scheme, GPTQ requires a separate calibration step and produces a quantized model file. That means GPTQ models are typically pre-quantized and loaded directly, while bitsandbytes can quantize a model on the fly. Another alternative is AWQ, which also uses activation-aware scaling to protect important weights. The key difference is that bitsandbytes is a general-purpose library that also provides optimizers and training support, while GPTQ and AWQ are focused on inference-time compression. If your goal is purely to serve a quantized model and you are willing to run calibration, GPTQ or AWQ might give better quality per bit, but they do not offer the training path that QLoRA does.
Maintenance, Licensing, and Upgrade Cost
The project is under active development. The last push was in September 2026, with a recent stable release 0.50.2 in August 2026 and a continuous-release wheel updated in September 2026. The release notes for 0.50.1 mention RTX Spark support, which indicates the project tracks new NVIDIA hardware. This cadence means you should expect frequent updates and plan for them. The license is MIT, which is permissive and does not impose copyleft obligations, so you can use it in commercial products without releasing your own code. That is a low-license-risk choice, though you should still review the full license text if you have specific legal concerns. The maintenance cost comes from keeping up with PyTorch updates. The README recommends using the latest PyTorch, which means every PyTorch release could require a bitsandbytes update. If you are integrating with Transformers, you also need to track version compatibility across three libraries. The project's sponsorship by Hugging Face and others suggests ongoing investment, but the documentation does not specify a formal support contract, so you are relying on community and maintainer responsiveness.
Editorial conclusion
Adopt bitsandbytes if you are a PyTorch user who needs to run or fine-tune large language models on limited memory, especially on NVIDIA GPUs with SM75+ or on CPU with AVX2. Avoid it if you rely on Gaudi accelerators for 8-bit optimizers (not supported) or if you need stable performance on aarch64 CPU (marked with an asterisk for lack of optimizations). Before adopting, verify that your hardware is listed in the accelerator table, that your PyTorch version is 2.4 or newer, and that you can meet the glibc requirement on Linux. Check the latest release notes, as the project is under active development with recent updates for RTX Spark support.
Community notes