turbovec: A Rust vector index that trades a training step for SIMD speed and 4-bit memory

What turbovec actually is

turbovec is a vector index written in Rust with Python bindings, built on Google Research's TurboQuant algorithm. The README claims a 10 million document corpus that takes 31 GB as float32 fits in 4 GB, and that searches are faster than FAISS. The core idea is a data-oblivious quantizer: no training phase, no parameter tuning, no rebuilds as the corpus grows. You add vectors and they are indexed immediately. This is a different starting point from product quantization, which typically requires a codebook trained on a sample of the data. The project targets RAG pipelines where memory, privacy, or latency matter, and it is explicitly positioned as a local, air-gapped alternative to managed vector services. The repository is MIT licensed, with the primary language listed as Python even though the index itself is Rust. The PyPI package is named turbovec, and there is a crates.io package of the same name.

Online ingest and the absence of a training step

The most distinctive design choice is that there is no train step. With FAISS IndexPQ, you typically fit a codebook on a sample of vectors before you can add the full corpus. turbovec skips that entirely. The README says you can add vectors and they are indexed immediately, with no parameter tuning and no rebuilds as the corpus grows. This is a real operational advantage for streaming workloads where new vectors arrive continuously and you cannot afford a separate training pass. But it also means the quantizer is data-oblivious: it does not adapt to the distribution of your vectors. The paper that TurboQuant is based on claims near-optimal distortion for a fixed rate, but the practical recall depends on how well that assumption holds for your data. The README itself notes that low-dimensional vectors are a harder regime, specifically GloVe d=200, where the asymptotic Beta assumption is looser. So the convenience of no training comes with a statistical risk that you must evaluate on your own corpus.

How search works: SIMD kernels and block-level filtering

Search performance comes from hand-written SIMD kernels. The README lists NEON SDOT/SMMLA on ARM, AVX-512 VNNI and vpermb on x86, with AVX2 and scalar fallbacks. The claim is that these beat FAISS IndexPQFastScan in every measured config, averaging 3.4x at 4-bit and 23% at 2-bit across eight cells on both architectures. Filtering is integrated into the kernel, not applied as a post-pass. When you pass an allowlist of ids or a slot bitmask to search(), the kernel processes vectors in 32-vector blocks. Blocks with no allowed slots are short-circuited before any lookup table or scoring work. Inside a scored block, non-allowed slots are dropped at heap insert. This means selective allowlists avoid most of the SIMD cost, rather than paying it and discarding results. The output length is min(k, n_allowed), where n_allowed counts distinct allowed vectors. This is a concrete mechanism, and it is a meaningful difference from a naive filter-then-search approach. The README gives an example of hybrid retrieval: an external system like SQL or BM25 narrows candidates, then turbovec reranks within that set.

Getting it running: Python and Rust entry points

Installation is straightforward. In Python, you run pip install turbovec. The basic usage is: from turbovec import TurboQuantIndex; index = TurboQuantIndex(dim=1536, bit_width=4); index.add(vectors); scores, indices = index.search(query, k=10). Vectors and queries must be 2-D float32 arrays of shape (n, dim). The README is explicit that other dtypes are rejected rather than silently converted, so you must cast with np.asarray(x, dtype=np.float32) if needed. For stable ids that survive deletes, there is IdMapIndex. You create it with the same constructor, then add_with_ids(vectors, np.array([1001, 1002, 1003], dtype=np.uint64)). Search returns your external ids. Removal is O(1) by id. Persistence comes in two forms: write and load for whole-file snapshots, and sync for incremental saves. sync(path) persists just what changed since the last sync, with one fsync per call, and the README claims it is crash-safe at any byte. A removal or small append costs milliseconds regardless of index size. In Rust, you add the crate with cargo add turbovec, then use TurboQuantIndex::new(1536, 4).unwrap() and the same add, search, write, load methods. The API surface is small, which is a plus for learning, but it also means you should read docs/api.md for the full reference before relying on it.

Recall numbers: where turbovec wins and where it loses

The README includes recall comparisons against FAISS IndexPQ with LUT256 and nbits=8, using the paper's Section 4.4 baseline. On 100K vectors with k=64, TurboQuant (calibrated, called TQ+) beats FAISS at R@1 on three of four cells across OpenAI d=1536 and d=3072, by 0.9 to 2.9 points. The exception is d=1536 at 4-bit, where it trails by 0.7 points. Both reach 1.0 by k=8, with values at or above 0.997 already at k<=4. For GloVe d=200, the harder low-dimension regime, TQ+ is ahead at R@1 at both bit widths (+1.9 at 4-bit, +0.8 at 2-bit), but FAISS keeps a slim edge at 2-bit from k around 8. The README is careful to note that it compares against FAISS IndexPQ, not the custom u8-LUT PQ in the paper, because FAISS uses a higher-precision LUT and k-means++ for codebook training. That is a stronger baseline. The uncalibrated numbers are available in JSON files named tq_recalls. These numbers come from the project's own measurements, not from an independent test. You should treat them as indicative, not as a guarantee for your data. The low-dimension caveat is real: if your embeddings are below 200 dimensions, expect recall to degrade.

Incremental saves and crash safety

Persistence is a notable feature. The write and load methods handle whole-file snapshots, which are simple to reason about. The sync method is more interesting: it persists only what changed since the last sync, with one fsync per call. The README claims the format is crash-safe at any byte, meaning an interrupted write should not corrupt the index. A removal or a small append costs milliseconds regardless of how large the index is. This is a meaningful advantage over systems that require rewriting the entire index on every update. For workloads that update frequently, such as a continuously growing document store, this could save significant I/O. However, the documentation does not specify the exact on-disk format or how recovery works after a crash. The claim of crash safety at any byte is strong, and you should test it yourself before trusting it in production. The README also mentions that sync works for both TurboQuantIndex and IdMapIndex, with ids included in the latter. There is no mention of concurrent access or locking, so you should assume a single-writer model.

Framework integrations and their limits

The README lists drop-in replacements for in-memory vector stores in four frameworks: LangChain, LlamaIndex, Haystack, and Agno. Each has an extra install: pip install turbovec[langchain], turbovec[llama-index], turbovec[haystack], and turbovec[agno]. The claim is that they replace the in-tree reference stores with the same public surface and persistence semantics. For LangChain, it replaces langchain_core.vectorstores.InMemoryVectorStore; for LlamaIndex, SimpleVectorStore; for Haystack, InMemoryDocumentStore; for Agno, LanceDb. This is a practical convenience if you already use one of these frameworks. But the integrations are only as good as the underlying index. If the framework expects a certain method or behavior that turbovec does not implement, you will find out at runtime. The README does not provide code examples for these integrations, only links to docs. You should read those docs before assuming the drop-in claim holds for your version of the framework. Also, the Agno replacement targets LanceDb, which is a different kind of store with its own features; the swap may lose functionality such as metadata filtering or vector type support.

Maintenance, license, and what to verify before adopting

The project is MIT licensed, which is permissive and allows commercial use with attribution. The repository is not archived, but there are no recent releases listed and the last push date is unknown. That is a yellow flag for a project you might depend on. The README references an arXiv paper (2504.19874) for TurboQuant, which gives you a foundation for understanding the algorithm, but the turbovec implementation itself is the thing you are adopting. There is no mention of a changelog or versioning policy. For maintenance cost, you should expect to track upstream changes to the Rust crate and the Python binding. The API is small, so upgrading is likely manageable, but the lack of release history means you cannot assess stability from version numbers. Before adopting, verify that the Python bindings work with your Python version and that the SIMD kernels actually compile on your target CPU. The README mentions AVX2 and scalar fallbacks, so it should run on older hardware, but you should test the search latency yourself. Also, check the docs/api.md for the full method list, because the README only shows a subset.

Editorial conclusion

Adopt turbovec if you need a local, air-gapped vector index with online ingest, filtered search, and incremental persistence, and you can accept a quantizer that skips the training phase. Skip it if you need the highest recall at low dimensions or if your workload demands a managed service. Before committing, verify recall on your own corpus against FAISS IndexPQ, check the bit-width and dimension constraints, and confirm the Python bindings accept only float32 input. The project's own recall numbers show it trails FAISS at d=1536 4-bit and at 2-bit for GloVe d=200, so measure, don't assume.

Official sources

1. Official documentation
2. Official README
3. Project repository