skyzh/vector-db-from-scratch: a Rust and DataFusion course that builds vector search into SQL
Learn vector search with Rust and DataFusion; the C++/BusTub track is deprecated.
At a glance
- What is it?
- This repository is a six-day guided course, not a library. You build an in-memory vector database in Rust, then expose IVFFlat, NSW, HNSW and residual IVF-PQ indexes through DataFusion SQL, with SIFT1M benchmarks for recall, build time and latency.
- Who is it for?
- Adopt this if you already write Rust and want to understand where ANN indexes sit in a query planner, because the course walks the optimizer rule and execution path rather than wrapping an ANN crate. Skip it if you need a deployable vector store, or if you want a maintained C++ path: the BusTub edition is deprecated and unmaintained.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 4 days ago.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
Who the course is for, and the gap it targets
Most vector search material starts at an HTTP API. You send embeddings to an endpoint, get neighbours back, and never see the index. That is fine for shipping a feature and useless for anyone who has to decide where an ANN index lives inside a database: which operator consumes it, when the planner picks it, and what recall you trade for latency.
This repository sits on that boundary. The README states the course "covers the algorithms, evaluation contracts, query planning, and execution path instead of hiding them behind an HTTP API or ANN library". The stated audience is systems and database engineers. Basic Rust is required; prior vector search or DataFusion knowledge is not.
The scope is deliberately small. The output is an in-memory table backed by Arrow, not a storage engine with persistence, replication or a wire protocol. If you want to learn why a query planner would rewrite a scan into an index probe, that narrow scope is the point. If you want a database to run in production, it is the limitation.
The DataFusion optimizer rule is the real subject
The architecture has two halves that meet in one place. Below the SQL layer sits an in-memory table whose columns are Arrow arrays, so vector columns are just fixed-size lists. Above it sits DataFusion, which parses SQL and produces a logical plan.
The interesting part is the optimizer rule the README describes as a "safe DataFusion optimizer rule". It rewrites a plan that would scan and score every row into one that consults an index. Everything else in the course exists to make that rewrite meaningful: IVFFlat partitions the space into coarse cells, NSW and HNSW add graph links so a search walks neighbours instead of cells, and residual IVF-PQ compresses the residual vectors after coarse assignment to cut memory.
The word to watch is safe. A rewrite that changes results silently is a bug, not an optimization. The course pairs approximate search with exact search so you can compare the two, and the shared benchmark tracks recall alongside build time and search latency. Recall is the contract that keeps the rewrite honest.
The workspace layout mirrors this split. Cargo.toml lists vector-db/core and vector-db/datafusion as separate members, with vector-db-starter/core and vector-db-starter/datafusion as the starting skeletons and vector-db-benchmark-support as a shared crate. The lints section sets unsafe_code to "forbid" across the workspace, so the index implementations cannot reach for unsafe pointer tricks when a graph traversal gets awkward.
Installing the toolchain and running the first checkpoint
There are no published binaries and no crate to add as a dependency. The README points at the online book, which contains "the setup, checkpoint commands, SQL walkthroughs, and benchmark instructions". The repository pins its toolchain in rust-toolchain.toml, so the reliable path is to clone and let rustup honour that file.
Start by checking out the repository and confirming the pinned toolchain resolves. The workspace declares edition 2024 and rust-version 1.88 in Cargo.toml, so an older compiler will fail before you reach any exercise.
git clone https://github.com/skyzh/vector-db-from-scratch
cd vector-db-from-scratch
rustup showrustup show prints the active toolchain for the directory. If it matches the pinned one, the workspace should resolve. The repository uses git submodules for the deprecated BusTub editions, so a plain clone leaves those directories empty; you do not need them for the Rust track.
From there, work through the book day by day. Each day adds one capability and ships "starter code, focused tests, and a completed reference", so the loop is: read the day's page, edit the starter crate, run its tests, then diff against the reference implementation. The reference lives under vector-db/ and the skeleton under vector-db-starter/, both members of the same workspace, which means a single cargo test at the root will try to build both trees.
Where this course stops being the right tool
The database is in-memory. Nothing in the README describes persistence, crash recovery, WAL, replication or concurrent write paths, so do not read the finished project as a storage engine. Restarting the process loses the data.
The benchmark is single-dataset. SIFT1M is the only corpus named, and the shared benchmark measures recall, build time and search latency on it. Results on a million 128-dimensional vectors say little about your own embedding distribution, your dimensionality, or your filter selectivity. Treat the numbers as a way to compare the four index types against each other, not as a capacity plan.
The C++/BusTub edition is explicitly "deprecated and unmaintained" in the README, though it "remains online for existing readers". If your team is a C++ shop, this repository no longer serves you. The two submodules, bustub-vectordb-starter and bustub-vectordb-solution, also "retain their own upstream license terms", so they are not covered by the repository's Apache-2.0 grant.
Finally, the course teaches one integration pattern: an optimizer rule inside DataFusion. If your system does not plan queries through DataFusion, the SQL half does not transfer directly. The index algorithms do, but you will be reimplementing the plumbing.
DataFusion versus a standalone ANN library
The obvious alternative is to skip the database layer and use a dedicated ANN library directly: build an HNSW graph over your vectors, call search, get ids back. That is less code and fewer moving parts, and for a single-process service with one index it is often the correct choice.
The difference in approach is where the filtering happens. A standalone index knows about vectors and ids only. Any predicate, join or projection lives in your application, so you either pre-filter ids (and lose the index's pruning) or post-filter results (and lose recall). This course takes the other route: the index is a physical operator that the optimizer inserts into a plan, so the predicate and the vector search are planned together. The README frames the goal as comparing query plans through SQL commands for creating indexes.
That is a real architectural difference, not a packaging one. It also explains why the course spends time on a "safe" optimizer rule: once the index is part of a plan, correctness is a planner problem. If you only ever need top-k over the whole table, the standalone library is simpler and this course is overhead.
Maintenance, upgrade cost and licensing
The last push to the default branch was on 2026-09-12, three days before this writing, and the repository is not archived. The workspace version is 0.2.0-alpha.1, which the version string itself marks as pre-release.
The upgrade cost is concentrated in one dependency. DataFusion is pinned to version 54 in the workspace dependencies, and the vendored datafusion-cli is 54.1.0. DataFusion moves quickly and its logical plan and optimizer APIs are the ones this course teaches against, so a major bump will likely touch the optimizer rule and the physical operator rather than the index algorithms. The old C++ track is a preview of what happens when a second implementation is not maintained: it stays online, and it drifts.
Licensing is split. The code is Apache-2.0, which is permissive and imposes no copyleft on your own work. The book, including its Markdown and figures, is CC BY-NC-SA 4.0, which is non-commercial and share-alike, so reusing the written material in a paid course or internal training deck is a different question from reusing the code. The BusTub submodules keep their own upstream terms. This is a description of the stated licences, not legal advice; check the LICENSE file and the submodule licences before redistributing anything.
Editorial conclusion
Adopt this if you already write Rust and want to understand where ANN indexes sit in a query planner, because the course walks the optimizer rule and execution path rather than wrapping an ANN crate. Skip it if you need a deployable vector store, or if you want a maintained C++ path: the BusTub edition is deprecated and unmaintained. Before starting, check rust-toolchain.toml for the pinned toolchain and confirm your checkout includes the course/ directory and both vector-db and vector-db-starter workspaces, since the starter and reference trees are separate members of the same Cargo workspace.
Frequently asked questions
What exactly is a vector database?
In this course it is an in-memory table backed by Arrow plus indexes that answer nearest-neighbour queries, exposed through DataFusion SQL. The README describes the focus as the boundary where vector-search algorithms become database features, which is why the optimizer rule and execution path matter as much as the index algorithms.
How to make a vector db?
Work through the six-day guided Rust course in the repository, which the README says has starter code, focused tests and a completed reference for each day. By the end you have an Arrow-backed table, IVFFlat, NSW, HNSW and residual IVF-PQ indexes, SQL commands for creating indexes and comparing query plans, and a SIFT1M benchmark.
Is SQL a vector database?
No. SQL is the query language here, and DataFusion is the engine that parses it; the vector search comes from the indexes the course builds and the optimizer rule that inserts them into the plan. The README frames the goal as using the resulting indexes from SQL through DataFusion.
Is MongoDB a vector DB?
The repository does not discuss MongoDB, so nothing in it supports an answer either way. The only comparison it makes is between building vector search into a DataFusion query plan and calling a standalone ANN library directly.
Community notes