VectorDBBench: A Repeatable Benchmark Harness for Vector Databases

What VectorDBBench solves, and who is actually supposed to run it

Published vector database benchmarks age badly. A number in a blog post usually comes from a specific instance size, a specific client version and a specific dataset, and the reader has no way to tell which of those changed the result. VectorDBBench attacks that by shipping the workload, the dataset registry and the runner together, so the same command can be pointed at a different endpoint and produce a comparable report.

The README frames the audience explicitly: the tool is meant to help users, including non-professionals, reproduce results or test new systems. That is a wider audience than most benchmark harnesses aim at. The practical reader is an engineer who has to choose between a managed service and a self-hosted deployment, and who needs numbers from their own network path rather than from a vendor's lab.

The project is sponsored by Zilliz, the company behind Milvus, and the README promotes a free test on Zilliz Cloud in the same paragraph that introduces the tool. That is worth stating plainly. The harness is MIT licensed and the database adapters are not restricted to Milvus, but the sponsorship is a fact a reader should weigh when reading any leaderboard the project publishes.

The mechanism: registered datasets, per-database clients, and a case type

The architecture visible in the repository is a client registry plus a dataset registry plus a case runner. Under vectordb_bench/ the package holds the database adapters, and pyproject.toml declares each adapter as an optional dependency group, so the base install pulls PyMilvus and every other backend is opt-in. The base dependencies include pydantic, polars, plotly, hdrhistogram and huggingface_hub, which tells you the shape of the pipeline: configuration is validated, latency is recorded as a histogram rather than as a list of samples, and results are rendered into charts.

Datasets are a first-class concept. The README lists SIFT and GIST corpora, Cohere embeddings, OpenAI embeddings generated from the C4 dataset, and the VIBE and VDBBench multimodal datasets hosted on Hugging Face. They differ in corpus size, dimension, distance metric and query distribution, which is the point: a benchmark that only runs one 1M-vector cosine dataset tells you very little about a 100M-vector filtered workload.

A case type selects the workload. The README names insertion, vector search, filtered search and full-text search, with the September 2026 update adding Hugging Face hosted datasets through the existing Performance case. Full-text search benchmarks BM25-style retrieval, starting with MS MARCO and HotpotQA, and reports recall, QPS and load metrics. Each run writes a result JSON, and the --note or --note-file option stores deployment, resource, client, network and constraint context under task_config.db_config.note. That last detail is the most useful design decision in the tool: the number and the conditions that produced it travel together.

Installing VectorDBBench and running a first benchmark

The README states a prerequisite of python >= 3.11, and the Dockerfile confirms the image is built on python:3.11. The base install brings PyMilvus and the Zilliz Cloud client; every other backend needs its own extra.

pip install vectordb-bench

For a non-Milvus target, install the matching extra instead. The README gives this example for Pinecone, and the same pattern applies to qdrant, weaviate, elastic, pgvector, redis, chromadb, mongodb, vespa and the rest of the table.

pip install 'vectordb-bench[pinecone]'

There are two entry points. init_bench launches the visual interface, and the vectordbbench command runs a single backend from the terminal.

init_bench

To see which clients the CLI exposes, and which flags a given client accepts, the README points at two help commands. The first lists the commands; the second prints the options for one of them, including --config-file, --drop-old/--skip-drop-old, --load/--skip-load and --search-serial.

vectordbbench --help
vectordbbench pgvectorhnsw --help

A real run needs a dataset and a case type. The README's September 2026 example runs the Milvus HNSW client against a Hugging Face hosted dataset through the Performance case, with k set to 100.

vectordbbench milvushnsw \
  --case-type Performance \
  --dataset-name glove-200-cosine \
  --k 100

The .env.example file controls where data lands and how much is logged. DATASET_LOCAL_DIR defaults to /tmp/vectordb_bench/dataset, which is worth changing on a machine with a small root volume, since the larger corpora are measured in millions of vectors.

LOG_LEVEL=INFO
LOG_FILE="logs/vectordb_bench.log"
DATASET_LOCAL_DIR="/tmp/vectordb_bench/dataset"

Before any of that, decide how you will record the deployment context. The README recommends --note-file over --note for structured or multiline context and warns against putting credentials, tokens or sensitive connection details in either.

vectordbbench zillizautoindex \
  --note-file ./run-context.json \
  <other options>

Expect a run to take real time and real disk. The tool downloads and caches datasets locally, loads them into the target system, then issues the search workload. Nothing in the README suggests a way to run a partial load and extrapolate, so plan for the full corpus.

Where VectorDBBench will mislead you

The biggest limitation is the one the README never addresses: it is a client-side harness, so the numbers it produces include your network path to the database. A managed service benchmarked from a laptop over a home connection and the same service benchmarked from a VM in the same region will produce different QPS, and the result JSON will not necessarily make that obvious unless you filled in the note field. The --note-file option exists precisely because the harness cannot infer this for you.

Cost-effectiveness reporting is scoped to cloud services. The README says so directly. If you are comparing two self-hosted deployments, the cost dimension the tool is proudest of does not apply, and you are left with the performance half.

Coverage is uneven by design. The optional dependency table lists around thirty backends, but the CLI help output the README shows only a handful of commands, and the full-text search release note is described as having caveats. A backend appearing in the install table is not the same as that backend being exercised by every case type. The Hugging Face dataset support is likewise scoped to the Performance case, so a filtered-search run against a VIBE dataset is not something the September 2026 note claims.

Finally, this is a benchmark harness, not an observability tool. It runs a workload and writes a result. If you want continuous production latency tracking, this is the wrong instrument, and the repository does not present it as one.

How VectorDBBench differs from Ann-benchmarks

Ann-benchmarks is the obvious comparison, and the difference is architectural rather than cosmetic. Ann-benchmarks is built around a fixed set of approximate nearest neighbour datasets and algorithm implementations, and it is designed to produce a static, reproducible comparison table across many algorithms. The unit of comparison is the algorithm.

VectorDBBench inverts that. The unit of comparison is a running database service, reached over a client library. That is why the install surface is a table of pip extras rather than a set of algorithm bindings, and why the cost-effectiveness report exists at all: you cannot price an algorithm, but you can price a managed instance. The trade-off is that Ann-benchmarks gives you a cleaner isolation of the index from the system, while VectorDBBench gives you a number that includes the connection, the client and the deployment you actually intend to run.

If your question is which HNSW variant has the better recall curve, Ann-benchmarks is the better fit. If your question is whether to move a workload from a self-hosted pgvector deployment to a managed service, VectorDBBench is built for exactly that question, and its pgvector extra covers pgvector, pgvectorscale, pgdiskann, alloydb, vectorchord and lakebase_vector under one install.

Maintenance, licence and the cost of keeping up

The repository is not archived and the last push was on 2026-09-11, four days before this writing, with v2.0.0 released on 2026-09-10. The release cadence visible in the notes runs from v1.0.21 in April 2026 through v1.0.22 in May to v2.0.0 in September, so the project is moving, and the 2.0 major version bump suggests the CLI or the result format changed in a way that may not be backward compatible. Anyone pinning a version for a published comparison should check that before upgrading.

The licence is MIT, declared both in the repository metadata and in pyproject.toml as license = "MIT". MIT is permissive and imposes no copyleft obligation on your own code. It does not, however, cover the datasets. SIFT and GIST come from an external corpus host, Cohere embeddings come from a Hugging Face dataset, OpenAI embeddings are generated from the C4 dataset, and the VIBE and VDBBench multimodal datasets are hosted on Hugging Face. Each of those has its own terms, and the README does not consolidate them. If you plan to redistribute results or derived data, check the dataset provenance separately from the tool's licence. This is not legal advice.

The upgrade cost is mostly environmental. The base dependency list pins pymilvus>=2.6.15,<3.0.0 and pydantic>=2.0,<3, and the README states python >= 3.11. A team on Python 3.10 cannot install the package at all without upgrading the interpreter, which in practice is the largest single cost of adopting this tool inside an existing service repository.

Editorial conclusion

Adopt VectorDBBench if you need a reproducible insert and search workload against your own deployment and want the result JSON to carry the deployment context alongside the numbers. Do not adopt it if you need an audited, vendor-neutral leaderboard, or if you cannot install a per-database client extra and download the dataset you intend to use. Before committing, verify three things: that python is 3.11 or newer, that the extra matching your target database installs cleanly, and that the dataset you want is registered for the case type you plan to run, since the Performance case and the Hugging Face catalog are documented separately.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. zilliztech/VectorDBBench on GitHub