# vector-db-benchmark: Testing Vector Search Engines on Equal Ground

> vector-db-benchmark is an open-source Python framework from Qdrant for running controlled performance tests across multiple vector search engines on shared hardware, so engineers can compare Qdrant, Milvus, pgvector, Redis, and others without relying on vendor-published numbers.

**qdrant/vector-db-benchmark** — Framework for benchmarking vector search engines

- Repository: https://github.com/qdrant/vector-db-benchmark
- Website: https://qdrant.tech/benchmarks/
- Stars: 374 · Forks: 156
- Language: Python
- License: Apache-2.0
- Published: 2026-09-15 · Updated: 2026-09-15 · Language: en
- Canonical page: https://hysenlabs.com/projects/qdrant-vector-db-benchmark

## The Problem: No Shared Baseline for Vector Database Performance

Every vector search engine vendor publishes performance numbers, but each uses different hardware, different datasets, and different query configurations. An engineer comparing Qdrant against Milvus for a recommendation system has no independent baseline that controls for those variables. vector-db-benchmark addresses this by running any supported engine against the same datasets on the same hardware pair, with the same client load profile and the same collection parameters.

The project is built for engineers in the engine selection phase, particularly those evaluating approximate nearest neighbor implementations. It is not a monitoring tool, a load tester, or a production profiling framework. The published benchmark results at qdrant.tech/benchmarks are generated with this framework, which gives a sense of the kind of output it produces.

## Server-Client Architecture and the Engines It Covers

vector-db-benchmark separates the engine under test from the benchmark client across two machines. The server runs the engine in a Docker Compose container; the client runs the Python harness, connects over the network, uploads vectors, and executes search queries. This separation prevents the measurement code from competing with the engine for CPU and memory on the same host.

The pyproject.toml lists the Python client libraries bundled with the project: qdrant-client from the dev branch, Weaviate 4.22.x to 4.23, Elasticsearch 9.x, Milvus 2.5.x, Redis 5.x, OpenSearch 3.x, and pgvector 0.5.x for PostgreSQL. Each engine has its own server configuration directory under engine/servers/. The client directory under engine/clients/ holds the Python implementation for each engine, structured around three abstract base classes: BaseConfigurator for collection setup and indexing, BaseUploader for data ingestion, and BaseSearcher for query execution.

## Installing the Client and Running a First Benchmark

The client requires Python 3.10 to 3.12 and Poetry. Install Poetry first:

```bash
pip install poetry
poetry install
```

Start the target engine on the server host using its Docker Compose file:

```bash
cd ./engine/servers/<engine-configuration-name>
docker compose up
```

The containers expose the ports the client expects. Once the server is running, activate the virtual environment and run a benchmark:

```bash
poetry shell
python run.py --engines "qdrant-rps-m-*-ef-*" --datasets "dbpedia-openai-100K-1536-angular"
```

Both flags accept shell wildcards. Omitting them defaults to all registered engines and all registered datasets. The --skip-upload flag lets you reuse data from a previous run and go straight to the search phase, which saves time when iterating on search_params without changing the index. Results are written to the ./results/ directory as JSON files.

## Configuration Files: Tuning Collection, Upload, and Search Parameters

Each engine experiment is defined by a JSON configuration file under experiments/configurations/. The file divides benchmark settings into four named sections.

connection_params holds the host and port used during the connection phase. collection_params covers index type, distance metric, and segment counts. upload_params controls batch size and upload parallelism. search_params is passed to the client during the query phase, and the framework allows multiple search configurations in one experiment run, so you can compare HNSW ef=64 against ef=128 without restarting.

Datasets are registered in datasets/datasets.json. On first use, the framework downloads the dataset automatically and caches it in the datasets/ directory. The existing entries reference well-known public datasets in HDF5 format. Adding a new dataset means adding an entry to that JSON file.

Adding a new engine requires implementing the three base classes and registering the engine in engine/clients/client_factory.py. The clients directory contains examples for all currently supported engines.

## Limitations and Cases Where This Framework Falls Short

The two-machine model assumes a LAN or similar low-latency link between client and server. The Docker Compose files expose ports directly and do not model TLS termination, authentication headers, or VPN gateways. Engineers benchmarking a cloud-hosted managed service will need to adapt the server configuration significantly, and the results will reflect network latency specific to their environment.

The qdrant-client dependency is pinned to a live dev branch on GitHub rather than a versioned PyPI release. A fresh install after an upstream commit can change client behavior silently, which means two runs installed at different times may not be comparable. The tight version windows on other clients (Weaviate 4.22.x to 4.23, Milvus 2.5.x) mean a newer client version requires dependency resolution work before it can be tested.

The project has no GitHub releases and no built-in aggregation tool for comparing runs. The results JSON files accumulate in the results directory without a local viewer. The comparison dashboard at qdrant.tech/benchmarks uses a separate pipeline not included in this repository.

## How It Differs from VectorDBBench

VectorDBBench, maintained by Zilliz, is the other widely referenced open benchmark for vector databases. Both projects use a client-driven load model. The main difference is orientation: VectorDBBench provides a graphical interface and is built for producing shareable leaderboard-style results. vector-db-benchmark is script-driven and oriented toward engineers running private tests on their own hardware with their own configuration variants.

vector-db-benchmark includes pgvector and OpenSearch among its supported clients, which gives coverage of PostgreSQL-based vector search that VectorDBBench does not provide in the same way. The JSON configuration files give finer control over collection and upload parameters per experiment. The trade-off is the absence of a built-in visualization layer and the manual environment setup required for each engine.

## Maintenance and License

The repository is maintained by the Qdrant team. The last push was on 2026-09-25, three days before this writing, which puts it in active development. The license is Apache-2.0, permitting commercial use, modification, and distribution without copyleft requirements.

Upgrade cost is moderate. The dependency stack is pinned tightly in pyproject.toml, and the qdrant-client dev-branch pin is the largest maintenance risk over time. Any breaking change to that upstream branch will affect benchmark behavior. The pre-commit-config.yaml indicates the project uses pre-commit hooks for code quality, which adds a setup step for first-time contributors but keeps the codebase consistent. Python version support covers 3.10 to 3.12 via the pyproject.toml constraint.

## Conclusion

Engineers evaluating a vector database for a new project should clone vector-db-benchmark and run it against their own hardware before choosing an engine. The framework is most useful in the selection phase, not for ongoing monitoring. Before running, verify that the qdrant-client dev-branch dependency has not changed since the last known-good state, and pin your Docker image tags for each engine configuration. The project does not cover TLS-authenticated managed cloud endpoints, so teams relying on hosted services will need to adapt the server configurations manually.

## FAQ

### How do I run vector-db-benchmark against a specific engine and dataset?

Activate the Poetry shell, then call python run.py with --engines set to the engine name pattern and --datasets set to the dataset name. Both accept shell wildcards. Results land in the ./results/ directory as JSON files.

### How do I add a new vector search engine to vector-db-benchmark?

Implement the three base classes, BaseConfigurator, BaseUploader, and BaseSearcher, using the existing clients in engine/clients/ as examples. Then register the engine in engine/clients/client_factory.py.

### Can I skip re-uploading data and run only the search phase?

Yes. Pass --skip-upload to run.py. The framework reuses data already loaded in a previous run and goes straight to the search phase, which is useful when comparing different search_params values without changing the index.

## Sources

- [Issues](https://github.com/qdrant/vector-db-benchmark/issues)
- [License: Apache-2.0](https://github.com/qdrant/vector-db-benchmark/blob/master/LICENSE)
- [Project website](https://qdrant.tech/benchmarks/)
- [qdrant/vector-db-benchmark on GitHub](https://github.com/qdrant/vector-db-benchmark)
- [README](https://github.com/qdrant/vector-db-benchmark/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/qdrant-vector-db-benchmark