qdrant/vector-db-benchmark: a harness for comparing vector engines on one machine
Framework for benchmarking vector search engines
At a glance
- What is it?
- The repository is a Python framework that runs a vector engine in Docker on one host and a client on another, then writes comparable result files. It is built for people who need numbers from their own hardware, not a vendor leaderboard.
- Who is it for?
- Adopt it if you need engine-to-engine numbers on your own hardware and are willing to write a client class for any engine that is not already covered. Do not adopt it if you want a hosted leaderboard or a one-command comparison of managed services; the framework expects you to supply the server and the dataset wiring.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem: every engine reports its own numbers
Vector search engines differ in indexing method, filtering support, and how they behave under concurrent load. The README states the case plainly: there is no clear definition of performance, and a specific deployment may care about one aspect while ignoring others. Vendor benchmarks usually measure on hardware the reader does not own, with parameters the reader cannot see.
This project's answer is to fix the hardware variable. The README describes it as a general framework for benchmarking different engines under the same hardware constraints, so the reader can choose what works best for them. The unit of comparison is an engine plus a dataset plus a scenario. A scenario may assume single or distributed server mode, a different client implementation, and a chosen number of client instances.
That framing matters. The repository is not a leaderboard service. It is a harness you run yourself, and its output is only as meaningful as the configuration you wrote.
Server-client split and the four configuration stages
Benchmarks run in server-client mode. The server runs on one machine, the client on another, and all engines are served through docker compose with configurations under engine/servers/. Containers are expected to expose the ports the client needs.
On the client side, each experiment is described by a configuration file in experiments/configurations/. The README lists four stages, each with its own path in that file: connection_params is passed to the client during connection, collection_params covers collection creation and usually indexing parameters, upload_params controls how data is uploaded, and search_params is passed during the search phase. The framework allows multiple search configurations for the same experiment run, which is how a single upload can be reused across several query parameter sets.
Exact parameter values are individual to each engine, so the configuration files are not portable between engines without editing. Adding an engine means implementing BaseConfigurator, BaseUploader and BaseSearcher, then registering the engine in ClientFactory at engine/clients/client_factory.py. The client implementations under engine/clients/ are the working examples to copy.
Installing the client and running a first benchmark
The client dependencies are managed with Poetry, and pyproject.toml pins Python to >=3.10,<3.13. Install Poetry, then install the project:
pip install poetry
poetry installStart the server for the engine you want to test. The README gives this pattern, where the directory name is one of the configurations under engine/servers/:
cd ./engine/servers/<engine-configuration-name>
docker compose upThen enter the Poetry shell and run the benchmark. run.py takes wildcards for both engines and datasets, and defaults to host localhost:
poetry shell
python run.py --engines "qdrant-rps-m-*-ef-*" --datasets "dbpedia-openai-100K-1536-angular"The README shows a second example that selects every engine and its configurations for the glove datasets using the pattern "*-m-*-ef-*" with "glove-*". Results land in ./results/. Datasets are declared in datasets/datasets.json, and the framework downloads them automatically into the datasets/ directory, so the first run of a large dataset spends time on download before any measurement starts. If the server is already populated, the --skip-upload flag avoids re-uploading.
Where the framework pushes work back onto you
The harness deliberately does not decide what to measure. It records the run; it does not tell you whether a difference is meaningful. If you change collection_params between two engines, or run one with a single client and another with several, the results directory will contain numbers that cannot be compared. The README's scenario concept exists precisely because those choices are yours.
The dependency list is another constraint worth reading before adoption. qdrant-client is pulled from a git branch rather than a released version, which means a fresh poetry install tracks that branch and can change behaviour without a version bump in this repository. The other clients (weaviate-client, elasticsearch, pymilvus, redis, opensearch-py, psycopg with pgvector) are version-pinned ranges, so those are more predictable.
The framework is also not a load generator for production capacity planning. It drives a client against a server under the parameters you wrote, and the README does not document how results are aggregated across repeated runs or how variance is reported. Treat a single run as a data point, not a verdict.
Compared with VectorDBBench
VectorDBBench is the closest well-known tool in this space, and the difference is in where the engine runs. VectorDBBench is commonly used to compare managed and self-hosted services through a single driver, which suits people who want a quick read on a hosted endpoint.
This repository takes the opposite position. Every engine is launched locally through docker compose from engine/servers/, so the hardware is yours and the container image is visible in the repository. The cost is setup: you need Docker on the server host, a reachable network path from the client, and a configuration file per engine per scenario. In exchange, the parameters that produced a result are files in the repository rather than settings inside someone else's service. If you need to benchmark a managed offering you do not control, VectorDBBench fits better. If you need to know how two engines behave on your own machine with your own index parameters, this is the more direct route.
Licence and the cost of keeping up
The repository is Apache-2.0, which permits commercial use and modification with the usual notice and patent terms. That is permissive enough to fork the harness for internal use. This is not legal advice; read LICENSE for the exact terms.
The maintenance picture: the last push to the default branch master was on 2026-09-15. The repository is not archived. There are no releases retrieved, so there is no tagged version to pin against; consumers track master. Because the engine clients are separate implementations, adding or updating an engine means touching its client class and its configuration files, and the qdrant-client git dependency means that particular client can shift independently of this repository. Budget for re-running your baseline whenever you pull, rather than assuming old result files remain comparable.
Editorial conclusion
Adopt it if you need engine-to-engine numbers on your own hardware and are willing to write a client class for any engine that is not already covered. Do not adopt it if you want a hosted leaderboard or a one-command comparison of managed services; the framework expects you to supply the server and the dataset wiring. Before trusting any number, verify that the dataset in datasets/datasets.json matches your own vector dimension and distance metric, and check whether the engine configuration you picked uses the same collection_params and search_params as the one you are comparing against.
Frequently asked questions
How do I run a qdrant/vector-db-benchmark test?
Start the engine server with docker compose from its directory under engine/servers/, then install the client with poetry install and run python run.py with --engines and --datasets wildcards. Results are written to the ./results/ directory.
Which vector engines does qdrant/vector-db-benchmark support?
The client dependencies cover Qdrant, Weaviate, Elasticsearch, Milvus, Redis, OpenSearch and pgvector, with client implementations under engine/clients/. New engines are added by implementing BaseConfigurator, BaseUploader and BaseSearcher and registering them in ClientFactory.
Where do the datasets for qdrant/vector-db-benchmark come from?
Datasets are declared in datasets/datasets.json, and the framework downloads them automatically into the datasets/ directory. The README's example uses dbpedia-openai-100K-1536-angular and the glove datasets.
Community notes