ClickBench is a reproducible 43 query benchmark for analytical databases

One dataset, one table, 43 queries

ClickBench is a benchmark for analytical databases hosted by ClickHouse, and its scope is narrower than the name suggests. The workload represents clickstream and traffic analysis, web analytics, machine generated data, structured logs and events data, and it covers the typical queries in ad hoc analytics and real time dashboards.

The dataset came from the actual traffic recording of one of the world's largest web analytics platforms, anonymized while keeping the essential distributions of the data. That origin is the reason for the realism claim, and the reasoning is specific: the realistic data distributions let you evaluate compression, indices, codecs and custom data structures, which random dataset generators prevent you from doing. The queries themselves were improvised to reflect realistic workloads and are not taken from production.

The size is fixed. The table consists of exactly 99'997'497 records, which the README calls rather small by modern standards but which allows tests to be completed in reasonable time. The workload is 43 queries, and the README says they test the efficiency of full scan and filtered scan, as well as index lookups and the main relational operations.

Because the data is derived from real traffic, the queries exercise different hardware characteristics in different ways. Some require high storage throughput, some benefit from a large number of CPU cores, some from single core speed, and some from high main memory bandwidth. That is the property that makes the benchmark useful for hardware comparisons as well as engine comparisons.

Adding a system means writing a shell script with a fixed output format

The contribution rules are unusually precise, and that precision is what makes the results comparable. To add a new system you copy one of the existing directories and edit the files. The main file is benchmark.sh, the script that runs the benchmark on a fresh VM, where Ubuntu 24.04 or newer should be used by default. For databases that can be installed locally the script is expected to run fully automatically, so it can be used in the benchmark automation through cloud-init.

The output format is prescribed. The script must print one or more lines of the form Load time: 1234 with the time in seconds, a line Data size: 1234567890 with the size in bytes including indexes and transaction logs where applicable, and then 43 consecutive lines in the form [1.234, 5.678, 9.012], for the runtimes of every query. Other lines may appear as logs and are not used for the report. For managed databases where the setup requires clicking through a user interface, the instructions say to write a README.md instead of a shell script.

The rest of the directory is a decomposition of the same run. The scripts check, data-size, install, load, query, start and stop each perform a sub task, with an overview in lib/benchmark-common.sh. There is a create.sql holding the CREATE TABLE statement, or a file like wtf.json for a NoSQL system, and a queries.sql with the 43 queries. The run.sh file is the loop that executes them, and every query is run three times.

That file layout is the mechanism behind the reproducibility goal. Because the output is machine readable in a fixed shape, results are collected by script rather than by copy and paste.

Results are files on disk, and history is kept on purpose

Results live in the results directory as JSON files, one per hardware configuration, under a subdirectory named with the UTC date of the run in the form results/YYYYMMDD/<machine>.json. Each new run for an existing machine goes into a new dated subdirectory and the older runs are kept for history, and the website displays the latest dated copy of each system and machine pair. The contribution guide asks you to double check that each file is valid JSON, with no comma errors, which tells you how often that has gone wrong.

The alternative to a new dated subdirectory is a new file inside the same dated directory, and that is the route for a different hardware configuration of an existing system. A different usage scenario is handled differently again: either copy the whole directory and name it differently, for example timescaledb and timescaledb-compression, or add a file under the dated subdirectory. Naming a variant directory is how the benchmark keeps a compression enabled run separate from a plain one rather than overwriting it.

The static site is generated from those files. index.html can be regenerated with ./generate-results.sh, and the CI does this automatically through GitHub Actions, so the step is optional for a contributor. There is also a collect-new-results.py and a collect-results.sh at the root for gathering results, and a CHANGELOG.md.

The scale of the comparison is visible in the directory listing alone. The top level of the repository holds one directory per system, covering self managed OLAP engines, managed services, serverless engines, NoSQL and document stores, and specialised time series databases, including names such as clickhouse, doris, duckdb, datafusion, databend, chdb, drill, dask, bigquery, athena, snowflake, databricks, cockroachdb, citus, cratedb, druid, byconity and many more, often with a -parquet or -partitioned suffix showing a storage format variation of the same engine.

The limitations section is longer than the goals section

What makes this benchmark credible is that it documents its own weaknesses at length. The overview states that the benchmark represents only a subset of all possible workloads and scenarios, and that while it aims to be as fair as possible, focusing on a specific subset of workloads may give an advantage to the systems that specialise in those workloads.

Five limitations are listed. The dataset is a single flat table, which is different from classical data warehouses that use a normalized star or snowflake model, so classical data warehouses may have an unfair disadvantage in ClickHouse. The table is rather small by modern standards, which allows tests to be completed in reasonable time. Most results so far have been obtained on single node setups even though the benchmark supports distributed systems and includes multi node and serverless setups. The queries run one after another, so concurrent query workloads and system capacity are not tested, and every query is run only a few times, which allows some variability in the results. And many setups are different enough to make direct comparison tricky, for example it is not possible to test the efficiency of storage used for in memory databases, or the loading time for stateless query engines.

The README closes that section with a blunt summary, that all benchmarks are liars, and the sentence that matters most for anyone quoting a number: the goal of the benchmark is to produce numbers, and you need to interpret them yourself.

One of those limitations is being addressed. The shared driver lib/benchmark-common.sh runs a supplementary concurrent QPS test after the main sweep, with BENCH_CONCURRENT_CONNECTIONS workers for BENCH_CONCURRENT_DURATION seconds, and single process engines must set BENCH_CONCURRENT_DURATION=0 in their benchmark.sh to skip it. That is a direct response to the point that concurrency was never tested.

Reproducibility means twenty minutes on a cheap VM

The first stated goal of the benchmark is reproducibility, and the detail that makes it credible is the time budget. The README says every test result can be reproduced quickly, in as little as 20 minutes, although some systems may take several hours, in a semi automated way. The setup is documented and uses inexpensive cloud VMs.

The process itself is documented as a shell script covering the installation of every system, loading the data, running the workload and collecting the result numbers. That is the whole pipeline in one sentence, and it is why the repository is primarily Shell rather than a harness in a language that would be more pleasant to write. The scripts are meant to run unattended in cloud-init, and cloud-init.sh.in at the repository root is the template for that path.

The second goal is compatibility. The tables and queries use mostly standard SQL and require minimum or no adaptation for most SQL database management systems, and the dataset has been filtered to avoid difficulties with parsing and loading. That filtering is a design decision with consequences: it makes more systems comparable, at the cost of the dataset being less of a stress test for parsers.

The third goal is diversity of systems, spanning modern and historical self managed OLAP engines, traditional OLTP systems as a comparison baseline, managed database as a service offerings, serverless cloud native databases, some NoSQL databases, document databases and specialised time series databases, for reference even if they do not specialise on the ClickBench workload.

For a reader deciding whether to trust the site, the point to hold on to is that any number on benchmark.clickhouse.com came from a script that someone else can run again on a comparable machine.

Running one system locally is a shell script and a fresh VM

The README does not publish a single install command for the whole benchmark, because the installation is per system. What it does publish is the contract. Pick a system directory, for example clickhouse/ or duckdb/, and the directory contains the install, load, start, stop, check, data-size and query scripts, plus create.sql and queries.sql. The main entry point is benchmark.sh, which is meant to run on a fresh machine with Ubuntu 24.04 or newer.

Running it produces the output format the collection step expects, which is the load time line, the data size line and the 43 runtime lines.

./generate-results.sh

That command regenerates index.html from the results directory, and the CI does the same automatically, so it is only needed if you are working on the site locally. A new result file belongs at results/YYYYMMDD/<machine>.json, and the README asks for a JSON validity check before you submit.

The dataset is published and made available for download in multiple formats, which is what lets a new system be brought up without reconstructing the data. The result of a local run is a JSON file with the same shape as the ones on the site, and the difference between a contribution and a private experiment is mostly where you put that file.

The project is not archived and the last push was on 2026-10-19, so the system list and the shared driver are still changing. There are no GitHub releases, so the version record is the commit history and the CHANGELOG.md. The licence is one GitHub cannot classify, with the terms in the LICENSE file at the repository root.

Editorial conclusion

Use ClickBench when you want a comparable, reproducible number for a system you can install yourself in twenty minutes, and treat the result as one data point rather than a ranking. Do not use it to decide between a normalized data warehouse and a column store, because the dataset is a single flat table and the README says that classical data warehouses may be at an unfair disadvantage. Verify first that your own queries resemble the 43 in queries.sql, and read the five stated limitations before you quote a number from benchmark.clickhouse.com.

Frequently asked questions

Official sources

1. ClickHouse/ClickBench on GitHub
2. Issues
3. Project website
4. README