Time Series Benchmark Suite: database comparison and first dataset

What TSBS is designed to measure

Time Series Benchmark Suite addresses a specific problem in time series database evaluation: measuring systems with the same generated input instead of allowing each database to receive differently shaped or differently timed data. It is intended for database administrators seeking a database for a particular workload and for time series database developers who want to add another database implementation.

The project separates evaluation into three phases. First, it generates data and queries. Second, it loads the generated data into a selected database. Third, it executes the generated queries and records execution results. This separation keeps generation work outside the measured loading and query phases.

TSBS covers bulk loading and query execution for systems including TimescaleDB, MongoDB, InfluxDB, CrateDB, Cassandra, ClickHouse, QuestDB, SiriDB, Akumuli, Timestream, and VictoriaMetrics. Support is not uniform. The README lists Dev ops support for every listed database, but only InfluxDB, QuestDB, and TimescaleDB list IoT support. VictoriaMetrics has documented query exclusions, while Akumuli lacks the `groupby-orderby-limit` query.

The two built-in workloads have different shapes. Dev ops can model nine monitored systems that produce 100 metrics per interval, or a simpler CPU-only form that produces 10 metrics per reading. Host tags identify each generated device. IoT models trucks, including diagnostics, metrics, metadata, batch ingestion, and out-of-order data. Its scale is the number of trucks rather than an unspecified row total.

Deterministic data and query flow

TSBS uses a pseudo-random number generator seed to produce deterministic data and queries. Given the same seed and generation settings, each target database is intended to receive identical data and run identical queries. That property is the main reason to use TSBS instead of ad hoc SQL scripts whose input may change between runs.

Data generation still represents a substantial workload even though it is performed first. The README's TimescaleDB example combines 4,000 trucks with a three-day interval from `2016-01-01T00:00:00Z` to `2016-01-04T00:00:00Z`, recording one reading every 10 seconds. According to the README, this produces just over 100 million rows, representing about one billion metrics. Extending the period by one day adds roughly 33 million rows. Compressing output reduces disk consumption, but it does not reduce the final database load.

Each target has its own pseudo-CSV or client-specific output format so that its native loader can ingest the generated values efficiently. TSBS uses native Go clients where possible, including `mgo` for MongoDB and the AWS SDK for Timestream. The generated query set contains workload-specific reads, and the implementation defines whether a database executes each query in full, excludes it, or only supports the use case partially.

This model is intentionally deterministic, not stochastic in its reproduction. Random-looking values are generated, but the seed makes the sequence repeatable. The README does not document a database-neutral results schema, so operators should not assume that loader speed alone represents application performance.

Building TSBS with the Go toolchain

The README describes fetching the repository under a `GOPATH` layout, changing into its source directory, and running `make`. The Makefile's default `all` target builds generators, loaders, and query runners, placing project binaries under `bin/` while also installing them through Go tooling.

Use the documented sequence below in a Go development environment configured for this repository:

$ go get github.com/timescale/tsbs
$ cd $GOPATH/src/github.com/timescale/tsbs
$ make

The build should produce the generator and loader command families listed by the Makefile, including `tsbs_generate_data`, `tsbs_generate_queries`, `tsbs_load_timescaledb`, and `tsbs_run_queries_timescaledb`. That statement follows the Makefile targets; it is not a claim that the project has been built here.

The module declares `go 1.14`, while the Dockerfile uses `golang:1.13.1-alpine` as its builder. That version difference is worth checking before adopting the container path. The README does not state the latest supported Go release, pin a tested modern toolchain, or describe an upgrade procedure for the commands. The repository has no GitHub releases, so the commit state is the relevant source for dependency and toolchain review.

The Dockerfile copies built binaries and `scripts/` into an Alpine image. Its entry point starts a shell that waits without launching a benchmark daemon, so this is an execution environment rather than an application service. Anyone automating TSBS must provide the actual generation, loading, and runner invocations.

Generating the first TimescaleDB dataset

Start with data generation because loading requires its output and query generation is a separate phase. The following is the complete documented TimescaleDB example. It selects the IoT use case, seed 123, 4,000 trucks, a fixed time range, and a 10-second logging interval.

$ tsbs_generate_data --use-case="iot" --seed=123 --scale=4000 \
    --timestamp-start="2016-01-01T00:00:00Z" \
    --timestamp-end="2016-01-04T00:00:00Z" \
    --log-interval="10s" --format="timescaledb" \
    | gzip > /tmp/timescaledb-data.gz

After the command finishes, the compressed output is located at `/tmp/timescaledb-data.gz`. The README warns that a loader must pipe this output through `gunzip`. It also states that each additional database requires a separate generation call so that the output matches that database's preferred format.

This is a real workload rather than a lightweight demonstration. The documented parameters are expected to create just over 100 million rows. Before choosing them, estimate storage, ingestion time, and decompression requirements. A smaller `--scale` or shorter interval can reduce the dataset, but the README does not publish a quick-start parameter set beyond these examples, so no undocumented smoke-test command should be substituted.

The first run should preserve the seed and settings. Changing the seed can produce a different deterministic dataset, which would make comparisons with existing results harder to interpret.

Database compatibility limits the comparison

TSBS is not a universal conformance suite, and partial coverage can distort how its results should be read. The support matrix is explicit. InfluxDB, QuestDB, and TimescaleDB implement both Dev ops and IoT, while the other listed systems implement only Dev ops. Cassandra, ClickHouse, CrateDB, MongoDB, SiriDB, Timestream, and Akumuli therefore cannot be compared on the documented IoT query workload.

Some differences are narrower but still important. VictoriaMetrics excludes `groupby-orderby-limit`, `lastpoint`, `high-cpu-1`, and `high-cpu-all`. Akumuli excludes `groupby-orderby-limit`. A database with fewer compatible queries has not passed a shared query suite if the result table fails to record those exclusions.

A second boundary is concurrency. The README explicitly says TSBS does not currently measure concurrent insert and query performance. It is therefore the wrong tool for deciding how a database behaves under simultaneous ingestion and user traffic. Its bulk load and query phases should not be described as a mixed-workload or production saturation test.

There is also no claim in the README that TSBS measures storage efficiency, ingestion durability, failover, operational complexity, or query correctness across every implementation. Those dimensions require separate procedures. The project answers a narrower question: how selected supported systems handle a shared generated load and query set.

TSBS compared with database-native benchmarks

The main alternative is a set of database-native benchmark scripts. A TimescaleDB evaluation might use TimescaleDB's own tooling, while an InfluxDB evaluation might use InfluxDB-specific data generation and query procedures. That approach can exercise version-specific tuning and deployment features, but each script may define its own data distribution, host count, timestamp range, client behavior, and query set.

TSBS takes the opposite approach. It generates target-specific files, uses a common seed and use case, and selects from predefined workloads. Native clients still matter: the README names `mgo` for MongoDB and the AWS SDK for Timestream. Thus, TSBS standardizes the benchmark inputs while allowing each target to use an appropriate client path.

This distinction affects result interpretation. Database-native benchmarks may be a better fit when the decision depends on a feature or tuning option outside TSBS's use cases. TSBS is more appropriate when the evaluator needs repeatable inputs across several supported engines and is willing to work within the documented support matrix.

The project is extensible by design. Database-specific loaders, runners, documentation, and query support can be added, but that extensibility also means an absent adapter or query is a real coverage gap. An organization with proprietary queries, custom schemas, or an unsupported database will need to extend TSBS or use another harness.

Maintenance and licensing implications

The repository was last pushed on 2026-10-01, and it is not archived. That confirms recent repository activity, but the project has no GitHub releases. Upgrade planning therefore cannot be based on published release tags or a documented compatibility matrix. Reviewing the current commit, `go.mod`, `go.sum`, client adapters, and any database-specific documentation is more concrete than assuming a formal release cadence.

The repository identifies `github.com/jackc/pgx/v4`, along with database drivers such as `github.com/gocql/gocql`, `github.com/globalsign/mgo`, and `github.com/clickhouse/clickhouse`. Upgrading a system or changing its schema can therefore affect the connection library, serialization format, query support, or client defaults represented in the repository.

TSBS is distributed under the MIT license. The license file permits use, modification, distribution, and sublicensing subject to its conditions. Database engines and external clients retain their own licenses, so an organization deploying a full benchmark environment must account for those components separately. This is a licensing inventory note, not legal advice.

The largest operational cost is likely test data rather than binary size. The documented IoT example can approach billion-scale metric counts quickly. Database provisioning, compressed source files, indexes or materialized storage, repeated loads, and query-result collection all increase the cost of keeping the benchmark environment reproducible.

Editorial conclusion

Database administrators comparing TSBS-supported systems under bulk loading or single-threaded query workloads should adopt the project as a reproducible evaluation harness. It is unsuitable for concurrent mixed-workload testing, and users should first verify native query support, Go toolchain compatibility, storage needs, and the exact client configuration for each candidate database.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. timescale/tsbs on GitHub