# Bencher: continuous benchmarking that runs on identical bare metal

> Bencher is a continuous benchmarking platform that runs your existing benchmarks on the same bare metal hardware locally and in CI, tracks the results over time, and fails a pull request when a real regression appears. The bet is that noisy shared runners, not slow code, are why performance regressions ship.

**bencherdev/bencher** — 🐰 Bencher - Continuous Benchmarking

- Repository: https://github.com/bencherdev/bencher
- Website: https://bencher.dev
- Stars: 926 · Forks: 42
- Language: Rust
- License: NOASSERTION
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/bencherdev-bencher

## The problem is noise, not slow code

Bencher's argument starts from a specific failure mode. Most teams begin with a benchmark comparison script running on GitHub Actions runners, and that approach breaks down when noisy shared CI runners hide real performance regressions. The project documentation puts figures on both sides of that comparison: typical CI runners are described as having more than 30 percent variance, while Bencher Bare Metal runners are described as having less than 2 percent variance. When a number moves, it means something.

The reasoning underneath those figures is what makes the case. Local benchmarks are not reproducible, so every check means stopping work to pull the baseline branch and wait for a comparison, and most engineers skip it. CI runners are shared and noisy, and noisy benchmarks train engineers to ignore alerts. If you cannot tell a real regression from noise, the results are worthless, so teams stop looking.

The examples of what a regression costs are concrete rather than abstract. A database regression on a hot path pages someone at 2am. A compiler regression silently makes every downstream build slower. A browser engine ships a 10 percent paint regression and users notice. A crypto library that adds a microsecond to a handshake breaks an SLA.

The framing at the end of the problem section is the one to keep: for the same reasons that unit tests are run to prevent feature regressions, benchmarks should be run to prevent performance regressions. Performance bugs are bugs.

The intended users are named in the documentation: teams where performance matters, spanning databases, compilers, browsers, runtimes, networking stacks, cryptographic libraries and operating systems.

## Run, track, catch: three steps and the CLI in the middle

The operating model is described in three steps, and it is worth reading as a pipeline rather than a feature list.

Run is where your existing benchmarks execute, locally or in CI, using the exact same bare metal runners and the benchmarking tools you already favour. The bencher CLI orchestrates that run and stores the results. This is the part that makes the platform a wrapper rather than a benchmark framework, and it is the main reason an existing suite can be adopted without rewriting it.

Track is the time dimension. Results are tracked over time, and you monitor, query and graph them in the Bencher web console, using the source branch, testbed, benchmark and measure as the axes of the view. Those four dimensions are the data model of the whole product: a measurement is only meaningful when you know which branch produced it, on which hardware, for which benchmark, in which unit.

Catch is the action. Regressions are caught locally or in CI on the same bare metal hardware, and the README credits state of the art, customizable analytics for detecting a performance regression before it merges.

The documentation also shows what a caught regression looks like from the outside, as a pull request comment. The sample report is a table with rows for the project, the branch and the testbed, followed by an alert line reading Threshold Boundary Limit exceeded, and per benchmark rows showing the measure with its units, a plot link, the benchmark result as a percentage delta, and the upper boundary limit as a percentage. The visible rows are for an Adapter::Json benchmark measured in microseconds.

So the developer experience is a comment on the pull request, not a dashboard to remember to visit.

## A Rust monorepo with separate API, CLI and runner services

Bencher is written in Rust and the repository is a Cargo workspace with resolver version 3. The workspace members list shows the shape of the system.

```toml
[workspace]
members = [
    "services/api",
    "services/cli",
    "services/runner",
    "lib/*",
    "plus/*",
    "tasks/*",
    "xtask",
]
exclude = ["services/benchers", "services/cargo-bencher"]
resolver = "3"
```

Three services sit at the top: the API, the CLI, and the runner. The runner is the piece that executes benchmarks on the bare metal, which is why it is a separate service rather than part of the CLI.

Under lib/ sits a long list of focused crates, each with a single responsibility: bencher_adapter, bencher_boundary, bencher_client, bencher_comment, bencher_config, bencher_context, bencher_endpoint, bencher_json, bencher_logger, bencher_noise, bencher_parser, bencher_plot, bencher_rank, bencher_rbac, bencher_schema, bencher_token and bencher_valid. The names say a lot about the internals. There is a crate for the rank and threshold logic, one for noise handling, one for the boundary detection, and one for plotting. The plus/ directory holds API endpoint crates including api_mcp, and there are separate endpoint crates for auth, checkout, organizations, projects, run, server and users.

The workspace package metadata gives version 0.6.13, edition 2024, homepage bencher.dev, and a license-file pointing at LICENSE.md. Other top level entries include action.yml for the GitHub Action, docker-compose.yml, an AGENTS.md and CLAUDE.md, DEVELOPMENT.md, SECURITY.md, a changelog.md, a rust-toolchain.toml, and an examples directory with C++, eBPF, Go, Python and Rust samples.

The examples directory is the most useful thing for an evaluator, because it shows which languages the adapters handle.

## Running the API and console locally with Docker Compose

The repository ships a docker-compose.yml that starts the API and the web console together, which is the quickest way to see the product without an account.

The API service runs the ghcr.io/bencherdev/bencher-api:latest image, publishes port 6610, and mounts two directories from the host: the configuration directory at ./services/api/etc to /etc/bencher, and the database directory at ./services/api/data to /var/lib/bencher/data.

```yaml
services:
  bencher_api_compose:
    image: ghcr.io/bencherdev/bencher-api:latest
    container_name: bencher_api
    ports:
      - 6610:6610
```

The configuration is supplied through the environment, with BENCHER_CONFIG taking a configuration string or file path and BENCHER_CONFIG_PATH taking a file path, both commented out in the shipped file.

The console service runs the matching bencher-console:latest image on port 3000 and points at the API.

```yaml
  bencher_console_compose:
    image: ghcr.io/bencherdev/bencher-console:latest
    container_name: bencher_console
    ports:
      - 3000:3000
    environment:
      - BENCHER_API_URL=http://127.0.0.1:6610
      - INTERNAL_API_URL=http://host.docker.internal:6610
```

The two URLs are not redundant. BENCHER_API_URL is the address the browser uses to reach the API, and INTERNAL_API_URL is the address the console container uses when it calls the API from inside Docker, which is why it points at host.docker.internal rather than at the API service name. The shipped comment says plainly that INTERNAL_API_URL should not be set when using a remote BENCHER_API_URL such as https://bencher.example.com, because in that case the console reaches the API over the network directly.

This local compose setup is also the self hosting story in miniature: the API and console are separate images with a database directory, so a team that cannot send benchmark data to a hosted service has a path to running the platform itself.

## The GitHub Action and the hosted service boundary

The documentation pushes the GitHub Action as the primary integration, linking it at the top of the README under the heading Use the GitHub Action with your project. The action.yml file at the repository root is the mechanism.

The important thing to understand about the platform's shape is that the bare metal runners are not something you install. They are hosted hardware that Bencher provides, and that is why the README describes the same exact bare metal for local runs and for CI runs. A developer who wants to reproduce a CI number locally needs access to the same machine class, and the hosted runners are what makes that possible. This is the substantive difference between Bencher and a comparison script running on GitHub Actions.

The boundary this creates is a real constraint for some teams. If your organisation requires that benchmark data never leave your infrastructure, the hosted runner model does not satisfy it out of the box. The compose file shows that the API and console can be self hosted, and the repository contains a whole services/ tree, but the documentation in the repository does not describe self hosted bare metal runners, so that piece of the story is not answered here.

There is also a commercial dimension visible in the repository structure. The Cargo workspace has a plus/ directory alongside lib/, and the endpoint crates under it include api_checkout and api_mcp, while the public licence file is LICENSE.md. The workspace version is 0.6.13, and the release list shows v0.6.13 on 2026-09-28 and v0.6.12 on 2026-08-22, with a canary release tagged Runner Canary on 2026-07-18. A canary release named for the runner suggests the runner component is versioned on its own schedule.

The last push was on 2026-10-20, so the tree is being worked on continuously.

## Licence, threat model and what the open repository does not tell you

The licence field on the repository says Other, a custom licence GitHub cannot classify, and the root carries both a LICENSE.md file and a license/ directory, with the Cargo manifest pointing at LICENSE.md through its license-file key. For a platform that ingests benchmark data from other people's repositories, and with a plus/ tree that separates commercial endpoints from the open ones, the licence needs to be read directly rather than inferred from the repository page. This is not legal advice.

Two other documents at the root are worth noting. SECURITY.md exists, which is the expected place to find how vulnerabilities are reported, and deny.toml alongside clippy.toml, which are the configuration files for cargo-deny and clippy, the supply chain and lint checks a Rust project runs in CI.

The docs/ directory holds the published documentation that the README links to, including the bare metal quickstart and the CLI installation page. There is a documentation site at bencher.dev/docs, a signup page, and a community gallery of benchmark reports visible in the pull request comment example.

The interesting omission is what the README does not say. It does not describe what data the hosted service retains, how long it is kept, what the pricing model is, or how the analytic thresholds are configured beyond the word customizable. It also does not document a rollback procedure for a false positive, or how to suppress an alert for a known change. Those details determine how much trust a team places in an automatic gate, and they are answered in the documentation site rather than the repository.

For a team considering making a pull request fail on a regression, the trust question is the whole decision. A gate that cries wolf gets disabled, and a disabled gate is worse than no gate because it creates a false sense of coverage.

## Conclusion

Bencher fits a team working on a database, compiler, browser engine, runtime, networking stack, cryptographic library or operating system, where a performance regression is expensive enough to justify dedicated hardware and a hosted record of past results. It does not fit an application team whose benchmarks are noisy by nature or whose performance targets are not measurable in a microsecond, and the bare metal model means the platform has to be hosted somewhere, not merely installed. Before adopting it, confirm your benchmark tooling can emit results the bencher CLI can ingest, and decide up front whether the hosted service or a self hosted deployment of the API and console fits your data policy.

## FAQ

### What is Bencher and what problem does it solve?

Bencher is a continuous benchmarking platform that runs your existing benchmarks on the same bare metal hardware locally and in CI, tracks the results over time, and detects performance regressions before they merge. It is aimed at teams building databases, compilers, browsers, runtimes, networking stacks, cryptographic libraries and operating systems.

### How does Bencher differ from running benchmarks on GitHub Actions?

The documentation attributes the difference to variance, describing typical CI runners as having more than 30 percent variance and Bencher Bare Metal runners as less than 2 percent. The reasoning is that noisy shared runners make teams ignore alerts, so regressions ship unnoticed.

### Do I need to rewrite my benchmarks to use Bencher?

No. The docs describe running your existing benchmarks with the benchmarking tools you already favour, with the bencher CLI orchestrating the run and storing the results. The examples directory includes projects in Rust, C++, Go, Python and eBPF, and there is a separate cargo-bencher service in the workspace.

### Can I self host Bencher?

The repository ships a docker-compose.yml that runs the API image on port 6610 and the console image on port 3000, with configuration and database directories mounted from the host. The README, however, does not describe self hosted bare metal runners, so the hosted runner side of the platform is the piece not covered there.

### What does a Bencher regression alert look like?

The documentation shows a pull request comment with a table listing the project, branch and testbed, an alert line reading Threshold Boundary Limit exceeded, and rows per benchmark showing the measure with its units, a plot link, the result as a percentage delta and the upper boundary limit as a percentage. The sample rows cover an Adapter::Json benchmark measured in microseconds.

## Sources

- [bencherdev/bencher on GitHub](https://github.com/bencherdev/bencher)
- [Issues](https://github.com/bencherdev/bencher/issues)
- [Project website](https://bencher.dev)
- [README](https://github.com/bencherdev/bencher/blob/main/README.md)
- [Releases](https://github.com/bencherdev/bencher/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/bencherdev-bencher