the-benchmarker/web-frameworks: a reproducible HTTP benchmark, not a framework ranking

What the benchmark actually measures, and who it is for

The repository answers a narrow question: how fast does a framework serve three trivial routes when nothing else is happening? Every implementation listens on port 3000 and returns the same responses. GET / returns an empty 2xx body, GET /user/:id echoes the id path parameter, and POST /user returns an empty 2xx body. That is the whole contract. Anything a framework does beyond routing and serialising a path parameter is out of scope.

The audience follows from that. If you are choosing between two HTTP routers in the same language and want to know which one adds less overhead per request, this is the right shape of test. If you are choosing a framework for a product with authentication, an ORM and background jobs, the benchmark will not help you, and the README says so directly: these are minimal HTTP throughput and latency tests, not a substitute for profiling a production application. Framework features, maintainability, ecosystem, security and database access are named as things the numbers do not capture.

The collected fields are requests per second, total data received, run duration, and the p50, p75, p90 and p99 latency percentiles. Percentiles matter more than the headline rate here, because a framework that wins on requests per second while showing a fat p99 tail behaves differently under real traffic than the average suggests.

One contract, three YAML layers, and a generated Makefile per framework

Each implementation lives in a directory named <language>/<framework>/ and contains a config.yaml describing its version, website, files, startup command and engine variants, plus the smallest application that satisfies the route contract and whatever dependency manifests the container build needs. The benchmark configuration is assembled from three YAML layers. config.yaml at the repository root carries provider and global settings, <language>/config.yaml carries the language image, runtime and engine settings, and <language>/<framework>/config.yaml carries the framework version, files, command and overrides. Later layers override earlier ones, which is how a single framework can be tested against several runtimes or server variants without duplicating the whole configuration.

Before any implementation is measured, the shared RSpec contract in .spec/route_spec.rb verifies the three routes. That check is what keeps the comparison honest: a framework that returns the wrong body or the wrong status never reaches the load generator. The default generated benchmark runs GET / for 15 seconds, disables keep-alive, applies latency correction, and records an oha JSON report. Concurrency and routes are configurable, and the results land under <language>/<framework>/.results/.

The generated Makefile is the interface. The root Makefile only offers a clean target that removes leftover ip-*.txt, .Dockerfile* and cid-*.txt files, so the cleanup path is intentionally blunt.

Running one implementation locally

The requirements are Git, Docker, Ruby and Bundler (CI currently uses Ruby 4), zrk on PATH, and jq. Install the Ruby dependencies and generate the Dockerfiles and Makefiles first. The rake task reads the three YAML layers and writes a .Makefile into each framework directory.

git clone https://github.com/the-benchmarker/web-frameworks.git
cd web-frameworks
bundle install
bundle exec rake config

Pick an implementation, for example javascript/fastify, and drive it through its generated Makefile. Build creates the container, test runs the RSpec route contract, warmup primes the process, and collect records the measurements. Results are written beneath the framework's .results/ directory. The unbuild target removes the container again, and clean removes the framework container.

make -f javascript/fastify/.Makefile build
make -f javascript/fastify/.Makefile test
make -f javascript/fastify/.Makefile warmup
mkdir -p javascript/fastify/.results/10
make -f javascript/fastify/.Makefile collect
make -f javascript/fastify/.Makefile unbuild

Concurrency levels and routes are set at generation time, not at collection time, so changing them means regenerating the Makefiles. Route entries use the METHOD:/path form.

CONCURRENCIES=64,256,512 \
ROUTES='GET:/,GET:/user/42,POST:/user' \
bundle exec rake config

One detail catches people out: you must create a matching .results/<concurrency> directory for every configured concurrency level before collecting. The batch runner does this automatically for its predefined levels, but a custom run does not.

The cost of a full run, and why the numbers age badly

The README carries a caution that a full benchmark consumes substantial CPU, memory, time, network bandwidth and container storage, and advises starting with one implementation while keeping the load generator separate from services you care about. That is not boilerplate. Hundreds of implementations, each in its own container, each generating load for at least 15 seconds per route and concurrency level, adds up quickly, and the load generator competes for the same CPU as the service under test if you co-locate them.

The harder limitation is comparability. A result only means something alongside its framework version, runtime or server variant, concurrency, benchmark revision and hardware. Two numbers from different revisions or different machines are not a ranking, they are two unrelated measurements. The README's own guidance is to compare runs produced by the same revision and machine, to check the selected runtime or server variant rather than only the framework name, to treat small differences as noise until repeated, and not to extrapolate to database-heavy workloads.

That last point deserves emphasis. The roadmap lists replacing the load generator, optimising the implementations themselves, building real infrastructure on a public cloud, and checking HTTP compliance per framework as open items. Until those land, treat the published figures as a measurement of a minimal API under controlled conditions, not as a verdict on any framework's production behaviour.

TechEmpower and the design difference

The obvious comparison is TechEmpower's Framework Benchmarks, which also runs many frameworks behind a shared specification in containers. The difference is in the specification's breadth. TechEmpower's suite covers many test types, including database access and multi-query workloads, which is what makes its results usable for people who care about the data layer. the-benchmarker/web-frameworks deliberately keeps one small API: three routes, no database, no templating, no serialisation beyond echoing a path parameter.

That narrowness is the point. A three-route contract is cheap to implement correctly, which lowers the barrier for adding a framework, and it isolates HTTP handling from everything else, which makes a regression attributable to the framework or its runtime rather than to a query plan. The trade-off is that the results say nothing about the layers where most production time is spent. If your question is which framework handles a JSON API against a database best, this repository does not answer it, and its README says explicitly that the numbers should not be extrapolated to database-heavy or full application workloads. If your question is which HTTP layer adds least overhead, the narrower contract gives you a cleaner signal.

The project also states it is independent and not affiliated with or endorsed by Gartner, in reference to the quadrant-style presentation it intends to build for a framework decision model. That model does not exist yet; the roadmap lists it as an unchecked item.

Maintenance, licence and what a contribution costs

The repository is not archived and the last push was on 2026-09-21, so the tree is current. The release tags are another matter: the most recent is 2018.9.26 from 2018-09-26, followed by 2018.7.30 and 2018.6.12. Tagged releases have not been cut for years, so if your workflow pins a version, there is no recent tag to pin. Track the develop branch, which is the default branch, and expect the configuration format to move with it.

The project is MIT licensed, which permits use and modification with the licence and copyright notice retained. That is a statement about the licence text, not legal advice about your situation; if you redistribute modified benchmark results or embed the tooling, read the LICENSE file and, where the stakes justify it, talk to someone qualified.

Adding or updating a framework means writing a config.yaml with version, website, files, startup command and engine variants, plus the smallest application that fulfils the route contract and the dependency manifests needed for a reproducible container build. Validation is the same three targets used for running: generate the config, build, test, unbuild. CONTRIBUTING.md is the document to read before opening a pull request, and the README notes that if you maintain the framework you are adding, saying so in the PR helps reviewers judge whether the configuration is idiomatic. The upgrade cost of the benchmark itself is the Ruby and Docker toolchain plus zrk on PATH; the upgrade cost of a framework entry is whatever changed in that framework's startup command or container base.

Editorial conclusion

Adopt it if you need a reproducible, container-isolated comparison of minimal HTTP handling across languages and you are willing to run one implementation at a time on a machine you can afford to load. Do not adopt it to pick a framework for a database-backed product; the contract is three routes with no database, and the README says the numbers must not be extrapolated to full application workloads. Before you trust a number, verify the benchmark revision, the machine, the runtime or server variant and the concurrency level behind it, because results are only comparable within the same revision and hardware.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. the-benchmarker/web-frameworks on GitHub