Web frameworks benchmark: three routes on port 3000 and the traps around them

Port 3000 and three routes are the whole contract

Every implementation is measured against the same three endpoints, and nothing else. Each one is expected to listen on port 3000 and answer GET / with a 2xx status and an empty body, GET /user/:id with a 2xx status and the id path parameter as the body, and POST /user with a 2xx status and an empty body. Before anything is benchmarked, a shared RSpec contract at .spec/route_spec.rb verifies those routes, so a framework that fails the contract never reaches a leaderboard. That is the entire API surface: an established full-stack platform and a small HTTP router are asked to do the same three things. The documentation is explicit about what that buys you, calling the runs minimal HTTP throughput and latency tests rather than a substitute for profiling a production application.

Configuration arrives as three stacked YAML layers

A number in the published results is the product of three files, and knowing which ones were edited tells you what you are comparing. config.yaml holds provider and global settings. The language layer, <language>/config.yaml, holds the language image, the runtime, and engine settings. The framework layer, <language>/<framework>/config.yaml, holds the framework version, its files, the command to start it, and any overrides. Results should be read with their benchmark revision so runs can be lined up. The practical consequence for a reader: two frameworks with the same name can sit in different rows because the language image, the runtime or an override differs, which is why the guidance insists you check the selected runtime or server variant and not only the framework name.

The default run is GET / for 15 seconds with keep-alive off

The generated benchmark has defaults you should know before you read a result. It hits GET /, runs for 15 seconds using zrk --closed with keep-alive, and records a JSON report. Concurrency and routes are configurable. From that report the collected fields are requests per second, total data received, run duration, and the p50, p75, p90 and p99 latency percentiles. The load generator runs closed loop over keep-alive connections: every connection sends its next request as soon as the previous response arrives, so the reported rate is the framework's sustained throughput across the whole run. That is also why a single 15 second default should not be quoted as a maximum.

Every collect run writes saturation.json beside the rate

Each collect writes .results/<concurrency>/saturation.json, holding the share of its allotted CPU the server actually used and the number of cores it kept busy. Once results are exported to data.json, bundle exec rake db:check_saturation lists the runs where the server stayed below 50%. For a multi-core server that usually means something else, most often the load generator, was the bottleneck, and the rate in that row does not describe the framework at all. This is the one feature of the harness that turns a number into a check, and it costs nothing to look at before you compare two rows.

THREADS, cpusets and LATENCY_RATE decide whose number you get

Three variables decide whether the load generator or the framework is being measured. THREADS sets how many load threads zrk uses, defaulting to the cores in LOAD_CPUS or every host core when unset, because zrk's own default of two threads is far too few to saturate a fast server. SERVER_CPUS and LOAD_CPUS take cpuset specs such as 0-3 and 4-15, pinning the framework container through --cpuset-cpus and the load generator through taskset to disjoint cores so neither steals cycles from the other. LATENCY_RATE adds a second pass per route at a fixed request rate with coordinated-omission correction: 50% means half of each framework's own closed-loop rate, 20000 means the same absolute rate for everyone. Those results land in <route>_latency.json as latency_at_rate_* metrics, separate from the headline rate, and the pass doubles collect time and is off by default.

CONCURRENCIES and ROUTES are set before rake config runs

Running one implementation locally needs Git, Docker, Ruby and Bundler, with CI currently on Ruby 4, zrk on PATH, and jq. The setup generates the Dockerfiles and Makefiles:

git clone https://github.com/the-benchmarker/web-frameworks.git
cd web-frameworks
bundle install
bundle exec rake config

Then you drive one implementation through its generated Makefile, here javascript/fastify, with the results directory for concurrency level 10 created by hand:

make -f javascript/fastify/.Makefile build
make -f javascript/fastify/.Makefile test
make -f javascript/fastify/.Makefile warmup
mkdir -p javascript/fastify/.results/10
make -f javascript/fastify/.Makefile collect
make -f javascript/fastify/.Makefile unbuild

Concurrency levels and routes are handed to the same generator as comma-separated values, in METHOD:/path form:

CONCURRENCIES=64,256,512 \
ROUTES='GET:/,GET:/user/42,POST:/user' \
bundle exec rake config

After that you run the same collect target. Results are written beneath <language>/<framework>/.results/, and the generated targets also include clean for removing the framework container.

Release tags stopped in 2018 while develop kept moving

The versioning here is two stories. The last three GitHub releases are dated tags: 2018.9.26, 2018.7.30 and 2018.6.12, all from 2018. The default branch is develop, and its last push is dated 2026-11-02, so the repository has moved a long way past anything a release tag marks. The practical consequence: there is no release to pin when you want a stable harness, so you pin a commit on develop and you quote the benchmark revision alongside the result. The repository is MIT licensed and declares PHP as its primary language while carrying 32 language directories, from c and cpp through go, java, javascript, php, python, ruby, rust, swift and zig, which is a fair hint about how much of the code is generated configuration.

The roadmap admits the load generator is the weak point

The published roadmap is a list of things the project knows it has not done yet: switch to a modern load generator, tracked in discussion 8088, optimize the implementations, tracked in discussion 8093, build real infrastructure on a public cloud, check HTTP compliance for each framework, build a framework decision model, and publish a practical business guide. The same section says the project is independent and is not affiliated with or endorsed by Gartner, and the introduction says the longer-term goal is an evidence-based guide rather than a leaderboard declaring one universal winner. Read that as a status report on what you get today: a careful throughput table for a minimal API, and a scoring model that does not exist yet. If your decision depends on more than requests per second, this is where the project tells you it cannot help yet.

Editorial conclusion

Treat this repository as a measuring instrument rather than a verdict. It earns its place when you want to compare HTTP throughput of candidate backends on hardware you control, with the framework version, runtime variant, concurrency and benchmark revision all in view, and when you are willing to read saturation.json before you read a rate. It is the wrong tool for picking a framework for a team: maintainability, ecosystem, security, database access and your own workload decide that, and the project says so. Before you quote a figure, check the benchmark revision and machine, confirm the server actually used its allotted cores, and remember the GitHub releases stopped in 2018 while the develop branch was still moving on 2026-11-02.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. the-benchmarker/web-frameworks on GitHub