speed-comparison and the case for one formula across seventy-five implementations

One formula, deliberately small, so the workload can be compared

Every implementation in this project evaluates terms of the same alternating series:

π ≈ 4 × (1 − 1/3 + 1/5 − 1/7 + …)

The README is explicit about why the formula was chosen this way. It is small enough to inspect across languages, it converges slowly, and it is useful as a repeatable arithmetic workload rather than an efficient way to calculate pi. Correctness is still enforced: outputs must be finite and pass a workload-dependent convergence check, so a fast implementation that returns nonsense is rejected rather than ranked.

That choice shapes everything downstream. A workload this small can be ported to a language in an afternoon, and since every port does identical arithmetic, the remaining differences come from the toolchain rather than from the problem. The series also gives the compiler something to chew on, since the terms alternate in sign and the loop body is a float operation that a vectoriser can attack.

What it does not give you is a general answer. The project states plainly that it is a single-threaded numerical microbenchmark and that it does not establish web-server, database, I/O or whole-application performance.

Ranked runs stay single-threaded, and SIMD gets its own target

The project draws a hard line between what is ranked and what is not. Ranked implementations stay single-threaded and use the Leibniz series, and explicit SIMD variants, which process several values per instruction, get separate targets. Compiler flags, relaxed floating-point math and paired-term transformations must remain visible rather than being folded into a single language name. A standalone Fortran OpenMP example is preserved outside the ranked comparison entirely, and the repository still carries that file under src/alt.

This is a methodological choice with real consequences for a reader. A target can be a language, a compiler choice or an optimized variant, so a result labeled with an optimization setting is not directly comparable to a plain one. Reading differences within a compatible hardware, workload and protocol group is the instruction the README gives, and it also tells the reader to check source and build settings before attributing a gap to the language.

The limits of the sample are stated too. Three samples and their observed range cannot establish a statistically reliable winner for every small difference, so a narrow margin in the chart is not a result. A SIMD target and its scalar counterpart sitting next to each other in the chart is a deliberate reminder that the ranking is not one dimension wide.

Checking the harness without running a single benchmark

The cheapest way to get a feel for the project is to run its test suite, which exercises configuration, selection, measurement and metadata regression tests without needing Docker or any language toolchains. The requirements are uv and Python 3.11 or newer, with CI on 3.12.

git clone https://github.com/niklas-heer/speed-comparison.git
cd speed-comparison
uv run --locked --project pipeline --extra dev pytest pipeline -q

A reader who runs this should see a pytest run against the pipeline directory, and nothing else. No benchmark executes, no container is pulled, and the point of the step is to confirm the harness itself is intact before trusting anything it produces.

The dependency set in pyproject.toml explains the shape of the project. The runtime dependencies are pandas, matplotlib, seaborn, numpy, pillow and cairosvg, which is a plotting and reporting stack rather than a benchmarking one, and the package also registers a console script named analyze pointing at analyze:main. The dev group adds pytest and ruff. Since the environment is described as locked through uv.lock, running with --locked means the toolchain resolution is pinned to what the repository recorded.

Running one implementation goes through Dagger, Nix and Devbox

The actual measurement path is the interesting part of the design. The Dagger Python SDK provisions its own CLI and engine, and Nix and Devbox run inside the container, so the host does not need a compiler for every target language. The README also warns that the first run needs network access and can spend much longer downloading toolchains than the calculation itself takes.

QUICK_TEST_ROUNDS=10000 USE_LOCAL_IMAGES=1 \
  uv run --locked --project pipeline python pipeline/benchmark.py \
  --output ./results/quick-go go

Three details matter for a first attempt. QUICK_TEST_ROUNDS shortens the workload, and the README says to keep it short while developing because it checks functionality, not performance. USE_LOCAL_IMAGES=1 builds the environment locally instead of pulling a published registry image. The output directory is explicit here, though omitting --output creates results/<run-id>/ automatically. Each attempt should use a new output directory.

Target IDs are discoverable rather than guessed:

uv run --locked --directory pipeline python -c \
  'from languages import LANGUAGES; print("\n".join(LANGUAGES))'

Omitting QUICK_TEST_ROUNDS falls back to src/rounds.txt, currently one billion terms. Some targets require x86-64 and AVX2 or AVX-512 instructions, and the README is blunt that emulation or a busy development machine are unsuitable for publishing comparable timings.

Every run leaves a bundle that can be opened later

The output layout is designed so a result outlives the machine that produced it. A quick check writes something like this:

results/quick-go/
├── run.json          # Status, protocol, tooling and whole-invocation clock
└── targets/
    ├── go.json       # Samples, output, source, commands and environment
    └── rounds.txt    # The workload actually used

Each target file carries the samples, the output, the source, the commands and the environment. Successful targets are saved immediately, including when a later target fails, so a run that breaks in the middle still yields the earlier measurements. That is the difference between a benchmark suite you can rerun and one you can only trust.

The analyzer is a separate step and expects the targets directory together with the matching workload file:

uv run --locked analyze.py \
  --folder ./results/quick-go/targets \
  --rounds ./results/quick-go/targets/rounds.txt \
  --out ./results/quick-go/report

Passing the rounds file alongside the samples is what lets the report state which workload produced the numbers, rather than assuming a default. A report generated this way is local, and the published site holds the same data rendered for a wider audience.

The published baseline is dated, and the project says so

The latest full run recorded in the README took 5h 18m 46s across 75 implementations with one billion terms per execution. That clock is labelled a historical Argo clock and it includes checkout, setup, measurement, analysis and a failed publication attempt, while excluding a later publication-only retry.

More important is the protocol note. That run used a six-execution protocol, which the README says is distinct from the new one-warmup, three-measurement protocol. The published September 5, 2026 baseline came from native Devbox jobs in Argo on a shared x86-64 VM and predates the current runner protocol.

The consequence is spelled out: new package pins and successful smoke checks do not replace that published dataset. Historical runs keep their original samples and missing metadata, and later recovery evidence is kept separately. The chart shows median times with observed sample ranges, and SIMD and relaxed math variants are labeled so the reader can tell which group a bar belongs to.

The repository also has no GitHub releases at all, so the published chart on the site and the current state of master are two different things, and the README is careful to keep them apart.

Where this benchmark is the wrong tool

The project is explicit about its own limits, and they are serious. It does not establish a universal ranking of programming languages, it does not measure web-server, database, I/O or whole-application performance, and it does not guarantee identical hardware conditions or reproducibility on every machine. It also does not show that different math modes or algorithms are interchangeable.

So the first wrong-tool case is the obvious one: a team that wants to know which language to write a service in. This benchmark cannot answer that, and its own documentation says so. The second case is concurrency, since ranked implementations stay single-threaded and the OpenMP example sits outside the comparison. The third is a machine without the right instructions, where AVX2 or AVX-512 targets cannot run natively.

The alternative approach is a general-purpose benchmark suite such as a language-specific community benchmark or a vendor suite, which covers I/O, database and application shapes that this project deliberately excludes. The difference in approach is the trade this project makes on purpose: one small, identical, inspectable arithmetic workload with full provenance for every sample, in exchange for any claim about real applications.

Licence, layout and the cost of picking this up

The project is MIT licensed, and pyproject.toml records it as text = "MIT" with the author as Niklas Heer. The repository tree explains how the work is split: a pipeline/ directory holds the measurement harness, src/ holds the implementations, analyze.py and publish.py sit at the top level, report_metadata.py assembles the report metadata, site/ holds the published site, docs/ and scripts/ the supporting material, and results/ the run output. The root also carries CLAUDE.md, a .coderabbit.yaml and a .vercelignore, which says something about how the project is worked on and deployed to Vercel.

The cost of adoption is mostly the harness rather than the code. Setting up means uv, Python 3.11 or newer, Docker for the real measurement path, and a Dagger engine that provisions its own CLI. The first run downloads toolchains, so an air-gapped or bandwidth-constrained environment is a real obstacle, and the README says the download can take far longer than the calculation.

The last push to the repository was on 2026-10-20, so the work is current, but there are no GitHub releases to pin to and the published dataset is a dated run rather than a live result.

Editorial conclusion

speed-comparison suits an engineer who needs to argue about a language or compiler choice with evidence they can open, and it suits a team that wants a reproducible harness rather than a leaderboard. It does not suit someone looking for an answer to which language is fastest, since the project itself states that it does not establish a universal ranking. Before adopting it, check whether your target language has a target ID in the LANGUAGES list, confirm you can run x86-64 with AVX2 or AVX-512 for the SIMD targets, and read the protocol note in the README, because the published September 5, 2026 baseline predates the current runner protocol and new pins do not replace that dataset.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. niklas-heer/speed-comparison on GitHub
4. Project website
5. README