Completely Unscientific Benchmarks compares one treap across twenty languages

What the project actually measures

For this benchmark the author implemented Treap in a few classic and a few hyped programming languages and tested their performance on Linux, Mac OS, and Windows, with the parenthetical that all of them ran on different hardware, so the results should not be compared between platforms.

That caveat is the most important line in the README, and it is stated before any table. The three operating systems each ran on their own machine, which means a row in the Linux table and a row in the Windows table are not measuring the same thing twice.

The README also opens with a quote, There are three kinds of lies: lies, damned lies, and statistics, which sets the tone. The project is deliberately not claiming to be a serious benchmark, and the name says so.

The choice of treap is deliberate too. The benchmark turned out to be a good benchmark of memory-intensive operations, which should have pushed memory management implementations to their edge. A treap allocates a node per element and keeps the tree balanced, so the workload is dominated by allocation, pointer chasing and garbage collection rather than by arithmetic. That makes it a reasonable proxy for the class of work that separates a managed runtime from a manual one, and a poor proxy for number crunching.

The languages covered are listed in the repository description, and the directories at the root match them: ada, c#, c++, clojure, crystal, d, fsharp, go, haskell, java, javascript, kotlin, lua, modula2, modula3, nim, oberon07, ocaml, pascal, php, python, ruby, rust and swift. The README describes them as a few classic and a few hyped, and the list includes several languages that are not in the description, which shows the project kept absorbing new entries.

The primary language of the repository is C++, the licence is dual with LICENSE-APACHE and LICENSE-MIT at the root, and the last push was on 2026-07-02 with no GitHub releases.

The metrics, and which ones are opinion

The metrics list is worth reading closely, because the project mixes measurements with judgements and says so.

There are four measured metrics and two scored ones. Real Time in seconds, smaller is better. Slowdown Time, the relative speed compared to the best tuned solution, smaller is better. Memory in megabytes, smaller is better. And Binary Size in megabytes, smaller is better.

Then there are the subjective ones. Expressiveness, scored e12s, from 1 to 10, where higher is better and the README notes it is a subjective metric based on the author's experience. And Maintenance Complexity, scored M.C., from 1 to 10, where smaller is better, also explicitly subjective.

The rationale for adding those two is explained at length. Experienced developers in system programming languages tend to work comfortably with raw pointers, and that makes the comparison of the solutions only by speed and memory consumption unfair. High-level abstractions tend to introduce some performance hit in exchange for safety and expressiveness.

So the two scores exist to record what a purely numeric comparison leaves out. The scale for expressiveness runs from 1, pure magic, to 10, easy to get started and express your intent. The scale for maintenance complexity runs from 1, easy to maintain, through 5, ugly yet safe, to 6 to 10, hard to keep it right, that is, risky. The ease of maintenance is estimated for a big project using the given language and the given approach.

That last qualifier is important for reading the numbers. A maintenance score is not about whether you can write the treap, it is about whether a large codebase in that style stays correct over years.

The author repeats the subjectivity warning next to both scores, twice in the metrics list itself, so a reader skimming the tables cannot miss it.

Naive versus tuned, and why there are two scoreboards

The results section explains a decision that changed the project over time.

Originally, this benchmark had a goal to implement the same natural and naive API in all the languages with the exception of C++, which would represent a bare metal performance. Over time, the author received optimized solutions in other languages, but it doesn't seem fair to put them on the same scoreboard.

So there are two scoreboards. The naive implementations are the ones a developer with enough experience in a given language would implement as a baseline good enough solution where correctness is more important than performance.

Within the naive set there is a further split for the two languages where it matters. First, the author tried to play by the rules of the garbage-collected languages, so there are ref-counted versions of implementations for C++ and Rust, but then the author still wanted to compare the results with idiomatic implementations for C++, called raw-pointers, and for Rust, called idiomatic.

That produces the rows you see in the Linux table, where C++ appears four times, once as java-like with clang, once as java-like with gcc, once as naive unsafe raw pointers with clang, once as naive unsafe raw pointers with gcc, plus a shared_ptr variant.

The tuned solutions are in a separate scoreboard, and the reason given is that all the solutions implement the same algorithm but were created with performance in mind and received quite an intensive profiling and tuning.

The fairness caveat is stated too. Most of the implementations, except for C++, Haskell and OCaml, were implemented by mostly adapting the syntax from the very first implementation of the algorithm in Kotlin. The author says that even Rust, considered to have the steepest learning curve among the tested languages, did not require any black magic, and that the solution does not require either unsafe code or lifetime annotations. C++ was implemented separately, so it has a few shortcuts, and thus it might not be a completely fair comparison, with a promise to implement a fair C++ solution and a C++-like Rust solution to see if the performance can be on par.

How the numbers were measured

The measurement section is short and has some real methodological caveats that any reader of the tables should hold on to.

To measure time the author used the time utility on Mac OS and Windows, in an msys2 environment, and cgmemtime on Linux.

Memory measurement was only available on Linux with the cgmemtime utility, which leverages CGroup capabilities to capture the high-water RSS plus CACHE memory usage. And given the limitations of the cgroup subsystem, which counts caches and loaded shared objects unless they are already cached or loaded by other processes, the author takes the lowest memory footprint among all the executions.

That last sentence is a significant methodological choice, and it cuts both ways. Taking the lowest run reduces the noise from cache and shared object effects, but it also means the reported memory is the best case rather than the typical case, and a reader comparing rows should know which number they are looking at.

The Makefile shows how the runs are driven. The LANGS list names every implementation directory, and there are two aggregate targets. The compile target depends on all of them and each one is built by recursing into its own directory with $(MAKE) --directory./$@. The run targets are generated from the same list by prefixing, and each run target depends on its language target, so make run builds everything and then runs everything.

The run recipe is where the measurement happens. For each implementation it asks the language's own Makefile for its run command with a dry run, then executes that command twenty times, each time through../cgmemtime with the -t flag, and cuts the output on semicolons to pull out the time and memory fields:

compile: $(LANGS)
run: $(RUNS)

The twenty repetitions are why taking the lowest memory figure is a deliberate choice rather than an accident, since it needs multiple samples to be meaningful at all.

Reading the tables without being misled

The Linux table shows the shape of the results and also the shape of the caveats.

The header gives the platform as Arch Linux, x64, with an Intel Core i7-4710HQ CPU, and the columns are language, expressiveness, maintenance complexity, real time in seconds, slowdown, memory in MB, binary size in MB and compiler version.

The best tuned solution is the reference point, at 0.167 seconds and 0.25 MB of memory, and everything else is expressed as a multiple of it. The Ada implementation with naive unsafe raw pointers runs in 0.24 seconds at x1.44 with 0.4 MB of memory and a binary size of 0.292 with GCC Ada 9.3.0. The C++ variants land between 0.19 and 0.36 seconds, with the gcc raw-pointer build at 0.19 and the shared_ptr build at 0.36 with clang and 0.35 with gcc.

Two things in that table are worth noticing. First, the compiler version column matters, because the same C++ implementation is listed twice with different compilers and slightly different times and binary sizes, which is a reminder that the language is not the only variable. Second, the C# row shows a real time of 0.73 with an asterisk next to it, and the memory column as 10 with a binary size of N/A, which is what a managed runtime on.NET Core 3.1 looks like when the process footprint is measured rather than the allocation.

The expressiveness and maintenance scores move independently of the speed. C# and Crystal and the D garbage collected implementation all score 9 or 10 on expressiveness and 1 on maintenance complexity, which is the author's judgement that these are pleasant languages whose baseline implementations are easy to keep right, while the raw-pointer variants score 6 for expressiveness and 8 for maintenance complexity.

None of that is a measurement, and the README says so in the metrics list and in the results narrative. The numbers tell you what the code did; the scores tell you what the author thinks of working that way.

Where this benchmark fits, and where it stops

The comparison worth making is with a serious benchmark suite and with picking a language from general reputation.

Against a serious benchmark, the differences are structural. A project like this has no standardised workload specification, no fixed set of hardware, no repeated runs with statistical treatment across many machines, and its own author says the implementations are not equivalent. What it has instead is the same algorithm written the way each community would naturally write it, which is a different and arguably more useful thing for a developer deciding between languages.

Against picking a language from reputation, the value is that the claim becomes concrete. The memory numbers make the cost of a garbage collector visible, the binary size column shows what a runtime costs you on disk, and the two subjective scores add the dimensions that a speed table cannot.

The limits are substantial and the README is upfront about most of them. The platforms ran on different hardware, so cross-platform comparison is out. The implementations were largely adapted from a first Kotlin version, so the idiomatic-ness of each is limited by that origin, and C++ was written separately and may not be a fair comparison. The expressiveness and maintenance scores are one author's opinion, from one author's experience. Memory was measured only on Linux. And the results are from a period when several of these languages have had new major versions.

What it is then is a reference and a conversation starter, not a decision procedure. The author is candid that the C++ implementation has shortcuts and promises a fairer C++ version and a C++-like Rust version, which tells you the numbers were treated as provisional even at the time.

The repository is also cheap to run yourself. Every implementation has its own directory with its own Makefile, the aggregate targets are two lines in the root Makefile, and the language directory list includes the more unusual choices, Modula 2, Modula 3, Oberon-07 and OCaml, which no commercial benchmark would bother with. The value of the project is as much in the implementations you can read as in the tables.

On maintenance, the last push was on 2026-07-02, the repository is not archived, and there are no GitHub releases, so the scoreboards are updated in place rather than tagged by version.

Editorial conclusion

Completely Unscientific Benchmarks suits a developer who wants a concrete sense of what a data structure costs in a language they are considering, since the same treap is implemented in each and the memory and binary size columns make the allocation strategy visible rather than theoretical. It is the wrong tool if you need a benchmark to decide between languages, because the author states the implementations are not comparable, the two headline scores are explicitly subjective, and the platforms ran on different hardware. Before quoting any number from it, read the measurement section, because memory was only measured on Linux with cgmemtime and the lowest run of each set was taken.

Frequently asked questions

Official sources

1. frol/completely-unscientific-benchmarks on GitHub
2. Issues
3. License: Apache-2.0
4. README