sbc-reviews: the raw benchmark data behind single-board computer reviews

One issue per board, and the data committed in real time

The sbc-reviews repository is the review and benchmarking data repo behind Jeff Geerling's single-board computer reviews, and its stated purpose is to be a public point of reference rather than a set of finished articles.

The motivation section explains the change of approach. The author tests a lot of boards from Raspberry Pi, Radxa, Orange Pi, Pine64, ODROID, ASUS and others. Previously the data was compiled in a single project folder and a blog post, with a few specific benchmarks featured in a video. The decision was to have a public repository with all the test data and review notes for the boards tested, compiled in real time.

The organisational unit is an issue, not a directory. For every board tested or reviewed, an issue is opened with the name of the SBC, and the data is compiled in that issue. That choice is unusual and it has a practical consequence: the board's data lives in a conversation rather than in a versioned file, so a reader looking for a specific measurement has to search issues rather than browse a tree.

The stated purpose of putting it in public, unless under NDA, is as a reference for the author and for anyone who runs into interesting results, with the example given being an RK3588 on the Rock 5 model B benchmarking slower than an RK3588S on the Orange Pi 5. That kind of specific, counterintuitive cross-board comparison is exactly the thing a manufacturer datasheet will never tell you, and it is the reason the data is worth publishing separately from the reviews.

The site at `sbc-reviews.jeffgeerling.com` lists all the reviews and all the test data.

The issue template is the test plan

There is a subtlety in the contribution rules that tells you how the project is curated.

If you would like to see a board added that is not already listed, the instruction is to start a new Discussion rather than an Issue, and explicitly not to open an Issue until the author has confirmed they are going to acquire and test that board. In other words, the issue list is a record of boards that were actually tested, and requests are handled separately. That keeps the repository from filling up with wishlist entries and preserves the property that every issue corresponds to a real measurement.

The list of all the tests attempted on a board is contained in the `sbc-board.md` issue template, under the `.github/ISSUE_TEMPLATE` directory. That file is the closest thing the project has to a specification, and it is worth reading before you draw conclusions from a missing measurement. A test that is not in the template was not part of the standard run for that board.

The author also notes that he often tests board-specific features beyond the standard list, depending on how much time he is willing to devote to a board and whether he will deep-dive or just get basic numbers. So the template is the floor, not the ceiling, and the shape of the data varies from board to board depending on the effort that went into it.

For a reader comparing two boards, that variability is a caveat worth keeping in mind. A board with three numbers and a board with thirty are not equally characterised, and the issue text is where the difference shows.

pyinfra runs the benchmarks over SSH, task by task

The benchmark scripts are run using pyinfra, and the installation is a single pip command.

pip3 install pyinfra

The workflow inside the `benchmark` directory is three steps. You modify `inventory.py` to point at the system under test, you modify `group_data/all.py` with the variables appropriate for your system, and you run:

pyinfra inventory.py main.py -y

This assumes you have already configured an SSH connection to the system under test. The `-y` flag is the confirmation that lets pyinfra apply changes without prompting, which is what makes it usable for a benchmark run where the tasks install packages, copy files and execute measurements on a remote board.

Individual benchmarks can be run separately by calling the task files individually:

pyinfra inventory.py tasks/tinymembench.py -y

That is the important operational detail. The `tasks/` directory holds one file per benchmark, and the tinymembench example is one of those task files. If you want only one measurement on a board, you run one task rather than the whole suite, and the requirement is that you have run `tasks/setup.py` at least once beforehand, if you are not running `main.py`. The setup task is the one that prepares the board, so the other tasks assume it has happened.

The structure of `group_data/all.py` holding the variables for a system is worth noting too, because it is where the per-board parameters live. The same task code runs against different boards, and the variables file is what distinguishes them.

Ansible and pyinfra side by side, and the reason for it

There is a parallel set of benchmarking scripts in the `benchmark-ansible` folder, and the reason given is unusual enough to be worth quoting in substance.

The author is setting up both side by side to do essentially the same thing, and the reason is that he wanted to give pyinfra a try since he had not used it for any production scenario before. So the duplication is an experiment, not an oversight, and it is a reminder that the choice of tool in this repository is partly a personal one.

That has a consequence for anyone picking this up. The pyinfra set is the one the README documents in detail, with the inventory file, the group_data variables and the per-task files, and the Ansible folder is described only in the note. If you are re-running benchmarks, the pyinfra path is the documented one, and the Ansible scripts are there to compare against.

The repository also has a `.gitmodules` file, so parts of the setup are submodules rather than plain directories. That is the kind of detail that matters when you clone the repository and a benchmark script turns out to be an empty directory because you did not initialise the submodules.

The website is Hugo, built in GitHub Actions

The published site is generated with Hugo, and the local development loop is two commands.

You check out Hugo's installation guide, change into the `web` directory and run `hugo server` for a local development environment and a preview of the site. The `web/` directory is one of four top-level directories, alongside `benchmark/`, `benchmark-ansible/` and the `.github/` directory, so the site content is entirely separate from the benchmark tooling.

The site is built using GitHub Actions, with the configuration stored inside `.github/workflows/hugo.yml`. That means the published site is regenerated from the repository rather than edited by hand, so the relationship between the data and the site is a build step, and a change to the data eventually reaches the site through the workflow.

The overall repository structure, then, is three concerns. The issues hold the data and notes. The `benchmark/` and `benchmark-ansible/` directories hold the scripts that produce measurements. The `web/` directory holds the site that publishes them. And a `.github/ISSUE_TEMPLATE/sbc-board.md` file defines what gets measured in the first place.

One structural detail is worth flagging for anyone automating against it. The repository has no GitHub releases, and the content is organised as issues rather than tags, so there is no versioned artefact to pin. A board's data changes over time as more tests are added to its issue, which makes the site the better reference for a snapshot of the current state.

Where this repository is the wrong tool

The first case is benchmarking your own hardware. The scripts are written for a single-board computer reachable over SSH, with a `tasks/` directory per benchmark and a `group_data/` file of board parameters, and the task list in the issue template is the set of tests that make sense for an SBC. If you want a general system benchmark suite, this is a starting point to read rather than a tool to run, and the per-board variables are the part you would have to replace.

The second case is wanting a leaderboard. The data is raw per-board measurements published as issues and a site, with no cross-board aggregation in the repository. Comparing two boards means reading both sets of numbers and judging whether the tests were run under comparable conditions, which the repository does not normalise for you.

The third case is expecting reproducible results across runs. The scripts drive a board over SSH and install things onto it, so what the numbers mean depends heavily on the software state of the board when the task ran. The `tasks/setup.py` requirement is a partial guard against that, and the fact that setup runs first is the author's way of making the environment consistent. Whether a given board was set up identically to another is a question the issue text has to answer.

For a board vendor who wants their hardware benchmarked, the process is also not self-service. The instruction is explicit that you should not open an issue until the author has confirmed they are going to acquire and test the board, so the testing is done by the maintainer, not by the vendor.

Editorial conclusion

This repository fits someone comparing single-board computers who wants the underlying measurements rather than a conclusion, since the same scripts and the same task list are used across boards and the data is committed in public. It does not fit someone who needs a benchmark suite for their own hardware outside the single-board-computer context, because the scripts assume a board under test reachable over SSH. Before running anything, read the `sbc-board.md` issue template, which lists every test attempted, and run `tasks/setup.py` first, since the other tasks assume it has already run.

Frequently asked questions

Official sources

1. geerlingguy/sbc-reviews on GitHub
2. Issues
3. License: MIT
4. Project website
5. README