asatarin/testing-distributed-systems: A Curated Reading List, Not a Test Harness

What the list is for, and who should open it

This repository is a bibliography. The README describes it as a list of resources on testing distributed systems curated by Andrey Satarin, and the topics attached to it (distributed-systems, fault-injection, fuzzing, jepsen, jepsen-tests, testing) describe the subject matter rather than the contents. Nothing in the repository layout suggests runnable test code: the top-level entries are .github/, .gitignore, .lycheeignore, 404.md, LICENSE.txt, README.md, _config.yml, _includes/, _layouts/ and assets/. That is the shape of a Jekyll site, not a library. The _config.yml plus _includes/ and _layouts/ directories exist so the Markdown list can be rendered as a browsable page at the project's GitHub Pages homepage, and .lycheeignore exists so a link checker can skip known-bad URLs.

The audience follows from that. If you are the person who has to decide how a storage or coordination service will be tested before a release, the list gives you the papers that the tooling in this space is built on. If you are looking for a library to add to a build file, this is the wrong repository and you will know within one screen.

How the material is organised: papers first, tools second

The README opens with a section titled Overview of Testing Approaches, and within it the first grouping is Research Papers, split into subheadings for Bugs, Testing and Fault Tolerance. Each entry is a link plus a sentence or two of annotation. The Bugs subsection cites work such as What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems, which the annotation says studies Hadoop MapReduce, HDFS, HBase, Cassandra, ZooKeeper and Flume, and TaxDC: A Taxonomy of Non-Deterministic Concurrency Bugs in Datacenter Distributed Systems, described as a taxonomy covering Cassandra, Hadoop MapReduce, HBase and ZooKeeper.

The Testing subsection is where the list earns its place. It includes Simple Testing Can Prevent Most Critical Failures, described as an overview of how simple testing helps if you focus it correctly, and Why Is Random Testing Effective for Partition Tolerance Bugs?, which the annotation says introduces notions of test coverage relating to network partition and explains why Jepsen-style random testing finds real defects. The Fault Tolerance subsection covers redundancy versus actual fault tolerance, limping hardware, partial network partitioning and partial failures, again with named systems in each annotation.

This ordering is a deliberate editorial choice: the list argues that you should understand the bug taxonomy before you choose a fault-injection tool. Whether you agree, the structure means you can read it top to bottom as a curriculum rather than as a link dump.

Reading the list, and checking its links, without installing anything

There is no install step in the README, and no package, binary or service to set up. The repository is published as a GitHub Pages site, so the first real use is simply opening the rendered page at the project's homepage and reading it there. The README carries a Jekyll table of contents marker, which the site build turns into a navigable page; the site settings live in _config.yml and the templates in _includes/ and _layouts/.

If you want the list in your own editor instead, the README is plain Markdown and can be read as-is. The one piece of tooling the repository configures is a link checker: .lycheeignore is present at the top level, which means the project expects a checker to run against the README and to skip the URLs listed in that file. If a URL in the list is permanently dead and you want to keep your own check clean, .lycheeignore is the file to edit. The README does not document a command for running the checker, so the exact invocation is up to whatever tool you point at the file.

The practical consequence is that the cost of adopting this list is reading time and a periodic manual link check, not a dependency update. Nothing here needs a version bump.

The archive.org dependency is the list's weakest seam

Several entries point at blog.acolyer.org through web.archive.org snapshots rather than at the original posts. The README links the review of An empirical study on the correctness of formally verified distributed systems, the piece on what bugs cause cloud production incidents, the early-detection-of-configuration-errors article and the Morning Paper review of the partition-tolerance random testing paper all through web.archive.org URLs. That is a sensible preservation move for a list whose value is longevity, but it means the reading experience depends on a third party staying up, and it means you are reading a snapshot rather than a maintained page.

A second limitation is that this is a single curator's selection. The README names Andrey Satarin as the curator and points to Twitter, Bluesky and a follow page for feedback, with no described review process and no contributor guidelines in the visible layout. That is normal for curated lists and it is also the reason to treat the list as a starting point: coverage of a given system depends on whether the curator found the paper interesting. If your system is not one of the ones named in the annotations (Hadoop, HDFS, HBase, Cassandra, ZooKeeper, Kafka, Mesos, YARN, MongoDB, Redis, RethinkDB, RabbitMQ, Elasticsearch, Spark, Kudu, Ethereum Blockchain, Raft LogCabin), the list will not tell you what has already been published about it.

Where a real test framework fits instead

The list repeatedly points at Jepsen as the reference implementation of random partition testing, and that is the honest alternative for anyone who reads this repository and concludes they need to run something. The difference in approach is fundamental. A framework such as Jepsen gives you a client library, a generator for operations against your system and a checker that looks for consistency violations across a recorded history; you write a test, run it against a cluster, and get a pass or a counterexample. This repository gives you the papers that explain why that style of testing finds bugs, plus pointers to other work, and stops there.

There is a middle path the list also implies. Papers such as Understanding and Detecting Software Upgrade Failures in Distributed Systems, which the annotation says proposes two new tools targeting upgrade failures and applies them to several systems, describe tooling that is not packaged for you. The list is most useful precisely at that boundary: it tells you which failure class has published detection techniques before you commit engineering time to writing your own harness. If you need a repeatable regression suite next sprint, pick the framework. If you need to know which failure classes your suite is missing, read the list first.

Licence, maintenance and the cost of keeping a list current

The repository is licensed CC-BY-4.0. For a bibliography that is a permissive choice: you can reuse and adapt the list, including commercially, provided you give attribution. It also means the annotations are the curator's copyrighted text, so copying a paragraph into internal documentation without credit is outside the licence even though the underlying papers are separately copyrighted by their publishers. The repository does not offer legal advice and neither does this article; read LICENSE.txt for the exact terms.

The last push to the repository was on 2026-07-28. There are no releases, which is expected for a list with no build artifacts. The upgrade cost is therefore not a version bump but link rot: papers move, publisher URLs change, and the .lycheeignore file exists because some links already need excluding. Budget a periodic link check rather than a dependency update, and expect that the archive.org substitutions will accumulate over time. The list's value decays slowly and continuously rather than breaking at a version boundary, which is a better failure mode than most software dependencies but still requires someone to run the checker.

Editorial conclusion

Adopt this list if you are designing a test strategy for a distributed system and need primary sources on fault injection, partition tolerance and upgrade failures before you pick tooling. Skip it if you want a framework to install today: there is nothing to run, and the repository is a Jekyll site whose only build artifact is the rendered page. Verify first that the linked papers still resolve, since the list leans on acolyer.org posts that are already routed through web.archive.org, and check the LICENSE.txt terms before republishing entries in your own internal documentation.

Frequently asked questions

Official sources

1. asatarin/testing-distributed-systems on GitHub
2. Issues
3. License: CC-BY-4.0
4. Project website
5. README