Library Checker problems, a competitive programming judge built on testlib

A problem set built to break implementations

Library Checker is an online judge with a specific bias. Instead of asking contestants to solve a puzzle, it asks them to implement a well-known library: a data structure, an algorithm, a numerical routine. The problems are named after the thing they test, and the test cases are the hard part.

This repository holds the problem data, described as the test case generator, the judge's solution, the task and so on, and it is the source code behind the judge site. The README makes the intent explicit: you can get test cases of problems from this repository.

That is the thing to understand about the project. It is not a problem archive to browse for practice problems, and it is not a tutorial set. It is a corpus of adversarial input data, which is why the generators are in the repository rather than just the test files. A generator is a program that can produce a new case given a seed, and a library author who runs those generators can produce far more input than the checked-in cases ever cover.

The categories in the top-level tree make the scope obvious. There are `big_integer/`, `common/`, `convolution/`, `data_structure/`, `enumerative_combinatorics/`, `geo/`, `graph/`, `linear_algebra/`, `number_theory/`, `other/`, `polynomial/`, `sample/`, `set_power_series/`, `string/` and `tree/`, plus a `test/` directory, a `categories.toml`, a `generate.py` and a `toml.py`.

The licence is Apache-2.0, and the project is not archived. The last push was on 2026-09-16, and there are no tagged releases, so the repository is consumed as a clone rather than as a versioned artifact.

Generating one problem's test cases

The usage section is a list of shell commands, and it is the clearest description of the layout.

cd /path/to/library-checker-problems

pip3 install -r requirements.txt
# or pip3 install colorlog

ulimit -s unlimited # for linux (doesn't need for OS X and WSL)

./generate.py -p unionfind # generate testcases of unionfind
# or ./generate.py datastructure/unionfind/info.toml

ls datastructure/unionfind/in/ # testcases of unionfind
ls datastructure/unionfind/out/ # solutions of unionfind
file datastructure/unionfind/checker # binary of output checker

Three directories per problem, and the names explain the whole model. `in/` holds the input files that are fed to a submission. `out/` holds the expected outputs, which are produced by running the judge's own solution. `checker` is the compiled output checker, a separate binary that decides whether a submission's output matches the expected output.

That separation is the point. A solution that computes the right answer in the wrong order, or prints the right numbers in the wrong format, is wrong, and only a checker that knows the problem's output convention can catch it. For a problem like sorting, where many orderings are acceptable, the checker has to be far cleverer than a diff.

The `ulimit -s unlimited` line is there because the generators use deep recursion, and the default Linux stack limit is small enough to overflow it. The comment notes that it is not needed on macOS or under WSL, which is the kind of platform detail that only appears once something has crashed.

The problem is selected either by a short name with `-p` or by the path to its `info.toml` file, and the `info.toml` is the per-problem configuration: the title, the time and memory limits, the checker, the generator settings. The `toml.py` at the top level and the `toml` parser in the thanks section are the machinery for reading those files.

Developer mode, test mode and the pull request gate

The developer section adds two flags that are the difference between a script you run and a project that maintains a corpus.

./generate.py -p unionfind --dev # developer mode
./generate.py -p unionfind --test # test mode (for developer, exec this command before PR)

./generate.py $(find . -name "info.toml" -not -path "./test/*") # generate all testcases

The `--test` mode is described as the command a developer should execute before a pull request, which tells you what the maintainers expect of a contribution: the generator must be reproducible and the generated cases must pass the checkers, not just compile.

The last line is the one to be careful with. The find expression walks the whole tree for `info.toml` files, excluding anything under `./test/`, and passes them all to the generator. That is a full regeneration of every problem's test cases, which on a repository this size is a long run and a large amount of disk. It is a maintainer's command, not a first-day command.

The two build requirements are worth stating plainly, because they are the reason this does not run everywhere. The requirements are Linux, macOS or Windows with MinGW-w64, Python 3.8 or newer, and g++ or clang++ needing `--std=c++17` and `__int128_t`. The 128-bit integer type is the giveaway: several of the problems in categories like `big_integer/` and `convolution/` exist precisely because their answers overflow 64 bits, and the generators use the same type to compute the expected values. A compiler without `__int128_t` is a problem, not an inconvenience.

The Python side has a single dependency. The `requirements.txt` file contains one line, `colorlog >= 4.1.0`, and the README even offers the shortcut of installing colorlog directly, since that is all the script needs.

Testlib does the verifying, and the checkers are per problem

The thanks section names the two external projects the whole system leans on. The project uses testlib for the verifier, meaning the input-checker, and for the output-checker, and it uses the toml parser as the TOML parser.

testlib is the checker framework from the competitive programming world, the one that judges like Codeforces and AtCoder use, and it is the reason the output checkers in this repository can be problem-specific. In testlib the convention is that a problem ships a checker program that reads the input, the participant's output and the expected output, and decides on a verdict, with the checker itself being a small C++ program compiled per problem. That is what the `checker` binary in each problem directory is, and the `file` command in the usage example is there to confirm it was built.

The alternative approach would be to check outputs by comparing them, but that only works when the answer is unique, and in a data structure problem it usually is not. A union-find problem can accept any spanning forest or any order of independent operations. A graph traversal problem can accept any valid ordering of the output. Writing a checker per problem is the cost of testing implementations rather than answers, and the repository already contains one per problem.

The judge itself is a separate project. The README lists the three repositories that make up Library Checker: this one for problems, library-checker-judge for the judge, and library-checker-frontend for the frontend. If you want to run a judge locally, you need the other two as well.

For testing your own library against these cases, the README points to two external tools. Online Judge Tools is described as useful for testing your solutions on your computer, and Online Judge Verify Helper as useful if you want to use GitHub Actions for testing your library against the problems. The second one is the more interesting for a library author, because it turns the corpus into a CI check on every commit.

Creating a problem follows a written guideline

The repository is also the place where new problems are designed, and the contribution rules are specific about how.

The README points at two guideline documents under `docs/`, one in English and one in Japanese, and describes them as the file structure and how to create the problem. The bilingual pairing is a deliberate choice, and it extends to the rest of the documentation: the troubleshooting section has an English version and a Japanese edition, and the contributing section has the same split.

The Japanese contributing text contains the most useful statement of the project's attitude to newcomers. It says the maintainers understand that competitive programmers are not necessarily used to GitHub, that many have never sent an issue or a pull request, and that however chaotic a contribution may be, it will not make them angry and will not break the repository, and that trying is a good way to get familiar with GitHub. That is an unusual thing for a project to write down about itself, and it tells you something about the scale of incoming pull requests on a repository with a public issue tracker and a Discord.

The contribution rules themselves are strict in one respect. Issues and pull requests are welcome, but except for minor changes, such as adding test cases or fixing statements, you should create an issue first. The reason is visible in the layout: adding a test case changes the corpus that every other user's generator run must reproduce, and fixing a statement changes what the judge presents. Both need a decision rather than a patch.

The top-level tree also shows an `AGENTS.md` file and a `.devcontainer/` directory, so the repository ships a container definition and agent instructions alongside the guidelines.

Where the boundaries of this project sit

It is worth being clear about what this repository is not, because the name invites a wrong assumption.

It is not a judging system. The judge lives in library-checker-judge, the frontend in library-checker-frontend, and this repository is the data both consume. You can clone this and you get generators, expected outputs and checkers, and you need a separate online judge or a local harness to actually run submissions against it.

It is also not a competitive programming problem set in the usual sense. There is no difficulty ladder and no narrative framing; the problems are named after the thing they test, and a solver who knows the topic can probably guess the intended approach. What the corpus optimises for is coverage and adversarial cases, not for teaching.

The practical limitation that bites hardest is the environment. Generating cases needs a C++17 compiler with `__int128_t`, Python 3.8 or newer, and on Linux a raised stack limit, and the toolchain is pinned to g++ or clang++ rather than being compiler-agnostic. The MinGW-w64 note for Windows means the supported Windows path is a specific cross-compilation environment rather than native Windows tooling. Anyone planning to generate cases in CI has to build that environment first, which the `.devcontainer/` directory presumably addresses but which the README does not walk through.

There is also the plain fact of scale. A full regeneration across every problem in the tree is a heavy operation, and the checked-in expected outputs and checkers are what most people will actually use. The generators are for the people who need more or stranger cases than the ones in the repository, and the `test/` directory at the top level suggests the project has its own test harness for the problem definitions themselves, separate from the generated data.

Editorial conclusion

library-checker-problems suits a competitive programmer or library author who wants to stress a data structure implementation against cases designed to break it, and it suits a judge designer who wants to see how a well-run problem set is structured, with per-problem info.toml files and a generator script. It does not suit someone who wants a problem archive to learn from, since the emphasis is on verification rather than on tutorials, and it does not suit a team that needs the whole judge to run, because the judge itself lives in a separate repository. Verify first that the problem you want has a generator that runs on your platform, since the requirements include a C++17 compiler with __int128_t and a raised stack limit on Linux, and check the .gitmodules setup before you expect a clean clone to contain the submodules the generators rely on.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. yosupo06/library-checker-problems on GitHub