yarpgen writes random C that hashes itself, so two compilers must agree

A generator whose output is an oracle, not a crash

yarpgen is a random program generator which produces correct runnable C, C++ and DPC++ programs, with the DPC++ work described as being in an early stage. The generator is specifically designed to trigger compiler optimization bugs and is intended for compiler testing.

The correctness property is precise. A generated random program is guaranteed to be a statically and dynamically correct program, which implies no undefined behavior, but allows for implementation defined behavior. That second clause is what makes the whole thing work, and it is the part worth understanding.

Undefined behavior is a licence for a compiler to do anything, so a program containing it cannot be used to compare two compilers. But implementation defined behavior is different. It is behavior the standard leaves open, where the implementation must document and consistently choose a behaviour. Two compilers that differ there are not both wrong, and the same compiler at two optimisation levels that differ there is a genuine miscompile.

So the generator's job is to produce programs that are legal, deterministic in their observable result, and rich enough to expose the places where a compiler is free to choose.

The project is written in C++, the repository is not archived, and the last push was on 2026-10-08.

The differential test: same program, many compilers, one hash

Here is the mechanism, and it is the reason the generator produces programs with a specific shape.

Each generated program consists of several files, and after being compiled and run it produces a decimal number, which is the hash of all program global variable values. This number is supposed to be the same for all compilers and optimization levels. If the output differs for different compilers and/or optimization levels, the README says you should have encountered a compiler bug.

The design follows from making the observable state global. A hash over global variable values is a single number that captures the entire final state of the program, and it is easy to compare across dozens of compiler and flag combinations. The cost of that choice is that the interesting behaviour has to end up in globals, and the generator is built around that constraint.

The comparison is automated by a script. The README recommends using run_gen.py, which runs the generator for you on a number of available compilers with a set of pre-defined options. The script runs several compilers with several compiler options and runs the executables to compare the output results. If the results mismatch, the test program is saved in the testing/results folder for your analysis.

That last detail is what makes this usable at scale. A failing program is not lost, it is written to disk with the exact configuration that produced the disagreement, so it becomes a regression test rather than a message in a log.

Random, but steered by policies toward the optimiser

The generator does not produce uniform noise, and the README lists three techniques for increasing the probability of triggering compiler bugs.

First, all variable types, alignments and values or value ranges are known at the time of generation. That is what allows the accurate detection of undefined behavior, and it also provides maximum variability of produced code, because the generator can pick any legal value in a range it has already proved safe.

Second, the generation is random, but it is guided by a number of policies which increase the likelihood of applying optimizations to the generated code. The policies are the difference between a random program and a random program that stresses the optimiser. A compiler bug is not in the front end, it is in the pass that turns an expression into something more efficient, and the policies aim the generated code at those passes.

Third, in some cases a high level model of computations with known properties is generated first and then augmented with other random computations. The README says this ensures the code is meaningful and is more similar to code written by a human being.

That third point is a design decision about the shape of the tests rather than their content. A purely random program is unlikely to contain the algebraic pattern, the redundant computation or the branch structure that an optimiser has a bug in. Growing a random program on top of a model with known properties gives the optimiser something to chew on while keeping the result legal.

Two versions, and the one on main is the loop tester

There are two versions of yarpgen, and the README is explicit about which is which.

One is designed to test loops, and it is under development and lives in the main branch. The other is designed to test scalar optimizations and lives in the v1 branch.

That split matters for anyone picking this up, because the version you get by cloning the repository default branch is the loop one, and the published papers describe different work. The loop version and its information are covered by a talk and a paper at PLDI 23, with the paper in the papers directory as yarpgen-pldi-2023.pdf. The scalar version has its own talk and a paper at OOPSLA 2020, published as yarpgen-ooplsa-2020.pdf, and that paper received an ACM SIGPLAN distinguished paper award.

So the repository's main branch is the newer, less settled one, and the older, more heavily published work is on a branch. If you are reading the papers to understand the technique, the OOPSLA 2020 one describes the version most people will encounter as v1, and the PLDI 23 one describes what is on main today.

The layout of the repository supports the split. There is a papers/ directory holding both PDFs, a src/ directory, a scripts/ directory with the driver script and the shims, a docker/ directory, and CMakeLists.txt at the top.

Building it is three cmake commands

The README says building yarpgen is trivial, and that all you have to do is to use cmake.

mkdir build
cd build
cmake ..
make

That is the whole build. There is no dependency list in the README and no configure step beyond pointing cmake at the source directory, which is unusual for a project of this kind and suggests the dependencies are either in the standard library or vendored in the src/ directory.

The CI badges are from Travis for Linux and Mac, and Appveyor for Windows and Ubuntu, and the repository has both.travis.yml and.appveyor.yml at the top level, so the project was built on both of the older CI systems rather than on GitHub Actions. The.github/ directory is also present for repository automation, and there is a SECURITY.md for reporting issues.

There is a check_format.sh script and a.clang-format file at the top level, which is the standard pairing for a C++ project that keeps its formatting consistent, and the.github/ directory alongside a.travis.yml suggests the GitHub side is for repository management rather than for the build itself.

Testing compilers you do not have, and the shims for ISPC and MSVC

The runner script is designed to compare against toolchains you may not have hardware for.

The standard way to test compilers for future hardware, which is not available to you at the moment, is to download the Intel Software Development Emulator, and run_gen.py assumes that it is available in your $PATH. So the same generated program can be run under an emulator for a future microarchitecture and compared against the native run.

For ISPC there is a similar arrangement with an extra step. If you want to test ISPC, you have to make sure it is present in your path, as well as ispc-proxy and ispc-disp, both in the scripts directory. After that you can use run_gen.py with --std=ispc as usual.

MSVC gets the same treatment with a different angle. MSVC can be tested from WSL, with cl.exe installed on the Windows host and driven through cl-proxy, described as the same shim idea as ispc-proxy. It needs its own testing sets, in test_sets_msvc.txt, passed via the --config-file option, and it has a few environment requirements, so the README points at README-msvc-wsl.md for the details.

The set of compiler options is not hardcoded either. You are invited to hack test_sets.txt to add or remove compiler options, which is how you would point the harness at a flag combination your own compiler has.

The shim approach is the interesting part of the design. A script that presents one compiler's command line interface as another's lets the same driver script and the same option sets work across toolchains that would otherwise need separate plumbing.

Where yarpgen is the wrong tool

The boundary is fairly clear, and it comes from what the generator is for.

This is not a general purpose program fuzzer. It is a differential oracle for compilers, and it depends on the property that a legal program must produce the same observable result everywhere. If you are looking for a fuzzer to find crashes in your own application, or a generator to explore your input space, this is the wrong category of tool, and the restriction to well defined programs makes it useless for that job.

It is also useless without several compilers. The comparison is the entire point. One compiler produces one number, and one number cannot disagree with itself. If you have only one toolchain available, the harness has nothing to compare, and the Intel Software Development Emulator path is a partial substitute that only covers future hardware for the compilers you already have.

The third limit is scope. The DPC++ support is described as being in an early stage, and the whole approach requires that the target compiler preserve observable semantics exactly. A compiler that legitimately changes implementation defined behaviour, or a language mode that permits undefined behaviour, is outside the model. That is not a defect, it is the boundary of the technique, but it means yarpgen cannot validate a compiler's entire specification, only the parts where the standard is unambiguous.

The fourth is the practical one of cost. Generating programs and running them across many compilers and option sets is a compute-heavy activity, and the projects that get value from this are compiler and optimiser teams, not application teams.

Licensing, attribution and the bug list you are asked to update

The licence needs a note. The repository is recorded under a custom licence that GitHub cannot classify, with the terms in the LICENSE file, and the README carries an Apache 2.0 licence badge pointing at LICENSE.txt. So the metadata says one thing and the badge says another, and the file is the authority.

For a tool you intend to run inside a compiler development loop, reading the LICENSE file is a step worth doing rather than assuming the badge settles it.

The authorship is a small detail with a large consequence for the project. The People section lists Vsevolod Livinskii, Dmitry Babokin and John Regehr, and the README gives special thanks to Martin Liška and Detlef Riekenberg who used yarpgen to find bugs in various compilers. The names of the users of the tool are recorded alongside the authors.

The bug list is a community artefact. The README says yarpgen managed to find more than 260 bugs in gcc, clang, ispc, dpc++, sde and alive2, and a comparable number of bugs in proprietary compilers. For the open source ones there is a list in bugs.rst, and the README asks that if you have used yarpgen to find bugs in open source compilers, you update that file. There is also an invitation to report experience with proprietary compilers and other tools you might want to validate.

So the repository treats its own bug list as a shared ledger rather than a marketing claim, which is the behaviour you want from a differential testing tool. The one piece of missing information is the 260 figure, which is the authors' own count with no methodology attached.

Editorial conclusion

Adopt yarpgen if you maintain a compiler, an optimiser or a code generation backend and you want a differential oracle for miscompiles rather than a hand written test case. Do not adopt it as a general fuzzing tool for your own application, because it is built to generate well defined programs whose outputs must match exactly, and it needs several compilers installed to be useful. Before running it, build it with cmake as the README describes, and read the bug list in bugs.rst to see whether your compiler is already covered, since the README asks users of open source compilers to update that file.

Frequently asked questions

Official sources

1. intel/yarpgen on GitHub
2. Issues
3. README