# Defects4J gives research 854 Java bugs with a failing test and a fixing commit

> Defects4J is a collection of reproducible bugs from 18 open-source Java projects, where every bug has a triggering test that fails before the fix and passes after it, and the whole set is driven by a Perl command line framework. It fits a software engineering researcher running controlled experiments on real faults, and it does not fit a team looking for a bug tracker or a test framework.

**rjust/defects4j** — A Database of Real Faults and an Experimental Infrastructure to Enable Controlled Experiments in Software Engineering Research

- Repository: https://github.com/rjust/defects4j
- Stars: 1,006 · Forks: 385
- Language: Perl
- License: MIT
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/rjust-defects4j

## What makes a bug count is a test that fails before the fix

Defects4J is a collection of reproducible bugs and a supporting infrastructure with the goal of advancing software engineering research. The repository is at version 3.0.1 and is written in Perl, with the project itself being the Java code it analyses.

The definition of membership is strict, and the list of properties is the actual contribution. Each bug has an issue filed in the corresponding issue tracker, with the issue tracker identifier mentioned in the fixing commit message. It is fixed in a single commit. The maintainers manually minimized it, pruning out irrelevant changes in the commit such as refactorings or feature additions. It is fixed by modifying the source code, as opposed to configuration files, documentation or test files. And a triggering test exists that failed before the fix and passes after it, with the test failure neither random nor dependent on test execution order.

That last property is the one that makes the dataset usable. A benchmark that runs a flaky test produces noise, and noise in a controlled experiment is worse than no experiment. The explicit exclusion of order-dependent failures is the kind of curation that makes the difference between a dataset and a pile of commits.

The buggy and fixed revisions are labelled with the id suffixed by b and f respectively, where the id is an integer, so a bug is always a pair of checkouts.

## Eighteen projects, and the bug counts are wildly uneven

Defects4J contains 854 bugs, plus 10 deprecated bugs, from the open-source projects listed in the README.

Closure dominates with 174 active bugs from closure-compiler, followed by JacksonDatabind with 110 from jackson-databind, and Math with 106 from commons-math. Then come Jsoup with 93, Lang with 61 from commons-lang, Compress with 47, Cli with 39 from commons-cli, Mockito with 38, Collections with 28, Chart with 26 from jfreechart, JacksonCore with 26, Time with 26 from joda-time, JxPath with 22 from commons-jxpath, Codec with 18, Gson with 18, Csv with 16 from commons-csv, and JacksonXml with 6 from jackson-dataformat-xml.

That distribution matters for anyone planning an experiment. A study of a specific library has a small pool, and some of the smaller projects have fewer than twenty bugs, which is usually too few for a meaningful statistical claim. The large pools are in the Google-maintained and Apache projects, which is also where the source code and issue histories are best documented.

There is also a concentration issue in the popular libraries. commons-lang, commons-collections and commons-codec are the classic subjects of mutation testing and fault localisation research, which means the dataset has been used on those projects many times, and a new experiment on them will be easier to compare against prior work and harder to claim as novel.

## Deprecated bugs are separated out, and the reason is Java version drift

The README marks a set of bugs as deprecated, and the reason given is a change in the Java runtime rather than a change in the defect.

Due to behavioral changes introduced by newer Java versions, some bugs are no longer reproducible. Defects4J therefore distinguishes between active and deprecated bugs. Active bugs can be accessed through active-bugs.csv. Deprecated bugs are removed from active-bugs.csv, but their metadata is retained in the project directory, and deprecated-bugs.csv details when and why a bug was deprecated.

The project does not re-enumerate active bugs because publications using Defects4J artifacts usually refer to bugs by their specific bug id. That is a sensible stability decision for a research dataset, where a published experiment must remain reproducible against a known set of bug identifiers, and it means the deprecation list grows over time rather than the active list being renumbered.

The deprecated counts are small relative to the totals. Lang has four deprecated bugs, 2, 18, 25 and 48, Cli has 6, Closure has 63 and 93, JacksonDatabind has 65 and 89, and Time has 21. The framework deliberately documents the reason for each, which is what a researcher needs in order to argue in a paper that a deprecation was not cherry-picking.

## Reproducibility depends on a Java version and a timezone

The Reproducibility section is the part of the README that will decide whether your experiment runs, and it is specific about two things.

The first is the Java version. All bugs have been reproduced and triggering tests verified using Java 11, with the CI configuration pointing at the specifics. Using a different version of Java might result in unexpected failing tests and/or non-reproducible bugs.

The second is the timezone. Defects4J generates and executes tests in the timezone America/Los_Angeles. If you are using the bugs outside of the Defects4J framework, set the TZ environment variable to America/Los_Angeles and export it. The README is blunt that not setting this option results in unexpected failing tests.

That is not a stylistic preference. A date-sensitive test that assumes local time will pass in the timezone the maintainers used and fail in yours, and in a research context a test that fails on the fixed version is indistinguishable from a broken environment.

The third requirement is the interface itself. The project recommends using Defects4J through the provided command-line interface, because all reproducibility tests within Defects4J rely on that interface. If you use the bugs outside the framework, you need to make sure you use the Defects4J export command to obtain the relevant metadata, such as source and test directories, classpath entries and sets of tests. Reproducing a bug means reproducing its environment, not just checking out two revisions.

## The setup is a Perl toolchain, and init.sh does the heavy lifting

The requirements are Java 11, Git 1.9 or newer, Subversion 1.8 or newer, Perl 5.0.12 or newer, and cpanm. The version history is noted too, since Defects4J version 2.x required Java 1.8, and version 1.x and 0.x required Java 1.7.

The Perl modules are all listed in the cpanfile, and on many Unix platforms they are installed by default. The setup steps are four.

Clone the repository, then initialise it. The initialisation step downloads the project repositories and external libraries, which are not included in the git repository for size purposes and to avoid redundancies:

```
git clone https://github.com/rjust/defects4j
cd defects4j
cpanm --installdeps .
./init.sh
```

Then add the executables to your PATH:

```
export PATH=$PATH:"path2defects4j"/framework/bin
```

And check the installation with:

```
defects4j info -p Lang
```

On some platforms such as Windows you might need to use perl "fullpath"\defects4j where these instructions say to use defects4j.

The reason for the initialisation step is worth noting. The actual project sources are not in this repository, they are downloaded by init.sh, which is why project_repos/ exists as a directory in the tree. A fresh clone is therefore not a usable installation, and the first thing any user does is run a script that pulls a large number of external repositories. That is a deliberate design for size, and it also means the setup needs network access to many hosts.

## There is a Docker path that bakes the environment in

The repository carries a Dockerfile and a docker-compose.yml, and they encode the reproducibility requirements as configuration.

The Dockerfile starts from ubuntu:20.04 and installs openjdk-11-jdk, git, build-essential, subversion, perl, curl, unzip, cpanminus and make. It then sets JAVA_HOME to /usr/lib/jvm/java-11-openjdk-amd64 and TZ to America/Los_Angeles, and goes further than the env var by linking the zoneinfo file into /etc/localtime and writing the zone name into /etc/timezone, so the container clock itself is in the right zone rather than just reporting one.

The rest of the image clones Defects4J from GitHub into /defects4j, runs cpanm --installdeps ., runs ./init.sh, and adds /defects4j/framework/bin to the PATH.

The compose file wraps that image with a workspace volume, the same two environment variables, TZ and JAVA_HOME, a published port 8080 to 8080, and stdin_open and tty set so the container can be used interactively. That is the practical detail that matters: Defects4J is a command line tool, and a container that drops the interactive session is not much use for a research tool where you run dozens of commands.

The top level of the repository shows the structure. framework/ holds the tooling, project_repos/ holds the checked-out project sources, developer/ holds contributor material, and there is a cpanfile, a license.txt, an init.sh and a .github/ directory with the CI workflow whose badge sits in the README header.

## Where Defects4J is the wrong tool

The first wrong-tool case is treating it as a bug tracker or a test framework. Defects4J holds 854 curated, reproduced faults with fixing commits and triggering tests. It is not a place to file new issues, it does not run your project's test suite, and the bugs it contains are eighteen years of other people's history.

The second is a project that needs recent faults. The dataset is fixed at 854 bugs from long-lived libraries, and the last few years of bug fixes in those projects are not in it. A study of how developers fix bugs in modern Java with modules, records or new language features needs a different source, and Defects4J will not have it.

The third is the environment cost. Java 11 is pinned, a specific timezone is required, Subversion and a Perl toolchain have to be present, and init.sh has to download the project sources. The Docker image solves the reproducibility but adds a container in the middle of a workflow that is otherwise a sequence of shell commands, and a researcher who wants to run the framework across many machines or many experiment configurations will be running containers rather than scripts.

The fourth is a subtle scientific one. Because the maintainers manually minimized the fixing commits, the commits are not what a real developer would have written. That makes the dataset excellent for fault localisation, where the size and shape of the patch is the signal, and less faithful for studies of how developers actually work, where a real fix commit contains the refactoring and the unrelated change that the minimization removed.

## Conclusion

Defects4J fits a software engineering researcher who needs real, reproducible faults with a known fix to run experiments against, and it does not fit a team that wants a bug tracker, a CI system or a test runner. Before using it, set the TZ environment variable to America/Los_Angeles and run on Java 11, because the project states that both are required for the triggering tests to behave as documented.

## FAQ

### What is Defects4J?

Defects4J is a collection of 854 reproducible bugs, plus 10 deprecated, from 18 open-source Java projects, together with a supporting infrastructure aimed at software engineering research. Every bug has an issue tracker reference, a single fixing commit, and a triggering test that fails before the fix and passes after it.

### What are the requirements for running Defects4J?

Java 11, Git 1.9 or newer, Subversion 1.8 or newer, Perl 5.0.12 or newer, and cpanm. After cloning, run cpanm --installdeps . and ./init.sh, which downloads the project repositories and external libraries that are not included in the git repository.

### Why does Defects4J require the America/Los_Angeles timezone?

Defects4J generates and executes tests in that timezone, and the README states that not setting the TZ environment variable to America/Los_Angeles and exporting it results in unexpected failing tests. The Docker image sets the same variable and links the zoneinfo file into /etc/localtime.

### What is the difference between active and deprecated bugs in Defects4J?

Some bugs are no longer reproducible because of behavioural changes in newer Java versions. Deprecated bugs are removed from active-bugs.csv but keep their metadata in the project directory, and deprecated-bugs.csv records when and why each was deprecated. The active list is not re-enumerated, so published bug identifiers stay valid.

## Sources

- [Issues](https://github.com/rjust/defects4j/issues)
- [License: MIT](https://github.com/rjust/defects4j/blob/master/LICENSE)
- [README](https://github.com/rjust/defects4j/blob/master/README.md)
- [rjust/defects4j on GitHub](https://github.com/rjust/defects4j)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/rjust-defects4j