OpenUnlearning: A Single Harness for Comparing LLM Unlearning Methods

The Problem: Unlearning Results That Cannot Be Compared

LLM unlearning research has a reproducibility problem that is partly infrastructural. A method is proposed, evaluated on one benchmark with one metric, and reported as a success. A second method appears, evaluated on a different split with a different metric, and the two numbers are placed in the same table. The comparison is meaningless, but nothing in the tooling prevents it.

OpenUnlearning is aimed at that gap. The README describes it as an easily extensible framework unifying LLM unlearning evaluation benchmarks, and the component table makes the scope concrete: three benchmarks (TOFU, MUSE, WMDP), twelve named unlearning methods, and a metric list that runs from Verbatim Probability through to six membership inference attacks. The intended user is a researcher or engineer who wants to answer a narrow question (does method A forget more than method B without destroying general capability) without writing a separate harness for each benchmark. The June 2025 technical report goes further and describes a meta-evaluation framework covering 450+ models released on Hugging Face, which is a different use case: studying whether the metrics themselves are trustworthy, not just running them.

How the Pipeline Is Assembled: Config, Trainer, Evaluator

The repository is a Python package with a configuration-driven experiment structure. Unlearning and evaluation are separate entry points, which matters because it means you can re-evaluate an existing checkpoint without re-running the training loop. The README's table of contents separates Perform Unlearning from Perform an Evaluation, and the updates note that eval log files are downloaded and cached locally, then refreshed by running python setup_data.py. That caching layer is the reason a metrics change requires a data refresh rather than just a code pull.

The evaluation side is the more interesting half. Metrics include Verbatim Probability, Verbatim ROUGE, Knowledge QA-ROUGE, Model Utility, Forget Quality, TruthRatio, Extraction Strength, Exact Memorization, and six MIA attacks (LOSS, ZLib, Reference, GradNorm, MinK, MinK++). The April 2025 update also added a holdout set to TOFU and support for computing MUSE's privleak on TOFU, which means the two benchmarks now share at least one metric rather than living in separate evaluation worlds. The May 2025 update integrated EleutherAI's lm-evaluation-harness, which is how general-capability benchmarks such as MMLU and GSM8K enter the picture and how WMDP evaluations are run. That integration is the mechanism by which the framework can report a utility cost alongside a forgetting score.

Getting It Running: Conda, Extras, and the Data Refresh Step

The Quickstart in the README gives the environment setup directly. Create a Python 3.11 environment, activate it, then install with an extra:

conda create -n unlearning python=3.11 conda activate unlearning pip install ".[lm-eval]"

The lm-eval extra is what pulls in the lm-evaluation-harness dependency, so install it if you intend to run WMDP or any of the general benchmarks it enables. The README's Quickstart snippet is truncated in the supplied material, so the remaining steps after the pip install are not visible here and I am not going to reconstruct them.

One operational detail is easy to miss and the maintainers call it out explicitly: after merging the latest version, run python setup_data.py. The stated reason is that downloaded eval log files must be refreshed to stay compatible with the latest evaluation metrics. In practice this means a git pull alone can leave you computing new metrics against stale logs, which is exactly the kind of silent mismatch the project exists to eliminate. The README also points to docs/links.md for per-feature resources, and describes a contribution path for adding new methods and benchmarks plus a leaderboard update mechanism.

The Metric Problem Is Not Solved by Having Ten Metrics

Here is the tension in this project. Unifying benchmarks makes comparisons possible. It does not make them meaningful. The README lists Forget Quality alongside Extraction Strength, Exact Memorization and six MIA attacks, and the technical report's meta-evaluation over 450+ models exists precisely because these metrics can disagree with each other. A method can score well on Verbatim ROUGE while remaining extractable under a MinK++ attack, and the framework will happily report both numbers side by side without adjudicating.

That is not a defect in the software. It is a limitation of the field that the software makes visible. But it changes what you should expect from adoption. OpenUnlearning gives you a consistent measurement instrument, not a verdict. If your goal is a single headline number proving that a model has forgotten something, no configuration in this repository will produce one, because the underlying metrics do not agree on what forgetting means.

There is a second, more mundane limitation. The supported model families are enumerated: TOFU covers Llama-3.2, Llama-3.1 and Llama-2; MUSE covers Llama-2; additional support exists for Phi-3.5, Phi-1.5, Gemma and Zephyr. If your model is not on that list, the framework is not a drop-in. Extending it is the stated design goal, but that is work you take on.

Where It Fits Against a General Evaluation Harness

The obvious comparison is EleutherAI's lm-evaluation-harness, which OpenUnlearning integrates rather than competes with. The difference in approach is the axis of measurement. lm-evaluation-harness measures capability: how well a model performs on a task. OpenUnlearning measures removal: whether specific training knowledge has been eliminated, and it does so with metrics that have no analogue in a general harness, such as TruthRatio, Forget Quality and the MIA attack family. The integration means you get both in one run, which is the practical argument for using OpenUnlearning instead of assembling the two yourself.

A different kind of alternative is the original locuslab/tofu repository. The README states plainly that OpenUnlearning replaces it and that the old codebase is no longer maintained. For anyone with TOFU experiments in that repo, the choice is not between two live options; it is between migrating and running unmaintained code. The migration is not free, since the configuration format and the data setup step differ, but the direction is unambiguous.

Maintenance, Licence and the Cost of Keeping Up

The repository is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is the permissive end of the spectrum and imposes no copyleft obligation on your own code. This is a description of the licence text, not legal advice; if you are shipping a product that depends on it, have your own counsel read the LICENSE file.

Maintenance cost shows up in two places. First, the data refresh requirement means upgrades are not purely a code operation: python setup_data.py has to run after a merge, and the README frames this as required rather than optional. Second, the project has moved quickly. Between February and June 2025 the updates added two methods, a benchmark, six MIA attacks, two metrics, a holdout set, an lm-evaluation-harness integration and a technical report. A team pinning to a specific commit will avoid churn; a team tracking main should expect to re-run setup and re-check metric definitions on a regular basis. There are no retrieved releases, so version pinning happens at the commit level rather than through tagged artifacts.

What to Verify Before You Commit

Check three things in order. First, confirm your model family is covered: the README's list is explicit, and anything outside it means writing an extension. Second, decide which metric you will report and read docs/links.md for the resource behind it, because the choice between Forget Quality, Extraction Strength and an MIA attack is a choice about what claim you are making. Third, run python setup_data.py on a clean clone and confirm the eval logs land where the evaluator expects, since a stale log cache is the failure mode the maintainers warn about by name.

If those three checks pass, the framework does what it says: one pipeline, three benchmarks, twelve methods, and a metric set broad enough that you can see where methods disagree. That last part is the real product. The disagreement is the finding.

Editorial conclusion

Adopt OpenUnlearning if you are comparing unlearning algorithms on TOFU, MUSE or WMDP and want one pipeline instead of three forks; the README notes it replaces the unmaintained locuslab/tofu codebase, so existing TOFU users should migrate. Do not adopt it if you need a production data-deletion service or a benchmark it does not implement. Before committing, run python setup_data.py, confirm your target model family appears in the supported list, and check whether the metric you intend to report is one of the 10+ implemented ones or something you will have to add yourself.

Official sources

1. Issues
2. License: MIT
3. locuslab/open-unlearning on GitHub
4. README