RobustBench: a standardized benchmark with a dated leaderboard

What RobustBench standardizes and who it is for

RobustBench was created to track what the authors call the real progress in adversarial robustness. The premise in the README is that thousands of papers exist on adversarial examples, yet it is often unclear which defenses genuinely work and which only produce overestimated robustness. The project answers that by fixing the evaluation procedure rather than by inventing a new attack.

Two artifacts come out of it. The first is a website at robustbench.github.io carrying leaderboards built from many recent papers, with separate views for Linf, L2 and common corruptions. The second is a Model Zoo, a collection of the most resilient models it knows about, packaged so a downstream user can load a checkpoint instead of training one.

The intended users are two groups at once. Researchers need a number that can be compared against other papers without reimplementing the attack pipeline. Applied teams need a checkpoint they can ship behind an image classification service. The benchmark covers Linf, L2 and common corruption settings because those are the settings most studied in the literature, and it is the starting point rather than the final word on adaptive evaluation.

The three eligibility rules that decide which defenses qualify

The interesting engineering decision in RobustBench is a restriction, not a measurement. Since evaluating Lp robustness in general requires adaptive attacks, the project imposes conditions on the defenses it will accept, so that a standardized comparison is possible at all.

The README lists three conditions. A defense must have, in general, non-zero gradients with respect to the inputs. It must have a fully deterministic forward pass, meaning no randomness. And it must not contain an optimization loop. The reasoning given is that defenses violating these principles usually make gradient-based attacks harder without substantially improving resilience, except in cases where they can present concrete provable guarantees.

This is the point where the project draws a line with the adaptive evaluation world. The README says external evaluations based on adaptive attacks are welcomed, especially where AutoAttack flags a potential overestimation, and it treats AutoAttack and adaptive attacks as complementary. For each model the aim is the best known resilient accuracy rather than a single authoritative score.

A reader should hold two conclusions at once. The restriction makes scores comparable, which is the point of a standardized benchmark. It also means a defense excluded by the rules cannot appear on the leaderboard at all, so absence from the leaderboard is not evidence that a method failed.

Where the leaderboard sits against adaptive evaluations and AutoAttack

The README addresses the neighbouring projects directly, and the differences are concrete rather than cosmetic.

Against the AutoAttack leaderboard, the relationship is historical. The AutoAttack leaderboard was the starting point of RobustBench, and the README states that the earlier AutoAttack leaderboard is no longer the reference point for new results. Against the adaptive evaluation site linked in the README FAQ, the difference is methodology. That site focuses on adaptive evaluations, which the README credits as very useful but describes as time-consuming and not standardized by definition. RobustBench argues that robustness can be estimated accurately mostly without adaptive attacks, provided restrictions are placed on the models considered.

The attack libraries form a third cluster. foolbox, cleverhans and advertorch supply implementations of different attacks. RobustBench does not compete with them on that ground. Its added contribution is the standardized benchmark plus the repository of the strongest models, which lets a user obtain one of them in a single line of code according to the README.

For a practitioner, the practical split is this. If you need an attack implementation of your own, use those libraries. If you need a published number for a defense that satisfies the three rules, use this leaderboard. If you need to know whether a specific defense holds up against an attack designed for it, the adaptive route is the one the README points to, and it is a different, slower workflow.

How the model zoo is packaged and shipped

The package is a plain Python distribution. The repository root holds setup.py, requirements.txt, a robustbench package directory, a model_info directory, a tests directory and a MANIFEST.in, with images used by the README and a CITATION.bib for referencing the work. The description in setup.py is precise about scope: the package provides the data for RobustBench together with the model zoo.

Dependencies pin the interesting corners. The project requires torch and torchvision, plus torchdiffeq and geotorch, which relate to the continuous-time and geometry-aware models in the zoo. The attack backend is pulled straight from GitHub at a fixed commit rather than from an index:

git+https://github.com/fra31/auto-attack.git@a39220048b3c9f2cca9a4d3a54604793c68eca7e#egg=autoattack

Two of the pins are exact. gdown is pinned to 5.1.0, and Jinja2 is held at approximately 3.1.2, while torch, numpy, pandas, requests, tqdm and timm carry only lower bounds. A fixed commit for the attack code is defensible for reproducibility, but it means the installed AutoAttack does not receive upstream fixes unless you vendor or repin it yourself.

Weights themselves are not in the repository. They are downloaded from Google Drive, and gdown is the tool for that job. The README warns that if automatic downloading fails, you should install the latest version from the git URL rather than from a package index, which suggests the published wheel lags the development state.

Installing the package and loading the first checkpoint

The README gives one install command explicitly, and it is the git form rather than a package index release. It is presented as the remedy when automatic model downloading misbehaves, but it works as the general installation path for anyone who wants the current tree.

pip install git+https://github.com/RobustBench/robustbench.git

Expect a heavy install. The core requirements are listed in the repository requirements file, and they include the full PyTorch stack plus the pinned attack backend:

torch>=1.7.1
torchvision>=0.8.2
torchdiffeq
geotorch
requests>=2.25.1
numpy>=1.19.4

The setup metadata classifies the package for Python 3, marks it as OS independent, and lists it under the Apache Software License in its classifiers, while the repository itself carries a custom licence that GitHub cannot classify. That mismatch is worth checking against your own compliance process before redistributing anything built on it.

For a first real use, the sequence the README describes is short: install the package, then load a model from the zoo in one line of code. What the truncated tutorial section does not spell out here is the exact loader signature and the checkpoint naming scheme, so you would read the robustbench package and the model_info directory in the tree to confirm the arguments before writing anything against it.

The ImageNet-C preprocessing change that reordered the leaderboard

Two entries in the project news describe exactly how evaluation can silently differ between two runs, which is the strongest argument for using a standardized pipeline at all.

The first is the ImageNet-C fix announced in May 2022. Previously, evaluations resized images to 256 by 256 and took a central crop to 224 by 224. That was unnecessary, because the ImageNet-C images are already 224 by 224. The README states that this change altered the ranking between the top-1 and top-2 entries. A number published before the fix and a number published after it are therefore not comparable, and this applies to anyone who reproduces older results.

The second is the extension of the common corruptions leaderboard on ImageNet with 3D Common Corruptions, that is ImageNet-3DCC. The stated reasons are that it includes more realistic corruptions and that it can assess generalization of models which may have overfitted to ImageNet-C. One caveat is documented: the leaderboard entries are still sorted according to ImageNet-C performance, so the 3DCC results are attached to a list ordered by a different metric.

Both are examples of benchmark hygiene rather than model progress. They also show how a benchmark like this needs a stable definition of the pipeline, and how quickly a fix can invalidate a comparison without changing any model.

Maintenance state and what an upgrade will cost you

The activity picture is thin and should shape expectations. The recent release list contains v1.0 on 2021-11-29, which corresponded to the NeurIPS 2021 Datasets and Benchmarks version of the whitepaper, v1.1 on 2022-09-28 which added timm integration and new models, and a test tag v1.1.1-test on 2023-04-09 that added Tian2022Deeper. The last push to the repository was on 2026-04-14, and the repository is not archived, but no release has shipped since 2023.

That gap has direct consequences for an adopter. The setup.py still declares version 1.1, so anything installed from the index is that 2022 snapshot rather than the current tree. The pinned AutoAttack commit and the exact gdown pin will age, and the Apache Software License classifier in setup.py may not reflect the current LICENSE file.

Expect different activity levels across the parts of this project. The README treats the leaderboard as the live reference in contrast to the AutoAttack leaderboard, while while the pip package and model zoo have not shipped a release since 2022. The surrounding package and the model zoo are a different story, and an evaluation pipeline built on them should pin versions and expect to patch the environment yourself. The paper and the leaderboard remain the durable value; the pip package is a convenience with a stale release.

Editorial conclusion

Use RobustBench if you need a widely accepted, comparable number for an image classifier under Linf, L2 or common corruptions, or if you want one of the published checkpoints rather than training your own. Do not expect a current attack surface from the package: the last release was v1.1 on 2022-09-28, the most recent release tag is v1.1.1-test from 2023-04-09, and the last push was on 2026-04-14, so budget for pinning and for fixing your own environment. Before relying on any leaderboard row, check whether the model uses the corrected ImageNet-C preprocessing, since the README states that fixing the resize and crop pipeline changed the ordering of the top entries.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. RobustBench/robustbench on GitHub