clip_benchmark measures CLIP-like models across tasks and datasets, and writes JSON so you can diff runs

One harness, several tasks, results as files

The stated goal of the repository is to evaluate CLIP-like models on a standard set of datasets across different tasks such as zero-shot classification, zero-shot retrieval and captioning. It is a measurement instrument, not a model.

The CLI is the primary interface, described as the easiest way to benchmark the models. You specify the model, the dataset and the task, and the results are written into a JSON file.

The command names the dataset, the task, the pretrained checkpoint, the model architecture, the output file and the batch size.

The output file is the point of the design. The CIFAR-10 example in the README shows what comes out.

{
    "dataset": "cifar10", "model": "ViT-B-32-quickgelu", 
    "pretrained": "laion400m_e32", "task": "zeroshot_classification",
    "metrics": {"acc1": 0.9074, "acc5": 0.998}
}

Each file records the dataset, the model architecture, the pretrained checkpoint, the task and a metrics object. Because every run is a separate file, two evaluations of the same model on the same dataset produce two files you can compare directly, and a sweep produces many files that can be aggregated.

The aggregation command takes those files and produces a table.

clip_benchmark build benchmark_*.json --output benchmark.csv

That build step is what turns a pile of evaluation runs into something you can put in a paper or a tracking spreadsheet. The filename convention in the recommended invocation encodes dataset, pretrained, model, language and task, so the files sort meaningfully on disk.

The dataset list is a file you download, not a flag you set

The recommended workflow starts with a models.txt file listing the models to evaluate, in a model name and provider pair.

ViT-B-32,openai

The dataset list comes from the repository as a text file, fetched with wget.

wget https://raw.githubusercontent.com/LAION-AI/CLIP_benchmark/main/clip_benchmark/datasets/webdatasets.txt

That file is the index of available datasets, and the evaluation command is then pointed at it.

clip_benchmark eval --pretrained_model models.txt \
    --dataset "webdatasets.txt" \
    --dataset_root "https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" \
    --output "benchmark_{dataset}_{pretrained}_{model}_{language}_{task}.json"

Two things about that invocation are worth understanding. The --dataset_root is a URL template with a {dataset_cleaned} placeholder, which resolves to a Hugging Face datasets tree. So the datasets themselves are hosted as webdataset shards, and the harness fetches them rather than expecting a local copy.

Second, the output filename is a template. Every placeholder in it is filled from the run's parameters, which is why a single command with a list of models and a list of datasets produces one uniquely named JSON per combination. That is a sensible design for a sweep, and it means you can interrupt and resume a long run without losing track of which files already exist.

The dataset sources are broader than the webdataset list. The features list names various datasets from torchvision, tensorflow datasets and VTAB, support for various multilingual datasets for classification and retrieval, and support for compositionality tasks.

Metrics change shape with the task

A harness that reports one metric for everything would be useless, and this one does not do that. The metric names in the output JSON depend on the task and on the dataset type.

For single-label classification the metrics object carries acc1, acc5 and mean_per_class_recall. The Japanese CLIP example shows all three.

{"dataset": "imagenet1k", "model": "ViT-B-32-quickgelu", "pretrained": "rinna/japanese-cloob-vit-b-16", "task": "zeroshot_classification", "metrics": {"acc1": 0.54636, "acc5": 0.72856, "mean_per_class_recall": 0.54522}, "language": "jp"}

For a multi-label dataset the metric changes completely. VOC2007 is described as a multi-label classification dataset, and the result file contains a single metric.

{"dataset": "voc2007_multilabel", "model": "ViT-B-32-quickgelu", "pretrained": "laion400m_e32", "task": "zeroshot_classification", "metrics": {"mean_average_precision": 0.7627869844436646}}

The README explains that mean average precision, or mAP, is computed in the context of multi-label classification, and links a walkthrough of the metric.

This matters more than it might look. Any tooling that aggregates your results has to know which metric applies to which task and dataset, and there is no single field to read. It also means a table built with a naive script that assumes acc1 exists will produce empty cells for retrieval and multilabel runs.

The language field in the JSON is the other variable to keep track of. It appears on the multilingual runs, and it is one of the filename placeholders, so it is part of the result identity rather than an afterthought.

Adding a model means satisfying a three-method contract

Adding a CLIP model that the harness does not know is a three step procedure, and the third step is the interesting one because it defines what the harness needs from any model.

First, add an identity file to load the model in clip_benchmark/models. Second, define a loading function that returns a tuple of model, transform and tokenizer, with clip_benchmark/models/open_clip.py given as the example. Third, add the function into TYPE2FUNC in clip_benchmark/models/__init__.py.

Then come the requirements. The new tokenizer and model must be able to do three things, as listed in the README against the OpenAI CLIP usage documentation.

- tokenizer(texts).to(device), where texts is a list of strings - model.encode_text(tokenized_texts), where tokenized_texts is the output of the previous call - model.encode_image(images), where images is an image tensor produced by the transform

That is the whole integration surface. Three methods, one tuple to return, and a registry entry. It is a deliberately thin contract, which is why Japanese CLIP and NLLB CLIP could be added without restructuring anything.

The feature list names OpenCLIP pre-trained models, Japanese CLIP and NLLB CLIP for general multilingual abilities as the supported model families. The README also shows the non-OpenCLIP path in practice, where a model_type flag of ja_clip selects the loader and the pretrained name identifies the checkpoint on the hub.

The invocation sets model_type to ja_clip to select the loader, names the pretrained checkpoint, sets language to jp and task to zeroshot_classification, and passes imagenet1k as the dataset with a dataset_root placeholder.

Four dataset families, each with its own extra dependency

The dataset support is not uniform, and the README is clear that each family carries an install step.

The base case is the webdataset list, which needs nothing beyond the package itself. torchvision datasets are named directly, for example the CIFAR-10 invocation uses --dataset=cifar10 with no prefix.

VTAB needs its dedicated package, installed as task_adaptation==0.1, and the dataset is then named with the vtab/ prefix.

The install command is `pip install task_adaptation==0.1`, and the evaluation then names the dataset with the vtab/ prefix, for example `clip_benchmark eval --dataset=vtab/eurosat --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`.

The README points at a specific line in clip_benchmark/datasets/builder.py for the full list of VTAB dataset collections.

TensorFlow datasets need tfds-nightly and timm, and the dataset name follows the template tfds/<DATASET_NAME>.

The install command is `pip install timm tfds-nightly`, and the evaluation uses the tfds/ prefix, for example `clip_benchmark eval --dataset=tfds/cifar10 --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`.

COCO captions retrieval is a separate case that needs pycocotools in addition, for example pip install pycocotools, because the caption data is read with the official COCO tooling.

This is a real usability wart. The evaluation command is uniform, but the environment is not, and a sweep that touches all four families needs three extra installs prepared in advance. The setup.py does encode two of these as extras, with a vtab extra pinning task_adaptation==0.1 and timm>=0.5.4, and a tfds extra for tfds-nightly, so pip install clip_benchmark[vtab] is the tidier route.

The packaging says 3.6, the release history says otherwise

There is a gap between what the packaging metadata claims and what the project has actually been doing, and it is worth knowing before you install it.

The setup.py declares python_requires of 3.6 or newer and lists classifiers for Python 3.6, 3.7 and 3.8. It also carries the development status classifier Development Status :: 2 - Pre-Alpha. The requirements.txt asks for torch>=1.8.1, torchvision>=0.8.9, tqdm, scikit-learn with an upper bound of 2, open_clip_torch, pycocoevalcap, webdataset and transformers.

The release history is different. Version 1.6.0 shipped on 2023-12-02, 1.6.1 on 2024-02-19 and 1.6.2 on 2025-07-25, and the last push to the repository was on 2026-07-23. So the most recent release is from mid 2025 and the most recent commit is from mid 2026, with the gap between them meaning unreleased work sits in the branch rather than in a published artifact.

There is a related detail. The setup.py url points at github.com/mehdidc/clip_benchmark and the author field names Mehdi Cherti, while the repository now lives under the LAION-AI organisation and the README links results from LAION research papers. The project was built by Mehdi Cherti and the URL was never updated.

The licence is MIT, with a LICENSE file at the root and an MIT classifier in the metadata. The repository also carries HISTORY.md, AUTHORS.rst, CONTRIBUTING.rst, a Makefile with test and coverage targets, tox.ini and a tests directory, so the project has a conventional Python packaging layout even if some of the metadata is stale.

For a harness you will run yourself, the practical advice is to install from the repository rather than from the pinned release, and to expect to adjust the Python version constraint yourself if you are on a modern interpreter.

Where clip_benchmark is the wrong tool

The first limitation is that it is offline and local. Every run downloads a dataset and runs inference on your hardware. There is no hosted endpoint, no batch submission and no result caching, so a sweep across many models and many datasets is a long local job with a large download footprint.

The second limitation is that the dataset set is fixed by the harness. You can add a model easily, as the three step procedure shows, but the README documents no procedure for adding an arbitrary dataset of your own, only for reaching the datasets already present in torchvision, tensorflow datasets, VTAB and the webdataset list. If your product depends on a private or domain specific dataset, this harness will not score it.

The third limitation is that the supported tasks are a fixed list: zero-shot classification and zero-shot retrieval, linear probing, and captioning. There is no detection, no segmentation, no grounding, no attribute or counting evaluation. The compositionality tasks are listed as supported, but they are a specific add-on rather than a general task framework.

The fourth is comparability across time. Because results are per-run JSON files assembled by a build command, the table you produce is only as comparable as the runs behind it. Changing the harness version, the dataset revision or the checkpoint can move a number without the model changing, and the result files record the dataset, model, pretrained name, task and language but not the harness version or the dataset revision.

For a paper-grade leaderboard the linked projects are the place to look: OpenCLIP results for 38 datasets, the reproducible scaling laws for contrastive language-image learning work, and the scaling laws work on comparing open foundation language-vision models and datasets.

How it compares with writing your own evaluation loop

The alternative is to write the evaluation loop yourself, which is what most people do before they find this harness.

A homegrown loop typically starts with a dataset class, a preprocessing transform matching the model's training pipeline, a prompt template per class, and a loop that encodes images and text and computes logits. Then there is the retrieval version, the multilabel version, the captioning version with its own metric implementation, and the linear probe. Each of those is a small amount of code and each is a place where two implementations quietly disagree.

The value of this harness is consistency across models rather than any single evaluation. Because the transform, the tokenizer and the prompt handling come from one place, a comparison between two checkpoints is a comparison of checkpoints and not of two slightly different evaluation scripts. When the Japanese CLIP loader and the OpenCLIP loader sit behind the same three method contract, the difference in the resulting acc1 is attributable to the model.

The second value is the aggregate. The build command producing a CSV from a directory of JSON files is exactly the thing that every hand-rolled harness grows eventually, usually worse. Doing it that way means one command rather than a notebook.

The third value is the model registry. The TYPE2FUNC pattern with a fixed contract is a small piece of design that removes a large amount of duplication, and it is what makes the harness extensible to models it has never heard of.

Against that, a homegrown loop is smaller and does exactly what you want. The harness is worth it when you are comparing models across many checkpoints on many datasets, and it is overhead when you are evaluating one model on one dataset once.

Editorial conclusion

clip_benchmark fits a research team that needs to compare vision-language checkpoints on a common set of datasets and tasks, and it fits a team that wants per-run result files it can diff, aggregate into a table and feed into a paper. It does not fit a production image search system, since it is an offline evaluation harness that downloads datasets and runs inference locally, and it does not fit a team that needs a leaderboard for production models, because the task list and dataset set are fixed in the harness rather than configurable at run time. Before adopting, check which of your datasets appear in the webdatasets.txt list or in the builder, confirm the extras for your source are installed, for example task_adaptation==0.1 for VTAB or timm tfds-nightly for TensorFlow datasets, and read the result JSON schema so your aggregation tooling matches the metric names per task, since a classification task and a multilabel task report different metrics.

Frequently asked questions

Official sources

1. Issues
2. LAION-AI/CLIP_benchmark on GitHub
3. License: MIT
4. README
5. Releases