SafeIMG ships four evaluation tracks and one generator

One generator, twelve scenarios, and a shift measured in one direction

The claim the benchmark rests on is distribution shift. Samples are produced with GPT Image 2 rather than only with GANs or earlier diffusion models, and the stated purpose is to expose how much that moves the ground under detectors trained on older distributions.

Everything in the dataset comes from that single generator. There is no second generator to compare against, so what the benchmark measures is distance from the older checkpoints, not spread across the current ones. A detector that fails here has failed against one new generator, which is a real finding but a narrower one than the taxonomy suggests.

The scope is twelve scenarios split into eight public-safety categories and four individual-safety ones, from natural disasters and transportation accidents through private receipts and chat records. Each category is tied to an evidentiary function rather than a visual theme, which is what makes the set about decisions instead of about pictures.

The categories are written as claims rather than as content. Natural disasters are claims about disaster impact and emergency response, private receipts are claims about payments, refunds, reimbursement, or disputes, and fabricated scene evidence is claims about presence, status, identity, or endorsement. That framing is what separates this from a detector trained on generic fake photos: the question is never only whether an image is generated, but whether it would change a decision someone has to make.

Four evaluation tracks instead of one real or fake label

The benchmark is organised as four tracks with different inputs and different expected outputs, and the table defining them is the clearest statement of what it measures.

Image authenticity takes one image and expects `real` or `ai_generated`. Artifact diagnosis takes one generated image and expects suspicious regions with artifact types. Rationale alignment takes an image plus model explanations and expects atomic visual reasons, which is how a model's evidence gets compared against the human annotations. Propagation robustness takes a degraded or re-encoded image and expects an authenticity prediction.

The third and fourth tracks are what separate this from a binary benchmark. Alignment asks whether the model points at the same things the annotators pointed at, and robustness asks whether the answer survives the compression and re-encoding that happen when an image is shared online, which is the condition under which a fake would actually do harm.

Seven artifact types, split between local defects and scene-level conflicts

Annotation is fine-grained rather than a single label. For each retained image the dataset preserves the risk category, the scenario instance, the complete generation prompt, and the media form. On top of that, an annotation may carry one or more suspicious regions, an artifact type for each region, a natural-language explanation of the concrete issue, and an image-level authenticity label.

The named artifact types are `Commonsense`, `Faces`, `Fonts`, `Physics`, `Hands`, `Lighting`, and `Other`. Those seven do not sit at the same level. `Fonts`, `Hands`, and `Lighting` are local, checkable defects you could in principle point a bounding box at. `Commonsense` and `Physics` are scene-level, meaning the anomaly is a conflict with how the world works rather than a malformed object.

That split is the reason the diagnosis track exists. A detector that catches garbled text has found an artifact, but a generator that produces a plausible-looking room with implausible physics has produced one too, and only the second kind is hard.

The repository ships evaluation harnesses and a loader, not a detector

The top level is worth reading as a specification of scope. It contains `evaluation_vlm.py` and `evaluation_vlm_concurrent.py` for running multimodal models, `evaluation_ai_supported_by_human.py` and `evaluation_human_covered_by_ai.py` for the two directions of the model and human comparison, `evaluation_spec.py`, `explanation_evaluation_utils.py`, `load.py`, `requirements.txt`, a `LICENSE`, an `assets/` directory, and the README.

What is absent is as informative as what is there. There is no training script, no model weights, no configuration files, and no data. The dataset lives on HuggingFace at `Snowstorm1492/SafeIMG`, the project page is at `safeimg.github.io`, and the images are not in the repository.

The two directionally named files are the closest trace of the detector side of the benchmark. They evaluate a forensic detector against a human baseline and a human against the model's output, so the detector implementations are assumed to be supplied by whoever runs the evaluation.

The requirements file pins the vision stack and leaves openai floating

The dependency list is short and mostly bounded, which is unusual for a vision repository and worth reading closely:

torch>=2.0,<3
torchvision>=0.15,<1
datasets>=2.14,<5
Pillow>=9.5,<12
numpy>=1.23,<2
scipy>=1.9,<2
opencv-python>=4.7,<5
scikit-image>=0.20,<1
openai

Every package carries an upper bound except one. `openai` appears with no version constraint at all, which is the one dependency that makes network calls and therefore the one most worth pinning yourself. The presence of `opencv-python` rather than the headless build also means a graphical stack gets pulled in on a machine that may not have one.

There is no install command anywhere in the documentation, only this file, so setting up means reading the nine lines above and deciding the versions yourself.

The stack itself says something about the intended audience. Torch and torchvision sit at the bottom because the forensic detectors being evaluated are deep models, `datasets` pulls from HuggingFace where the benchmark actually lives, and Pillow, numpy, scipy, opencv-python, and scikit-image cover the compression and re-encoding work the propagation robustness track needs.

The dataset, the code, and the paper all appeared in the same month

The news section records two entries and both are dated 2026-07: the dataset and evaluation code released, and the paper released. The default branch was last pushed on 2026-07-28, and the repository publishes no GitHub releases at all, so there is no versioned artifact to pin.

Licensing is the other loose end. GitHub reports the license as NOASSERTION, which means it could not classify it, while a `LICENSE` file is present at the root. If you need a clear grant for redistribution or for a derived detector, read that file rather than trusting the badge, and check whether the synthetic depictions are covered by it or carved out.

There is a warning at the top of the documentation that belongs in your own notes before you use any of this. The images depict disasters, accidents, violence, emergencies, and transactions, they are for research and evaluation only, and they must not be presented as records of real events.

Filtering happens before annotation, so the discarded samples leave no trace

The construction pipeline is five stages, and the order is what shapes the dataset. A risk-oriented taxonomy defines the public and individual safety categories. A scenario-space specification writes a risk summary, a content list, and a media-form list for each category. Prompt construction expands each scenario using a 5W+1H narrative structure with category-specific constraints.

Then generation and filtering together: images are produced with GPT Image 2 and invalid or severely off-topic samples are removed. Human annotation comes last, on whatever survived.

The consequence is that the filtering step is a black box in the released artifact. Nothing in the repository says how many images were generated before removal, what counted as invalid, or what threshold decided severely off-topic, so the category counts you see are post-filter counts and the yield per scenario is unknowable from what is published.

Editorial conclusion

SafeIMG is worth using if your detector or multimodal model has only been tested on general natural images, because the distribution it exposes is deliberately unlike that: generated news photographs, transaction records, chat screenshots, and identity endorsements rather than scenery. Treat the numbers with care. Every image comes from one generator, GPT Image 2, so the shift the benchmark measures is shift away from older GAN and diffusion checkpoints rather than variation across current ones, and the repository contains no detector, no training script, and no configuration files, only evaluation harnesses and a loader. Load the data from the HuggingFace dataset rather than expecting it in the repository, pin `openai` yourself since the requirements file leaves it unbounded, and read the warning at the top before you publish any of it: these images must never be presented as records of real events.

Frequently asked questions

Official sources

1. Issues
2. README
3. Snowstorm1492/SafeIMG on GitHub