APSD: measuring whether an adversarial perturbation is visible to a person

Why pixel metrics disagree with the eye

The paper asks a single question, how stealthy is an adversarial perturbation to the human eye, and the reason it needs a dataset is the answer to the obvious follow up. Widely used metrics such as the Lp norms, PSNR and SSIM were not designed specifically for subtle adversarial perturbations, and they can disagree with human visual perception. That disagreement is a practical problem for anyone who has to decide whether a defence or an attack is subtle enough to matter, because the number on the dashboard and the judgement of the person looking at the image may point in different directions. The work offers a systematic study of pixel-level adversarial stealthiness assessment from both a subjective and an objective direction, and it introduces three things. There is a new subjective assessment criterion that covers both hardly perceptible and clearly perceptible perturbations, APSD, a large-scale dataset of human opinion scores, and A2SM, an attention-based model built to mimic human scoring habits so that stealthiness can be predicted without running a study for every image.

A two-step protocol with amplified difference maps

The subjective protocol is the part that is reusable, and it has two steps. In the rough assessment, observers inspect a clean and adversarial image pair for a fixed period and classify the perturbation as strong stealthiness, where the difference is hardly perceptible, or weak stealthiness, where the difference is readily perceptible. The fine-grained step then branches on that answer. For strong stealthiness, observers use progressively amplified Scaled Grayscale Difference Maps to distinguish subtle pixel-level differences that are invisible in the original pair. For weak stealthiness, the observers instead assess the degree of visible degradation or annoyance. The resulting scores are aggregated into a Mean Opinion Score in the range zero to one, where a larger value means better stealthiness, which is the label the dataset ships. The repository also includes a screenshot of the rating software itself, so the interface the observers used is visible rather than described.

Ten thousand pairs, four hundred references, sixty observers

The dataset statistics table is where the scale becomes clear. APSD holds 400 reference images and 10,586 clean and adversarial pairs drawn from more than 200 image categories, with 12 attack methods across 5 attack families. Sixty human observers took part, and every adversarial image was evaluated by 15 of them, with the Mean Opinion Score as the label. The images used for the subjective study were 512 by 512. The design intent is stated as covering a broad range of perceptual stealthiness, so that models trained on it learn both subtle and visually obvious adversarial distortions rather than only one end of the spectrum, which is what makes the set useful as a benchmark instead of a collection of near-invisible perturbations.

Twelve attacks across five families

The coverage is deliberately broad, and the families matter more than the count. The gradient-based group contains FGSM, MIFGSM, PGD and NES, which are the standard first-order and iterative white box attacks. The optimization-based group has CW, the classic targeted attack with a different loss. The generative model-based group is the largest and the most interesting for this dataset, with CDP, AdvGAN, GAP and GUAP, that is, attacks that generate the perturbation rather than optimise a perturbation, and they produce qualitatively different artefacts from an optimised one. The pixel-based group covers SimBA and Pixel, and the block-based group has Square. The README shows a distribution figure of the generated images across attacks, with the suffix of each attack indicating the corresponding parameter, which tells you the set is parameterised rather than one image per method.

A2SM and the reported agreement with human scores

The objective half of the work is A2SM, the Attention-based Adversarial Stealthiness Assessment Model, and the mechanism the README describes is a combination of global and local features with self, cross and spatial attention mechanisms, which is how it mimics human scoring habits rather than a pixel difference. The headline number is stated in the highlights table, where A2SM achieves 0.984 PLCC and 0.978 SROCC on APSD, and the label describes that as high HVS consistency, that is, agreement with the human visual system. Those two coefficients are the standard measures for a rating prediction system, one for linear correlation and one for rank correlation, and both being close to one means the model tracks both the magnitude and the ordering of human judgements. A reader should treat the figures as the authors' own evaluation on the dataset they built, and note that the value of the dataset for that purpose depends on the protocol described earlier being followed by anyone who extends it.

Downloading the package and the directory layout

The dataset is distributed through the project repository, with the download link on the project page pointing at a hosted archive. The README states that the released dataset should contain the clean images, the adversarial images and their corresponding human subjective scores, and it recommends a local organisation with a reference folder and an adversarial folder, the latter split into one subdirectory per attack, alongside a score file, and the README says the exact released names should follow the actual package:

The README adds a note that the exact released directory and file names should follow the actual dataset package, and that the tree should be updated if the final release uses a different structure. That caveat deserves serious attention, because a script that assumes the folder names above will break silently if the release layout differs, and the number of reference folders in the tree does not match the stated count of 400, so the tree is illustrative rather than literal.

An unfinished licence section, and a paper to cite

The licence is the part of this repository that is clearly not finished. The badge on the README reads License-TBD, and the licence section at the bottom asks that the licence applying to the released code and dataset be added before public release, with the additional note that if the dataset contains images inherited from ImageNet, COCO or VOC2012, their original licences and terms should also be respected. The repository licence field is recorded as unknown as well, and the tree holds only a README, an assets directory and a src directory, with no licence file and no code to speak of. The paper itself is published, in IEEE Transactions on Information Forensics and Security in 2024, volume 20, pages 898 to 913, with the DOI given in the README, and the citation block is provided for the dataset and the model:

@article{liu2024stealthiness,
  title   = {Stealthiness Assessment of Adversarial Perturbation: From a Visual Perspective},
  author  = {Liu, Hangcheng and Zhou, Yuan and Yang, Ying and Zhao, Qingchuan and Zhang, Tianwei and Xiang, Tao},
  journal = {IEEE Transactions on Information Forensics and Security},
  volume  = {20},
  pages   = {898--913},
  year    = {2024},
  doi     = {10.1109/TIFS.2024.3520016}
}

For questions there is a GitHub issue tracker and the authors, and the last push to the main branch was on 2026-09-02 with the repository not archived, although the project publishes no GitHub releases, so the dataset and the model are the artefacts rather than the code.

Editorial conclusion

APSD fits a research group working on adversarial attack evaluation that needs a human-aligned measure of perturbation visibility rather than a pixel metric, and it fits anyone who wants to train or evaluate a model that predicts how obvious a perturbation looks to a person. It is the wrong choice for a team that needs a licensed dataset today, since the licence section explicitly asks for the applicable licence to be added before public release, and it is a poor fit if your images came from ImageNet, COCO or VOC2012 and you cannot accept their inherited terms. Before you build on it, read the licence section and the note about inherited image terms, and download the package to confirm that the score file and the per attack folders match the layout described in the README.

Frequently asked questions

Official sources

1. hcliucs/APSD on GitHub
2. Issues
3. README