ArGue: attribute-guided prompt tuning for CLIP under distribution shift

The problem is background correlation, not raw prompt quality

ArGue is the official implementation of the paper 'ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models', accepted at CVPR 2024, and the arXiv preprint is 2311.16494 with a project page alongside it.

The stated motivation is specific. ArGue addresses distribution shift limitations in soft prompt tuning by aligning vision-language models with LLM-generated visual attributes, introducing attribute sampling and negative prompting to suppress spurious correlations.

That last phrase is the crux. Soft prompt tuning, in the CoOp formulation this work builds on, learns a set of continuous tokens that are prepended to the class name to form the text prompt. The tokens are trained on a base set of classes, and when you then ask the model about classes it has never seen, or about images whose distribution differs from the training set, the learned tokens can latch onto whatever correlates with the training data rather than onto the concept itself. In a dataset of sketches, the prompt may key on colour because every training photograph had it.

The ArGue approach is to make the prompt say more about the visual content of the class. The soft tokens are still there, but the class name is followed by a visual attribute, and the model is additionally trained to be unconfident when the prompt refers to the background. The two ideas work together: attributes give the model something class-specific to key on, and negative prompting removes the shortcut that was previously doing the work.

The work builds on CoOp and CoCoOp, uses LASP as the previous state of the art it compares against, uses Dassl as the training framework and CLIP as the backbone, and evaluates on 11 benchmark datasets following the standard CoOp and CoCoOp splits. Attributes are generated using the GPT-3 API.

Three components: attributes, sampling and the negative prompt

The method is three components built on top of soft prompt tuning, and the README describes the mechanics of each.

Attribute-guided prompting changes the shape of the prompt. Instead of [soft tokens] + [class name], it concatenates a visual attribute appended after the class name, giving [soft tokens] + [class name] + [attribute]. Attributes are generated by GPT-3 using multiple in-context-learning templates, J=15 per class. The final prediction averages logits over all attributes per class.

Attribute sampling handles the fact that an LLM will produce a lot of useless attributes. The attribute pool is clustered in CLIP's text feature space, and within each cluster the attribute with the highest similarity to training images is selected, which filters out non-visual attributes such as edible and domain-irrelevant ones such as colour attributes for grayscale sketch datasets. Only N=3 representative attributes per class are kept for training. The README puts a number on the effect: retaining 20 percent of attributes while improving accuracy.

Negative prompting, called ArGue-N, constructs a negative prompt of [soft tokens] + [background of] + [class name]. Training enforces a uniform predictive distribution under this prompt by maximizing entropy, which suppresses background spurious correlations without requiring per-class manual annotation.

The illustration is worth following. Given a background-activating prompt such as the background of a cat, vanilla CLIP produces biased predictions. ArGue-N is trained to output a uniform distribution under such prompts, forcing the model to focus on class-specific attributes rather than background cues.

The component analysis in the README shows each piece contributes. Introducing attributes alone improves CoOp by +7.64 percent on average, attribute sampling further adds +0.34 percent while cutting 80 percent of the computation, and negative prompting brings an additional +0.40 percent with no extra manual annotation.

What the ArGue-N training objective actually optimises

The negative prompting component is the one that is worth understanding in detail, because it changes the training objective rather than the prompt.

The construction is simple. For a class, a negative prompt is built as [soft tokens] + [background of] + [class name]. In training, the model is required to produce a uniform predictive distribution under that prompt, which the README describes as maximizing entropy. The stated effect is that it suppresses background spurious correlations without requiring per-class manual annotation.

The mechanism is a form of negative learning. Rather than teaching the model what a cat looks like by showing it more cats, you teach it that the background of a cat carries no information, by making the model maximally uncertain when asked about it. Any capacity the model would otherwise spend on background cues is spent on nothing, which leaves more of it for the class-specific signal. That is why the approach needs no annotation: the negative prompts are generated from the class names by prefixing, so the number of training signals scales with the number of classes and not with the number of labelled backgrounds.

The entropy maximisation is a strong constraint, and it is worth being clear that it is only applied under the negative prompt, not under the normal one. The ordinary prompt still gets the standard cross-entropy objective, so the two prompts pull in different directions during the same training run: the positive prompt says be confident about the class, the negative prompt says be unconfident about the background of the class.

That pairing is what makes the reported numbers coherent. The positive prompt is strengthened by the attribute appended after the class name, which gives it something concrete to key on, and the negative prompt removes the competing shortcut. The ablation figures in the project description are consistent with that reading: attributes alone give the largest single gain of +7.64 percent on average over CoOp, and the other two components add +0.34 and +0.40 percent on top, with attribute sampling also cutting 80 percent of the computation by keeping only N=3 attributes per class rather than all 15.

Running the pipeline end to end

The project description walks through the whole sequence, and it is a four stage pipeline rather than a single training command.

The requirements are Python 3.7 or newer, a CUDA-compatible GPU, and the torch, dassl and clip packages. The install is:

git clone https://github.com/xytian1008/ArGue.git
cd ArGue
pip install -r requirements.txt

Dassl is installed separately following its own instructions. The requirements file is small, ftfy, regex, tqdm, gdown, a direct git URL for Dassl.pytorch and openai.

Datasets are downloaded following the CoOp instructions, and the supported list is long: ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, UCF101, DTD, EuroSAT, ImageNetV2, ImageNet-Sketch, ImageNet-A and ImageNet-R.

Attribute generation comes next, and it is the stage that costs money or at least an API key:

python generate_descriptors.py

That calls GPT-3 with the text-davinci-003 model using multiple in-context-learning templates to produce J=15 attributes per class.

Attribute sampling then selects representative attributes from the pool using CLIP-guided clustering:

bash scripts/ARGUE/select_attr.sh

which runs select_attr.py, clustering attribute text features and picking the attribute per cluster most similar to training images, retaining N=3 per class.

Training and evaluation are shell scripts taking a dataset and a seed, with a base2new variant and an xd variant for each of train and test.

The reported numbers, and the one claim that stands out

The results section has three parts, and one of them is a first-of-its-kind claim worth reading closely.

On novel class prediction, ArGue-N outperforms LASP by +1.70 percent on average harmonic mean across 11 datasets. The stronger claim is the second half: it is the first prompt tuning method to surpass zero-shot CLIP on novel class accuracy in 10 out of 11 benchmarks. That is a meaningful framing because beating zero-shot CLIP is a threshold many prompt tuning methods never cross, and beating it on most datasets is unusual.

On out-of-distribution generalisation, ArGue-N consistently outperforms all baselines across all four OOD variants of ImageNet, with notable gains on ImageNet-A at +1.47 percent and ImageNet-Sketch at +0.50 percent. Those two are the interesting ones, because ImageNet-Sketch is exactly the case the attribute sampling is designed for, where a colour-correlated prompt would fail.

The repository layout matches the pipeline. There is generate_descriptors.py, select_attr.py and descriptor_strings.py at the top level, alongside argue_train.py and the comparison trainers coop_train.py and lasp_train.py, so the baselines are in the same tree. There are scripts/ for the shell entry points, configs/, datasets/, trainers/, a clip/ directory, a fast_pytorch_kmeans/ directory for the clustering, an assets/ directory with the figures used in the README, and a docs/ directory.

Two things to keep in mind when reading the numbers. The attribute generation depends on the text-davinci-003 model, which is a legacy completion endpoint, so reproducing the exact attribute pool requires access to that model rather than a current one, and any change in the model changes the pool, which changes the sampling and therefore the results. And the evaluation is on the standard CoOp and CoCoOp splits, which is what makes the comparison to LASP fair, but also means the numbers describe those benchmarks rather than general behaviour.

Editorial conclusion

ArGue suits a researcher reproducing the CVPR 2024 results, since the pipeline scripts cover attribute generation, sampling, training and evaluation for both the base-to-new and out-of-distribution protocols, and the ablation numbers in the README tell you which component contributes what. It does not suit you if you want a maintained library or a new backbone, because the code is pinned to Dassl, CLIP and the CoOp data splits, and the attribute generation step calls the GPT-3 text-davinci-003 model, which is a legacy endpoint. Before you run it, check the requirements file, which installs Dassl directly from a git URL, and confirm the datasets you need are prepared following the CoOp instructions.

Frequently asked questions

Official sources

1. Issues
2. README
3. xytian1008/ArGue on GitHub