# AutoPrompt hands every prompt in a run the same trigger tokens

> UCI NLP's masked language model prompt search, read closely: one trigger set shared by all prompts, a label-map flag that changes shape between its two stages, four setup paths that disagree with each other, and an evaluation step that lives in a fork of LAMA.

**ucinlp/autoprompt** — AutoPrompt: Automatic Prompt Construction for Masked Language Models.

- Repository: https://github.com/ucinlp/autoprompt
- Stars: 642 · Forks: 86
- Language: Python
- License: Apache-2.0
- Published: 2026-09-17 · Updated: 2026-09-17 · Language: en
- Canonical page: https://hysenlabs.com/projects/ucinlp-autoprompt

## One trigger set is shared by every prompt in a run

The template is the easiest place to read the method. A prompt is described as mapping the original input and trigger tokens onto a shape like `[CLS] {sub_label} [T] [T] [T] [P]. [SEP]`, where `{sub_label}` stands for the subject of a (subject, relation, object) triplet and `[P]` marks the slot a special `[MASK]` token fills in. The part worth pausing on is `[T]`. The page states that each trigger token belongs to the set of trigger tokens that are shared across all prompts, so one search yields a single trigger set that every example in the task reuses, rather than a fresh trigger sequence per input. Which special tokens frame the template depends on the backbone: BERT keeps `[CLS]` and `[SEP]` stuck to each end, RoBERTa uses `<s>` and `</s>` instead. The template string is therefore not portable between the two models, and switching `--model-name` means rewriting the frame as well.

## The same `--label-map` flag carries word lists in one stage and integers in the other

Two modules do two jobs: `autoprompt.create_trigger` searches the trigger tokens, and `autoprompt.label_search` selects the label tokens. The flag that ties classes to tokens changes shape between them. During trigger creation, `--label-map` holds lists of vocabulary items, and the sentiment example gives class `0` five entries and class `1` five more.

```
python -m autoprompt.create_trigger \
    --train glue_data/SST-2/train.tsv \
    --dev glue_data/SST-2/dev.tsv \
    --template '<s> {sentence} [T] [T] [T] [P] . </s>' \
    --label-map '{"0": ["Ġworse", "Ġincompetence", "ĠWorse", "Ġblamed", "Ġsucked"], "1": ["ĠCris", "Ġmarvelous", "Ġphilanthrop", "Ġvisionary", "Ġwonderful"]}' \
    --num-cand 100 \
    --accumulation-steps 30 \
    --bsz 24 \
    --eval-size 48 \
    --iters 180 \
    --model-name roberta-large
```

Class `0` lists `Ġworse` and `ĠWorse`, the same word in two cases, which is only possible because the entries are raw vocabulary strings rather than normalized words. The leading `Ġ` marks a word start in the byte level vocabulary, and the inference example writes the same marker as the escape `\u0120` in its ENTAILMENT and CONTRADICTION lists while leaving `abet`, `icy`, `oder` and `agna` unmarked. Under label search the lists disappear: sentiment maps its two classes to plain integers and inference maps `ENTAILMENT`, `CONTRADICTION` and `NEUTRAL` to 0, 1 and 2. Nothing on the page says whether those integers are class ids handed to the model or a different form of the same word search.

## Label search points at different files than trigger creation does

One page spells the SST-2 training file two ways. Trigger creation reads `glue_data/SST-2/train.tsv` with a dev split beside it, while label search for the same task reads `../data/SST-2/train.tsv`. Inference drifts further: the create_trigger example names bare `SICK_TRAIN_ALL_S.tsv` files, and the label_search example walks down into `../data/SICK-E-balanced/3-balance/`.

```
python -m autoprompt.label_search --train ../data/SST-2/train.tsv --template '[CLS] {sentence} [T] [T] [T] [P]. [SEP]' --label-map '{"0": 0, "1": 1}' --iters 50 --model-name 'bert-base-cased'
```

Nothing on the page builds any of those trees. The datasets arrive as a link to a shared Google Drive folder, and the surrounding text describes dataset variants rather than a local layout, so the four relative paths for two datasets have to be reconciled against whatever the download actually unpacks. The sentiment label search also runs on `--train` alone, with no dev file, where the sentiment trigger run passes both.

## The Compose recipe runs the sentiment job for 20 iterations instead of 180

compose.yml at the repository root defines exactly one service, `sentiment-trigger-creation`, built from the Dockerfile with the working directory bind mounted at `/workspace`. Its command repeats the sentiment invocation from the usage section flag for flag, with two differences: it calls `python3` where the page calls `python`, and it sets `--iters 20` where the page's own example sets `--iters 180`. The GPU is requested under `deploy.resources.reservations.devices` with `driver: nvidia`, `count: 1` and `capabilities: [gpu]`, which is the reservation block Compose uses for deployments rather than a device request placed in the service itself. Because the bind mount points the container back at the checkout, anything a run writes lands in the host tree. Sentiment is also the only task with a container recipe; inference, fact retrieval and relation extraction have none anywhere in the repository, although the image itself is general enough to run any of them.

## `$path` is left unexpanded in the two commands that feed LAMA

Fact retrieval and relation extraction are the two commands whose output goes to an external evaluator, and both are printed with a shell variable still in place: `--train $path/train.jsonl` and `--dev $path/dev.jsonl`. No part of the usage section exports `path` or shows a value for it, so the literal command fails until the reader supplies one. These two also carry flags the earlier tasks never use. `--label-field 'obj_label'` names the field inside those jsonl files that holds the object, `--tokenize-labels` asks for the labels to be split into tokens, and `--print-lama` appears in both and nowhere else.

```
python -m autoprompt.create_trigger \
    --train $path/train.jsonl \
    --dev $path/dev.jsonl \
    --template '<s> {sub_label} [T] [T] [T] [P] . </s>' \
    --num-cand 10 \
    --accumulation-steps 1 \
    --model-name roberta-large \
    --bsz 56 \
    --eval-size 56 \
    --iters 1000 \
    --label-field 'obj_label' \
    --tokenize-labels \
    --filter \
    --print-lama
```

The relation extraction variant swaps in `'[CLS] {context} [SEP] {sub_label} [T] [T] [T] [P] . [SEP]'`, so the subject slot sits after the context and before the mask, and adds `--use-ctx`. Across the four commands the search budget ranges from 50 iterations in label search to 1000 in fact retrieval, `--num-cand` sits at 100 for sentiment against 10 for the two LAMA tasks, `--accumulation-steps` at 30 against 1, and batch sizes at 24, 120, 56 and 32. The inference example passes none of these numbers at all and runs on defaults the page never prints.

## Evaluation is configured by commenting code inside a LAMA fork

Scoring fact retrieval and relation extraction happens outside this repository, in a fork of LAMA, which is meant to be cloned and set up separately because its dependencies differ. The AutoPrompt data folder is copied into LAMA's `data` directory, or `data_path_pre` in `scripts/run_experiments.py` is pointed at a custom location. Configuration is done by editing that script: every setting carries a `[CONFIGURABLE]` comment, the language model's block you want is uncommented while the other is commented out, and `common_vocab_filename` must be `common_vocab_cased_rob.txt` instead of the usual `common_vocab_cased.txt` when both BERT and RoBERTa are scored. `use_ctx` is set to `True` for relation extraction, `synthetic` to `True` for perturbed sentence evaluation, and generated prompts are pasted into LAMA's `data/relations.jsonl` by hand. RoBERTa is unpacked by hand too:

```
mkdir pre-trained_language_models/roberta
cd pre-trained_language_models/roberta
curl -O https://dl.fbaipublicfiles.com/fairseq/models/roberta.large.tar.gz
tar -xvzf roberta.large.tar.gz
```

The list of switches stops in the middle, right after it names the `get_TREx_parameters` function and `data_path_pre` again, and the table of contents promises a Citation section that follows it. Anyone who needs the remaining settings is left reading the fork's script instead.

## setup.py declares no dependencies and still numbers itself 0.0.1

Packaging and installation describe different projects. `setup.py` names the package `autoprompt`, sets `version='0.0.1'`, passes an empty `install_requires=[]`, and offers pytest as the only extra under `test`. All the real dependencies live in requirements.txt, pinned to exact versions: streamlit 0.79.0, torch 1.4.0, transformers 5.5.0, spacy 2.2.0, pandas 1.2.1, numpy 1.17.2, matplotlib 3.1.1, tqdm, termcolor, colorama and pytest. Someone who installs the package gets none of that list. setup.py also opens `README.md` through a relative path while it is being imported, so a build has to happen from the repository root. The container is pinned just as tightly: `nvcr.io/nvidia/cuda:10.1-devel-ubuntu18.04`, pyenv installing 3.7.17, `pip install -r requirements.txt`, `python -m spacy download en`, and an entrypoint of `/bin/bash`, while the setup section asks for `python=3.7` in conda. Dates drift too: the single release tag is v1.1, marked EMNLP 2020 and dated 21 November 2020, the package metadata has never left 0.0.1, and the last commit on the default branch is dated 2 September 2026.

## The repository ships an app, a prompts directory and a devcontainer the page never mentions

The top level holds `.circleci/`, `.devcontainer/`, `app/`, `autoprompt/`, `prompts/`, `scripts/`, `tests/`, `pytest.ini`, `index.html`, `compose.yml` and `Dockerfile`. The dependency list explains one of them: streamlit 0.79.0 is pinned, and an `app/` directory next to a root `index.html` has the shape of a small web front end for the search. None of the five sections in the table of contents mentions it. The `prompts/` directory is the larger gap, because scoring requires generated prompts to be moved into LAMA's `data/relations.jsonl` and `--print-lama` shows up only in the two commands that feed that evaluator. How the checked in directory relates to the flag is left unstated. `tests/` and `pytest.ini` sit next to the `test` extra in setup.py, so the test story is a directory, a config file and a pin, with no command for running it anywhere on the page.

## Conclusion

AutoPrompt is a set of research scripts, not a packaged tool: the search commands read clearly, the four task recipes disagree in ways the documentation never reconciles, and pip metadata, the Compose file and the LAMA fork each hold a different answer to what running this project means. Before trusting any of it, settle the dataset layout the four commands disagree about, treat setup.py as unrelated to requirements.txt, and plan to read the fork's run_experiments.py for the settings its own page leaves out. It suits someone who already has the EMNLP 2020 environment on hand; it does not suit anyone who expects `pip install` to produce a working prompt search.

## FAQ

### What is AutoPrompt?

A method from UCI NLP that builds prompts for masked language models by gradient-guided search over trigger tokens. The page covers four tasks: sentiment analysis, natural language inference, fact retrieval and relation extraction, with the paper and a website linked from the top of the description.

### What does prompt mean in AutoPrompt?

The page never defines the word in prose. It shows a prompt concretely as a template with a mask slot, such as `[CLS] {sub_label} [T] [T] [T] [P]. [SEP]`, where `[P]` marks the position the masked language model fills and `[T]` marks one of the trigger tokens shared across all prompts.

### What is automatic prompt optimization, and is that what AutoPrompt does?

The documented search is gradient-guided over trigger tokens, driven by two modules, `autoprompt.create_trigger` for the trigger tokens and `autoprompt.label_search` for the label tokens. The phrase automatic prompt optimization does not appear in the setup or usage sections, and the iteration count is a flag, `--iters`, rather than a separate optimizer.

### Can ChatGPT generate prompts the way AutoPrompt does?

Nothing in this repository involves ChatGPT. The requirements list streamlit, tqdm, pandas, numpy, torch, transformers, spacy, termcolor, colorama, matplotlib and pytest, the models named in the commands are roberta-large and bert-base-cased, and every template reserves a `[MASK]` slot for fill in the blank scoring.

### Does AutoPrompt's setup line up with what its evaluation needs?

Not on the page. Setup pins `python=3.7` in conda and 3.7.17 in the image, requirements pins torch 1.4.0 and transformers 5.5.0, and evaluation expects a separate conda environment for a LAMA fork plus a RoBERTa large tarball unpacked under `pre-trained_language_models/roberta`. The page records no version combination behind its own results.

## Sources

- [Issues](https://github.com/ucinlp/autoprompt/issues)
- [License: Apache-2.0](https://github.com/ucinlp/autoprompt/blob/master/LICENSE)
- [README](https://github.com/ucinlp/autoprompt/blob/master/README.md)
- [Releases](https://github.com/ucinlp/autoprompt/releases)
- [ucinlp/autoprompt on GitHub](https://github.com/ucinlp/autoprompt)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/ucinlp-autoprompt