DoLa decodes a language model by contrasting its own layers

The idea: contrast two layers, not one

DoLa is the official implementation for the paper DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models, published at ICLR 2024, with the arXiv identifier 2309.03883 and the paper hosted on arxiv.

The problem it addresses is stated first: despite their impressive capabilities, large language models are prone to hallucinations, generating content that deviates from facts seen during pretraining.

The proposed approach is described as a simple decoding strategy for reducing hallucinations with pretrained LLMs that does not require conditioning on retrieved external knowledge nor additional fine-tuning. That combination of constraints is the interesting part. Most factuality work in this area either adds a retrieval step or trains the model, and both change the deployment shape of the system. DoLa changes only the decoding.

The mechanism is that the next-token distribution is obtained by contrasting the differences in logits obtained from projecting the later layers versus earlier layers to the vocabulary space. The paper exploits the fact that factual knowledge in a language model has generally been shown to be localized to particular transformer layers. The name is the mechanism: Decoding by contrasting Layers.

The reported result is that DoLa consistently improves truthfulness across multiple choice tasks and open-ended generation, with performance on TruthfulQA improved by 12 to 17 absolute points for the LLaMA family models.

Early exit layers, and three decoding modes from one flag

Everything the method does is controlled by one argument, and the argument doubles as the mode selector.

The --early-exit-layers argument takes a string containing a sequence of layer numbers separated by commas, with no spaces in between. The number of layers specified determines the mode.

With one layer specified, for example -1, you get naive decoding from the final layer output. That is the baseline, the same thing the unmodified model does.

With two layers, for example 16,32, you get DoLa-static decoding, with the second specified layer as the mature layer and the first as the premature layer.

With more than two layers, for example 0,2,4,6,8,10,12,14,32, you get DoLa decoding, with the last specified layer as the mature layer and all the preceding layers as the candidate premature layers. The static mode is then the special case of picking exactly one premature layer out of the candidate set.

So the same flag spans the baseline and two versions of the method, and the only thing that changes is which layers you name. That is an unusually tidy experimental design, because the baseline and the method run through the same code path and differ only in the argument.

A vendored copy of transformers, and why it is there

The setup section gives four install commands, and the first one is the one that tells you the most:

pip install -e transformers-4.28.1
pip install datasets
pip install accelerate
pip install openai # -> only for truthfulqa and gpt4_eval

The transformers install is an editable install from a directory in the repository. The top level of the repository contains a transformers-4.28.1 directory alongside the scripts, so the project carries its own modified copy of the library rather than depending on a published version.

That is the reproducibility mechanism. To implement contrasting layers you need to intervene in the forward pass, project an intermediate hidden state to the vocabulary space yourself, and compare it against the final layer's projection. Doing that through a published API means patching the library, and an editable install of a patched copy is the standard way to keep such a patch reproducible across machines.

The cost is equally clear. A project that carries a forked library at 4.28.1 is pinned to that library's model API, which is a 2023 release. Any current model release is unlikely to work with it without further work, so the natural use of this repository is reproducing the paper's numbers rather than applying the method to a model you are shipping.

The other three installs are the standard evaluation stack: datasets for loading the benchmarks, accelerate for the model loading and device placement, and openai, which the comment scopes to the TruthfulQA and GPT-4 evaluation scripts only.

The evaluation scripts, and the hardware they assume

The repository is a set of evaluation scripts, one per benchmark, and they share a common argument set.

The --model-name argument specifies the model to use, with the note that currently only LLaMA-v1 is supported, and the example is huggyllama/llama-7b. The --data-path is the path to the dataset file or folder, --output-path is where to store the output results, --num-gpus is the number of GPUs to use, and --max_gpu_memory is the maximum GPU memory size in GiB to allocate, with a default of 27 for a 32G V100.

The --num-gpus argument is documented with a mapping: 1, 2, 4 and 8 GPUs for 7B, 13B, 30B and 65B model sizes respectively. That is model parallelism across devices, and the memory argument caps the allocation per device.

The scripts cover FACTOR for multiple choice, where the data file wiki_factor.csv has to be downloaded from the AI21Labs factor repository, TruthfulQA in its multiple choice form, and further scripts visible in the repository for GSM8K, STRQA, open-ended TruthfulQA, a GPT-3 rating, a GPT-4 judge and a conversation file.

The FACTOR baseline commands show the shape, and the DoLa commands show the difference, which is only the early exit layer list:

python factor_eval.py --model-name huggyllama/llama-7b --data-path /path/to/wiki_factor.csv --output-path output-path.json --num-gpus 1

python factor_eval.py --model-name huggyllama/llama-7b --early-exit-layers 0,2,4,6,8,10,12,14,32 --data-path /path/to/wiki_factor.csv --output-path output-path.json --num-gpus 1

And the layer list changes with model size, which is the detail to get right: 0,2,4,6,8,10,12,14,32 for 7B, then 0,2,4,6,8,10,12,14,16,18,40 for 13B, 0,2,4,6,8,10,12,14,16,18,60 for 30B and 0,2,4,6,8,10,12,14,16,18,80 for 65B.

Why the TruthfulQA path needs an OpenAI key

One of the install commands carries a comment that scopes it: pip install openai is only for truthfulqa and gpt4_eval.

That tells you the evaluation is not purely local. The TruthfulQA task has two forms in this repository: a multiple choice form, run by tfqa_mc_eval.py, and a generation form that is judged. The judging is what needs the OpenAI package, and the repository holds a gpt4_judge_eval.py script and a tfqa_gpt3_rating.py script, which together suggest a pipeline where the model generates an answer and then a separate judge model scores it.

That is the standard way TruthfulQA generation is measured, and it comes with a property worth understanding before you use it: the evaluation depends on a second model, so the numbers are not a property of the decoding method alone. The 12 to 17 point improvement quoted in the paper is a fact about the paper's setup, including its judge.

The alternative reading of the multiple choice form is that it needs no external service at all, which is why the setup instructions for the multiple choice path only mention that the data path should be a folder containing TruthfulQA.csv, and that the file will be downloaded automatically if it does not exist.

A research repository, and what it is actually for

This is a paper repository rather than a library, and the repository layout says so plainly. The top level holds dola.py, the DoLa implementation itself, and a set of evaluation scripts: factor_eval.py, tfqa_eval.py, tfqa_mc_eval.py, tfqa_gpt3_rating.py, gpt4_judge_eval.py, gsm8k_eval.py, strqa_eval.py and conversation.py.

There is a dola_evaluation.ipynb notebook, and the readme carries an Open in Colab badge pointing at it, so the Colab route is for running the evaluation without a local setup. There is a figure.png for the method diagram, a requirements.txt that lists datasets, accelerate and openai, and a .gitignore.

The authors are Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass and Pengcheng He, with the affiliations given as the Massachusetts Institute of Technology and Microsoft, and the paper is the ICLR 2024 publication.

The maintenance record is the thing to note. The last push was on 2026-07-12, and the repository publishes no GitHub releases. So the code is still being touched, while the version of the underlying library it builds on stays where it was. For a research artifact that has already been published and reviewed at a conference, that is a normal end state: the numbers are settled, the method is written down, and the code exists so that others can check.

The badge row also has a MIT licence badge, while the repository licence field is recorded as unknown, so the terms are the ones in the MIT badge rather than a classified licence file.

Editorial conclusion

Adopt DoLa if you want to reproduce the paper's measurements, since the vendored transformers copy and the pinned early exit layer lists are what make the published numbers comparable. Do not expect to drop the decoding logic into a current model stack, because the repository pins transformers 4.28.1 and the model argument only supports LLaMA v1. Before you run it, check three things: which early exit layer list matches your model size, since the numbers in the documented commands change with parameter count, how much GPU memory you can allocate, since the default is 27 GiB for a 32G V100, and whether the evaluation you need is one of the scripts in the repository, because TruthfulQA needs an OpenAI key for the GPT-4 judge and the FACTOR data has to be downloaded separately.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. voidism/DoLa on GitHub