Atlas: retrieval-augmented language models, and a repository frozen in place

The state of the repository, stated up front

The second line of the README is the first thing a reader should take in: REPO NO LONGER MAINTAINED, RESEARCH CODE PROVIDED AS IT IS.

That notice is unambiguous and it should shape everything else. This is a paper's code, frozen at the point where the paper was published, and it is not going to receive dependency updates, bug fixes or compatibility work. The last push to the repository was on 2026-07-02, the project is not archived, and the repository publishes no GitHub releases.

What is in it, then, is the pre-trained models, corpora, indices and code for pre-training, finetuning, retrieving and evaluating for the paper Atlas: Few-shot Learning with Retrieval Augmented Language Models, and is available as an arXiv paper.

The primary language is Python, the licence field is recorded as a custom licence GitHub cannot classify, and the default branch is main.

For a reader deciding whether to spend time here, the value is documentary rather than operational. The code is a complete, worked example of a training stack that is now well understood but was not when it was written, and the README is a serious piece of technical documentation. Whether that is worth your afternoon depends on whether you are trying to reproduce the paper or you are trying to understand how retrieval-augmented training was actually put together, including the parts that the paper leaves out.

One thing that has not aged well is the dependency stack, and that is a fact to plan around rather than discover. The README lists python 3 tested with 3.8, fairscale at 0.4.6, transformers at 4.18.0, numpy at 1.22.4 and faiss at 1.7.2, and the install instructions pin pytorch 1.11.0 with cudatoolkit 11.3. The requirements file is short: fairscale 0.4.6, transformers 4.18.0, rouge 1.0.1, tensorboard and wget. That is a 2022 stack, and transformers 4.18 in particular predates several changes in how that library is structured.

The paper in three sentences

The README's own summary is a good description of the contribution.

The authors jointly pretrain a retrieval-augmented seq2seq language model, comprised of a passage-based dense retriever and an encoder-decoder language model. They perform evaluations on a wide range of tasks, including MMLU, KILT and NaturalQuestions, and study the impact of the content of the document index, showing that it can easily be updated.

The headline results are the interesting part for anyone assessing the work. Atlas reaches over 45 percent accuracy on Natural Questions using only 64 examples when supplied with a Wikipedia index from 2018, outperforming a 540B parameter model by 6 percent despite having 50 times fewer parameters. And Atlas also works very well when finetuned on larger datasets: finetuned on the full Natural Questions data, it sets a new state of the art of 64 percent, 8 points higher than the current state of the art.

The second result is the one that most shaped the field, because it demonstrated that retrieval could substitute for parameters. The third is the one that makes the work useful rather than merely striking, since it shows the approach does not stop working when you have data to train on.

The study of the index content is the third contribution and the most relevant to this repository specifically, because the index is the part you can change without retraining. If the document index can be updated easily, then a model that was trained against one corpus can be pointed at a different one, and the repository is where that machinery lives.

The blog post is linked for a quick overview of the project and how to run the code with torchrun, with the slurm free option noted, which matters because a multi-GPU training run of this kind is otherwise hard to run outside a cluster.

What the codebase supports

The feature list is long and specific, and it is the best catalogue of what the code actually does.

Training large fusion-in-decoder seq2seq models, tested up to 11B parameters. Distilling relevance signals from fusion-in-decoder models into dense retrieval models using a variety of different distillation approaches. Performing end-to-end retrieval-augmented training over a user-supplied corpus of passages, tested with up to 400 million passages and about 40 billion words, with retrieval-in-the-training-loop.

Then a long list of tasks: Masked-Language modelling, prefix-language modelling, Wikipedia section generation, Open-Domain Question Answering, Multiple Choice Question Answering, fact checking and KILT, with the note that arbitrary seq2seq tasks can also be supported.

And three capabilities that are about the retrieval machinery rather than the training: a fast, parallel distributed GPU-based exact and approximate maximum inner product search for dense vector retrieval, support for fast in-place index refreshes, and various memory optimizations and methods for maintaining fast and accurate retrieval while training retrievers in-the-loop. The list closes by pointing to the command line arguments or the README for additional features.

The 400 million passage figure is the one that tells you the scale of the engineering. That is not a laptop-scale retrieval problem, which is why the distributed GPU index described later matters so much.

The architectures are fixed and the README is clear about it. The project supports T5 architectures for the encoder-decoder language model and Contriever architectures for the retriever, with support for other architectures not currently planned but pull requests welcome. Atlas models are comprised of a Contriever retriever and a fusion-in-decoder architecture using T5. The README links to the Contriever and FiD repositories for background, and notes that all required functionality has been reimplemented in this codebase, so you are not expected to install either as a dependency.

Installation, and the code layout

The installation uses conda and pins the CUDA toolkit, which is the practical reality of running this code:

git clone https://github.com/facebookresearch/atlas.git
cd atlas
conda create --name atlas-env python=3.8
conda activate atlas-env
conda install pytorch==1.11.0 cudatoolkit=11.3 -c pytorch
conda install -c pytorch faiss-gpu=1.7.2 cudatoolkit=11.3
pip install -r requirements.txt

The pinned versions matter here. The GPU-aware FAISS build and the cudatoolkit 11.3 pin mean this is a CUDA 11.3 environment, and anyone on a newer driver stack will need to reconcile that.

The repository listing shows a small, task-oriented structure. There is train.py and finetune_qa.py at the root, evaluate.py for evaluation, and then src/ for the library code, preprocessing/ for data preparation, example_scripts/ and evaluation_scripts/ for the per-task and per-dataset scripts, and requirements.txt.

The separation of train.py, finetune_qa.py and evaluate.py is worth noting because it maps onto the three phases the README describes: pretraining, finetuning and evaluation. The finetune_qa.py script is specific to question answering, which is the task the headline results come from.

The requirements file is short:

fairscale==0.4.6
transformers==4.18.0
rouge==1.0.1
tensorboard
wget

Tensorboard and wget are there for training and data fetching, rouge is the evaluation metric for generated text, and fairscale and transformers are the two libraries doing the heavy lifting. Fairscale at 0.4.6 is what provides the distributed training and the memory optimisations, and it is worth remembering that fairscale has been superseded as a project, which is one of the reasons this environment is hard to reproduce on a current stack.

The retrieval machinery, and the index refresh strategies

This is the most interesting part of the README for anyone thinking about retrieval systems, and the document is unusually good at explaining what happens under the hood.

The biggest difference to most standard NLP training codebases is that Atlas performs retrieval on-the-fly, and can refresh its retrieval embeddings index in-place. This is achieved using a custom-designed distributed GPU index, which automatically handles fast and scale-able retrieval.

The sequence of operations is described step by step. When launching a training or evaluation run, the codebase first loads pretrained models, then each GPU worker loads a shard of the passages it will retrieve from, so with N GPUs each loads a shard of 1/N passages. Each worker then embeds its shard of the passages using the retriever embedder and keeps the passage embedding shard in GPU memory, and optionally builds a FAISS index. At this point the passage and embedding shards, referred to as the index, can be optionally saved to disk to avoid recomputing indices for every run. Retrieval is then performed in parallel, with each GPU worker performing an exact maximum inner product search for all the queries for its shard.

The note at the end is the reassuring part: all of the above is handled automatically by the codebase, so users should not need to know or worry too much about it.

The table of contents lists three separate strategies for dealing with stale indices, which is the term for the problem of a document index that no longer matches the corpus. The Index Refresh strategy rebuilds the index. The Over-Retrieve with Reranking strategy fetches more candidates than needed and reranks them, which is the standard trick when the index is a lossy approximation. And Query-Side Finetuning adapts the model to the queries rather than the index.

There is also a Retrieve-only mode and support for using pre-retrieved or cached passages, which is what you want for evaluation, where retrieval happens once and the model runs many times over the same candidates.

The Flat versus Faiss comparison and the index saving and loading sections fill in the rest, and the whole structure is a genuine reference for how to build retrieval into a training loop, which remains a hard thing to do well.

How to evaluate it, and what to be careful about

There are four things to consider, and the first is whether you should be running this code at all.

The repository says it is not maintained and provides research code as it is. If your goal is to build a retrieval-augmented system today, this is the wrong tool, because the field has moved and there are maintained implementations of the same ideas. If your goal is to reproduce the paper, or to study how a 2022 implementation handled a problem that is still hard, then it is exactly right, and the documentation quality is a gift.

Second, the environment cost. Python 3.8, torch 1.11.0, cudatoolkit 11.3, transformers 4.18.0, fairscale 0.4.6 and a GPU-enabled FAISS. Building that in 2026 means an old CUDA stack, and the dependency on fairscale is the part that most often needs work, since that project has been wound down. The requirements file is short, which is good, but short does not mean easy.

Third, the scale of the hardware. The feature list mentions training tested up to 11B parameters and retrieval over 400 million passages, which is a cluster-scale operation. The blog post points at torchrun with a slurm free option, which tells you the intended entry point, but a single machine will not get you to the headline configurations.

Fourth, the licensing, which is described as split. The table of contents has separate Code License and Data License entries, and the repository licence field is a custom one GitHub cannot classify. The code and the pre-trained models, corpora and indices are therefore under different terms, and for a project whose main artefact is downloadable data and checkpoints, the data licence is the one to read carefully. The code licence and data licence are both spelled out in the README's own section on it.

What you get for the effort is a complete, well-explained snapshot of a significant piece of work. The retrieval-in-the-training-loop design, the distributed GPU index, the three strategies for stale indices, and the distillation of relevance signals from a fusion-in-decoder model into a dense retriever are all things that were hard to get right in 2022 and are still hard now. Reading how they did it is worth a few hours even if you never run a line of it.

Editorial conclusion

Adopt the Atlas repository as a reference if you are studying retrieval-augmented generation, if you want to see how a 2022 implementation put retrieval inside a training loop, or if you are reproducing that paper. Do not adopt it as a base for new work, because the README says outright that it is no longer maintained and the environment is a 2022 stack with torch 1.11.0, cudatoolkit 11.3, transformers 4.18.0 and fairscale 0.4.6. Before you commit, read the code licence and the data licence separately, since the pre-trained models, corpora and indices are the main artefacts here and they carry their own terms, plan for a GPU cluster rather than a single machine given the 11B parameter and 400 million passage configurations that were tested, and use the blog post's torchrun entry point with the slurm free option if you have no scheduler.

Frequently asked questions

Official sources

1. facebookresearch/atlas on GitHub
2. Issues
3. README