Long-form factuality: LongFact, SAFE and the F1@K metric

What the repository is actually for

This is the official code release that accompanies the paper on long-form factuality in large language models, and the README makes no attempt to be a general purpose library. It contains four things. LongFact is a prompt set of 2,280 fact-seeking prompts that require long-form responses, rather than a short answer and a label. SAFE, the Search-Augmented Factuality Evaluator, performs automatic evaluation of model responses in long-form factuality settings. F1@K extends the F1 score to long-form settings using recall from a human-preferred length. And an experimentation pipeline benchmarks OpenAI and Anthropic models using LongFact together with SAFE. The audience is therefore people who measure models rather than people who call them: an evaluation team comparing a new model against a baseline, a researcher who wants to reproduce the published numbers, or a developer who suspects that a short-answer benchmark says little about how a model behaves when it writes a thousand words. The homepage is the arXiv entry for the paper, and the README asks anyone who finds the code useful to cite that paper.

LongFact and the pipeline that produced it

The prompt set lives in the longfact/ folder, and the README in that folder holds the details about the dataset itself. What is easy to miss is that the prompts are not the only artifact. The repository also includes the data generation pipeline that was used to generate LongFact, and the README gives the command that runs it. That matters for anyone who wants to understand how the prompt set was built rather than only to consume it, and it matters even more for anyone who wants to extend or re-filter the set.

python -m data_creation.pipeline

The additional detail lives in the README inside data_creation/, and the top level README consistently points down into the subfolders rather than repeating the content. A second experiment follows the same pattern. The full implementation of SAFE is located in eval/safe/, with its own README explaining how SAFE works, and a separate command evaluates SAFE against FActScore human annotations.

python -m eval.correlation_vs_factscore

That second command is the important one for judging the evaluator itself. SAFE is an automatic judge, and automatic judges are only useful if they track the judgement of people. The README frames this as a correlation experiment against FActScore human annotations, and the further details are in the README inside eval/.

Why F1 needed a long-form variant

F1 is a poor fit for a long answer, and the repository's third component is the reason. F1 combines precision and recall over matched tokens, which works when the reference and the candidate are both short and roughly the same shape. A long-form response to a fact-seeking prompt is a different animal: it may contain the fact somewhere in the middle of a thousand words, it may bury the fact under a lot of correct but irrelevant context, and the reference answer is itself long. Measuring overlap against the whole reference punishes a longer answer even when every claim in it is right, and measuring overlap against a token budget punishes an answer for being thorough. F1@K addresses this by using recall from a human-preferred length, which is a way of saying that the length of the answer a person would have preferred sets the denominator of the recall term, and the F1 score is then computed over that budget rather than over the entire reference. Combined with LongFact and SAFE, the three pieces form a full evaluation: a prompt set that demands long answers, a metric that scores them fairly, and an automatic evaluator that produces the scores without a human in the loop for every run.

Automatic search-augmented judging against human annotation

The difference between SAFE and a human annotated reference is the difference between a metric you can run on every candidate model and one that needs an annotation budget. The README names the human side explicitly: FActScore human annotations, used as the reference for the correlation experiment. The approach taken here is automatic evaluation in long-form factuality settings, and the search-augmented part matters because a long answer makes many claims, and a checker that only sees the answer and the prompt has to decide claim by claim whether the statement is right, including claims the prompt never asked about. A search-augmented evaluator can look up a fact before judging it, which is why the method needs a web search backend and why it is a heavier dependency than a pure string comparison. The consequence for a user is practical. A human-annotated set gives you a small number of trustworthy judgements but is expensive to extend. SAFE gives you judgements at the scale of a full model sweep, and its credibility rests on the correlation experiment rather than on the annotations themselves, so if you are going to rely on the numbers, read the README inside eval/safe/ for how the search and the claim splitting work, and run the correlation command above on your own copy of the annotations.

Setting up the environment and running the pipeline

The README gives a complete setup path. You clone the repository, move into the folder, create a Python 3.10 environment with conda, activate it, and install everything listed in requirements.txt.

git clone https://github.com/google-deepmind/long-form-factuality.git
cd long-form-factuality
conda create --name longfact python=3.10
conda activate longfact
pip install -r requirements.txt

Before the benchmarking pipeline runs, you add your API keys to common/shared_config.py, and the README warns not to publish those keys, which is worth taking seriously because that file sits inside a git repository. The main experimentation pipeline is then run as a module, and the README in main/ holds the details of that pipeline.

python -m main.pipeline

Once you have prompt and response pairs, the evaluation step takes the path to the JSON file holding them through the --result_path argument.

python -m eval.run_eval \
    --result_path=

The repository also has a unit test convention: every file has a corresponding test file with the _test suffix, and the run command for an individual test is written inside the test file itself. All of them run at once with:

python -m unittest discover -s ./ -p "*_test.py"

A pinned requirements file with CUDA wheels in it

requirements.txt is fully pinned, and the pins tell you what world this code was written for. The model clients are old: openai is pinned at 0.27.2, anthropic at 0.16.0, and google-generativeai at 0.3.2, with google-ai-generativelanguage and google-api-core pinned alongside. A long list of nvidia packages is present, including nvidia-cuda-nvrtc-cu12, nvidia-cudnn-cu12, nvidia-nccl-cu12 and the cu12 cublas, cufft, curand, cusolver and cusparse builds, so a clean install pulls a large GPU oriented stack whether or not you plan to use a local model. The rest of the file covers the natural language processing work: nltk, a spaCy small English model installed directly from a wheel URL on the spaCy models release page, networkx, pandas, numpy, matplotlib, and rank-bm25 for the ranking component. There are also development oriented packages such as pyglove and langfun. The practical consequence is that the environment is not a lightweight one, that a version conflict in your own project is likely, and that the pinned client libraries may not match the API shapes you are used to. This is a research code release with a frozen environment, and treating it like a current library is the most common way to waste an afternoon.

Two licences in one repository, and no releases to track

The licence situation is the kind of thing that needs two sentences rather than one. Copyright in the repository is held by DeepMind Technologies Limited, dated 2024. The software is under the Apache License, Version 2.0, and everything else in the repository is under the Creative Commons Attribution 4.0 International License, with both licence texts and a disclaimer of warranty included. GitHub classifies the repository licence as something it cannot categorise, which is why the licence file itself is the authority here. In practice that means code reuse is permissive under Apache 2.0, including inside a commercial product, while the dataset and the paper-adjacent content come with an attribution requirement under CC-BY 4.0. The README also states plainly that this is not an official Google product, which is a reminder that the code is a research artifact rather than a supported service. On maintenance, the repository has no GitHub releases at all, so there is no version history to track, and the last push was on 2026-06-18. The paper is the durable reference for the method, and the repository should be read as the artefact that accompanied it.

Editorial conclusion

Adopt this repository if you are evaluating long answers from a language model and want the exact prompts and the automatic search-augmented judge that the published work used, rather than your own proxy metric. Do not adopt it as a production dependency or a service, because there are no releases, the requirements file pins old client libraries such as openai 0.27.2, and the code expects your own API keys placed in common/shared_config.py. Before you run it, read the README files inside longfact/, data_creation/, eval/, eval/safe/, main/ and common/, since the top level README points to each of them for the details it omits, and confirm that the Python 3.10 environment you create can install the pinned NVIDIA CUDA 12 wheels listed in requirements.txt.

Frequently asked questions

Official sources

1. google-deepmind/long-form-factuality on GitHub
2. Issues
3. Project website
4. README