LongMemEval asks a chat assistant to remember 500 questions worth of conversation, then tests what came back

Five memory abilities, and the question types that map onto them

The benchmark releases 500 high quality questions aimed at five core long-term memory abilities: Information Extraction, Multi-Session Reasoning, Knowledge Updates, Temporal Reasoning and Abstention. Those five names are the design, and the dataset encodes them in a field rather than only in a paper. Each instance carries a question_type, which is one of single-session-user, single-session-assistant, single-session-preference, temporal-reasoning, knowledge-update and multi-session. Abstention is encoded differently: if the question_id ends with _abs, then the question is an abstention question. That encoding is worth noting because abstention is the case where the correct behaviour is to say you do not know, and a benchmark that folds it into an identifier rather than a type still gets it counted. The types are the units a system owner would look at when a score drops, since a system that handles single-session extraction well and multi-session reasoning badly is a different problem from one that fails at knowledge updates.

Each question ships with a compiled, timestamped history to memorise

The mechanism behind the benchmark is an attribute-controlled pipeline that compiles a coherent, extensible and timestamped chat history for each question, inspired by the needle-in-a-haystack test. The crucial property is that the history is not incidental background. LongMemEval requires chat systems to parse the dynamic interactions online for memorization, and the answer comes only after all the interaction sessions are over. So the test is not retrieval from a static document store, it is memory formation over a stream of sessions and later recall from that memory. The per-instance fields describe the pieces. haystack_session_ids lists the ids of the history sessions, haystack_dates lists their timestamps, and haystack_sessions holds the actual user-assistant chat history, where each session is a list of turns and each turn is a dict with role set to user or assistant and content holding the message. Turns that contain the required evidence carry an extra field, has_answer set to true, and that label is what turn-level memory recall accuracy is scored against.

Three dataset sizes, and the token cost of the small one

The data package contains three files, and they differ in how much history you have to absorb. longmemeval_s.json is LongMemEval_S, and concatenating all its chat history roughly consumes 115k tokens for Llama 3, which works out to about 40 history sessions. longmemeval_m.json is LongMemEval_M, where each chat history contains roughly 500 sessions, and it is the variant you scale up when you want the benchmark to get harder. longmemeval_oracle.json is the version with oracle retrieval, where only the evidence sessions are included in the history. Each of the three holds 500 evaluation instances. The oracle file is the one that isolates memory from retrieval, since the evidence is guaranteed to be present, and it is also the file the documented evaluation command points at. The two larger files are the ones where the model has to find the relevant sessions itself, and the session id lists in them are sorted by timestamp, while the oracle file's are not sorted, since there is nothing to sort.

Scoring your own outputs needs a jsonl file and one command

The intended workflow avoids asking you to run someone else's memory system. You feed the timestamped history to your own chat system, collect the output, and save the results in a jsonl format where each line has two fields, question_id and hypothesis. Then the evaluation script does the rest, taking the model name, your hypothesis file and the dataset path.

export OPENAI_API_KEY=YOUR_API_KEY
export OPENAI_ORGANIZATION=YOUR_ORGANIZATION     # may be omitted if your key belongs to only one organization
cd src/evaluation
python3 evaluate_qa.py gpt-4o your_hypothesis_file ../../data/longmemeval_oracle.json

The script writes logs to a file named after your hypothesis file with a .log suffix, and every line in it gains a new field called autoeval_label. The script already reports the averaged scores, but you can aggregate from the log separately with print_qa_metrics.py, again passing the model name, the log file and the dataset path. The .log file and the autoeval_label field are what make per-question error analysis possible, since a single average tells you nothing about which memory ability failed.

Two environment options, and the difference is whether you run the memory systems

The setup is split into a lite path and a full path, and the choice depends on what you want to run rather than on how much memory you have. If you only need to calculate the metrics on outputs produced by your own system, the minimal requirement set is enough, and it allows you to run src/evaluation/evaluate_qa.py.

conda create -n longmemeval-lite python=3.9
conda activate longmemeval-lite
pip install -r requirements-lite.txt

If you also want to run the memory systems introduced in the paper, the full environment applies.

conda create -n longmemeval python=3.9
conda activate longmemeval
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements-full.txt

The full path pins torch 2.3.1, torchvision 0.18.1 and torchaudio 2.3.1 from the CUDA 12.1 wheel index, and the authors state they tested this environment on a Linux machine with CUDA 12.1. If you are on a different platform, the documentation warns you may need to modify the requirements. Both paths use Python 3.9, which is worth noting if your own tooling has moved on, since the environment version is pinned by the project rather than derived from your machine. The repository keeps two requirement files at the root, requirements-lite.txt and requirements-full.txt, which is what makes the two paths possible.

The dataset lives on Hugging Face, and the history was revised after release

The data is not in the repository. It is officially released on Hugging Face at the xiaowu0162/longmemeval-cleaned dataset, and the setup instructions tell you to download and uncompress it into the data folder.

mkdir -p data/
cd data/
wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json
wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json
wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_m_cleaned.json
cd ..

The file names in those commands are the cleaned variants, which is a deliberate choice on the reader's part. The project news records that in 2025/09 the authors further cleaned up the history sessions to prevent interference on answer correctness, with the updated benchmark and a change log published separately. Interference here means a session that accidentally contains the answer or a giveaway, which would let a system score well without having remembered anything. The download step and the news entry together mean the version you fetch determines what you measure, and the change log is the place to check what changed between the original release of 2024/10 and the cleaned version. The paper was accepted at ICLR 2025.

You can compile histories of arbitrary length yourself

The last piece of the tooling is a custom history compiler, and it is the part that lets you push past the shipped difficulty. LongMemEval supports compiling a chat history of arbitrary length for a question instance, so you can scale up the difficulty over LongMemEval_M. The corpus for that is a separate compressed download, and the documentation has a section for downloading it. This matters because a fixed dataset eventually becomes a ceiling. Systems start to saturate a benchmark, and at that point the interesting question becomes whether the numbers still say anything. Extending the history length changes the scale rather than the question distribution, so it tests whether a memory system degrades gracefully or falls over, and the shipped M variant at roughly 500 sessions per history is the starting point for that. The authors also point to a separate LongMemEval-V2 project, described as long-term memory in agentic context, which is the natural place to look if your interest is agent memory rather than chat assistant memory.

Editorial conclusion

LongMemEval fits teams that are adding a memory feature to a chat product and need a shared yardstick for whether recall works, since it supplies the histories, the ground truth answers and the scoring scripts rather than asking you to construct a test set. It is the wrong tool if you need to test live retrieval against a live corpus, because the intended workflow is to feed the history to your system offline and score the file of outputs afterwards, and it is also the wrong tool if you want agentic behaviour, which the authors have moved to a separate LongMemEval-V2 project. Before you start, download the cleaned dataset rather than the original release, since the history sessions were revised in 2025/09 to stop interference with answer correctness. The licence is MIT, the repository has no tagged releases, and the last push was on 2026-05-11.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. xiaowu0162/LongMemEval on GitHub