MemoryAgentBench: what the ICLR 2026 memory benchmark actually measures, and what it does not
Open source code for ICLR 2026 Paper: Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
At a glance
- What is it?
- MemoryAgentBench packages four memory competencies into chunked multi-turn datasets and ships bash entry points for long-context, RAG and agentic-memory baselines. It is an evaluation harness, not a memory library, and its strict exact_match parsing is the first thing to check before you trust a score.
- Who is it for?
- Adopt MemoryAgentBench if you are writing a paper or an internal report on memory mechanisms and you can pin the environment, because the four-competency split and the inject-once-query-many design give you a comparable number across long-context, RAG and agentic-memory baselines. Do not adopt it if you want a memory component to ship inside a product: there is no library, no server, and no incremental-update API here.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 26 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The four competencies MemoryAgentBench separates
Most memory evaluations collapse into one question: can the system find the right passage again. MemoryAgentBench splits that into four competencies. Accurate Retrieval (AR) covers pulling the right span back out. Test-Time Learning (TTL) covers whether the agent improves from what it has already seen, without weight updates. Long-Range Understanding (LRU) covers reasoning that spans far more context than a single turn. Conflict Resolution (CR) covers what happens when a later statement contradicts an earlier one and the agent must pick a side. The datasets map onto those categories: event_qa, ruler_qa1 and ruler_qa2 for AR, the ICL series (ICL_banking, ICL_clinic, ICL_nlu, ICL_trec_coarse, ICL_trec_fine) for TTL, detectiveQA for LRU, and fact_mh plus fact_sh for CR. Two of the datasets, EventQA and FactConsolidation, were constructed by the authors rather than reformulated from prior work. The rest are collected and reformulated from RULER, InfBench, HELMET and LongMemEval, which the acknowledgement section names directly. The useful part of this split is that a system can score well on retrieval and badly on conflict resolution, and the benchmark will show you that instead of averaging it away.
Inject once, query many times: the data flow that makes this cheap to run
The design decision the README calls out by name is inject once, query multiple times. One long text is chunked and loaded into the agent once, then several questions are asked against that same loaded state. That is a direct response to the cost profile of multi-turn memory evaluation, where re-ingesting a long document per question dominates the bill. The README states this design significantly improves evaluation efficiency. It also means the benchmark is measuring memory under a specific interaction shape: the agent sees the material in chunks, simulating a conversation, and then answers a batch of questions. Chunk size is treated as a variable rather than a constant, which is why there is a dedicated ablation script for it. That is the right call, because chunk boundaries are exactly where retrieval systems fail, and a benchmark that fixes chunk size hides the failure. If your production system uses a different chunking strategy, the ablation results are the part of this benchmark most likely to transfer to your setting.
Getting it running: conda, pip pins and the .env keys
Setup is a conda environment plus three pip steps. The README gives python=3.10.16 for the environment, then torch, then requirements.txt, then a numpy pin below 2. That last pin matters because a lot of retrieval and embedding code still assumes the NumPy 1.x ABI. Two packages are deliberately excluded from requirements.txt. The README states that hipporag is left out because the current version causes package version conflicts, and suggests a separate environment for it. cognee and letta are handled with an unusual install-then-uninstall pair, which the README offers as a fallback if you hit package errors after installing requirements.txt. Data comes from the HuggingFace dataset ai-hyz/MemoryAgentBench and, per the README, can download automatically when you run the code, though the entity2id.json file for the Movie Recommendation task is called out separately as something not to forget. Credentials live in a .env file at the project root with OPENAI_API_KEY, Anthropic_API_KEY and Google_API_KEY, plus LLM_MODEL and LLM_API_KEY for Cognee. Entry points are bash scripts: bash_files/eniac/run_memagent_longcontext.sh for long-context agents, run_memagent_rag_agents.sh for RAG and agentic memory methods, and run_memagent_rag_agents_chunksize.sh for the chunk-size ablation. The scripts take --agent_config and --dataset_config paths. LLM-judge scoring for LongMemEval and InfBench summarization runs through llm_based_eval/longmem_qa_evaluate.py and llm_based_eval/summarization_evaluate.py.
The hipporag pin is the environment's fault line
The README is explicit that hipporag 2.0.0a3 requires openai==1.58.1, and that this may prevent the latest OpenAI models from being used in the same environment. That is not a small footnote. If your evaluation target is a current OpenAI model, you cannot run the hipporag baseline and that model in one environment. You either maintain two environments and reconcile results across them, or you drop hipporag from your comparison. Neither is free. Running the same benchmark across two environments introduces a second source of variance that has nothing to do with memory design. Dropping hipporag weakens the baseline set in a paper about memory methods. The README's own suggestion, a separate environment for hipporag, is the pragmatic answer, but it pushes the reconciliation problem onto whoever writes the results table. Plan for that before you start collecting numbers, not after.
Strict exact_match will fail answers that are right
The metric mapping is where most readers will get burned. The README states that accuracy in the paper is shorthand for either substring_exact_match or exact_match in the JSON output, and the table spells out which is which. AR datasets and CR datasets use substring_exact_match. detectiveQA and the entire ICL series use exact_match. Recsys reports Recall@5. LongMemEval uses an LLM as judge with no JSON field listed. InfBench summarization reports F1 via LLM-as-judge following HELMET. The warning attached to exact_match is blunt: parsing is strict, so if the ground truth is 43 and the model returns label: 43, it counts as incorrect. The README recommends flexible parsing when adapting the benchmark to your own pipeline. Take that seriously. A model that reasons correctly but formats verbosely will look worse than a model that guesses a bare token, and the resulting comparison is about output formatting discipline, not memory. If you report exact_match numbers without checking the parser, you are reporting something other than what you think you are reporting.
How this differs from LongMemEval and HELMET
MemoryAgentBench is built on top of other evaluations rather than beside them. LongMemEval supplies data that MemoryAgentBench reformulates and chunks, and it also remains a scoring path inside the repo through llm_based_eval/longmem_qa_evaluate.py. HELMET is the source of the InfBench summarization F1 protocol. The difference in approach is the one the README names: MemoryAgentBench chunks the source material to simulate incremental multi-turn interaction, and then reuses one loaded context across multiple questions. LongMemEval as used here is a single dataset with an LLM judge. HELMET is a summarization evaluation protocol. MemoryAgentBench's contribution is the four-competency framing applied across a set of reformulated datasets, plus the two new datasets EventQA and FactConsolidation, run through a shared harness. If you only need summarization F1, HELMET is the more direct route. If you need a memory-mechanism comparison across retrieval, test-time learning, long-range understanding and conflict resolution, the shared harness is the reason to use this repo instead of wiring the four datasets together yourself.
Maintenance, licence and what the repository does not promise
The last push recorded for the repository is 2026-08-20, and the README's update log runs from July 2025 through a May 2026 entry adding GPT-5-Mini results. There is no release list, so versioning is by commit. The licence is MIT, which permits commercial and academic reuse with attribution and without warranty, but this is a description of the licence text and not legal advice; check the LICENSE file and your own obligations, particularly around the upstream datasets from RULER, InfBench, HELMET and LongMemEval, whose terms are separate from this repository's. The README closes its update section with a note that more details, such as dataset collection, are coming, and a TODO list with the LRU dataset item struck through. That tells you the documentation is still filling in. The benchmark is a moving target: the paper was accepted at ICLR 2026 and the README states the authors will make improvements. Pin a commit hash if you intend to cite numbers, because a later commit can change a dataset or a parser and invalidate your table.
Editorial conclusion
Adopt MemoryAgentBench if you are writing a paper or an internal report on memory mechanisms and you can pin the environment, because the four-competency split and the inject-once-query-many design give you a comparable number across long-context, RAG and agentic-memory baselines. Do not adopt it if you want a memory component to ship inside a product: there is no library, no server, and no incremental-update API here. Verify two things before you publish a table. First, whether your model output survives the strict exact_match parser, since the README warns that a ground truth of 43 against a model answer of label: 43 counts as incorrect. Second, whether hipporag 2.0.0a3 and its openai==1.58.1 pin belong in the same environment as the models you intend to test, because the README states that this conflict may block newer OpenAI models.
Community notes