LoCoMo Refined: a stricter judge and a cleaned QA set for long-conversation memory

The scoring gap LoCoMo Refined was built to close

The README states the original LoCoMo evaluation could over-credit answers that were close in topic but wrong in detail, naming time, missing facts, and unsupported additions as the main offenders. That is a scoring problem, not a dataset problem, and it matters most for the questions where a memory system has to place an event on a calendar, count the members of a set, or avoid padding an answer with plausible but unsourced detail. The target user is whoever publishes a long-conversation memory number and wants it to survive scrutiny: benchmark maintainers, memory-layer vendors, and evaluation engineers comparing two systems on the same conversations. The repository reports that its refined judge agrees with human labels on 86.33% of 300 human-alignment samples, against 43.67% for the original judge. Those two figures are the entire argument for the project, and they come from the project's own at-a-glance table rather than from an independent replication.

Five correctness rules and the judge that applies them

The refined judge is documented as a single principle, quoted in the README as inclusive without contradiction, complete without overreach, expanded into five conditions. A prediction is correct only if it includes all required information from the gold answer, does not contradict it, introduces no unsupported extra details, preserves the correct temporal granularity, and handles list-style answers without dropping required items or adding unsupported ones. The judge prompt lives in src/llm_judge.py and the runtime in src/llm_judge_runtime.py. The configuration table pins temperature at 0.0 and disables thinking mode where the model supports it, which is what you would expect for a judge you intend to run repeatedly and compare across systems. The original judge is kept alongside the refined one and is selectable, so the two prompts can be run over identical predictions. That design choice is the useful part: the delta between the two judges is itself the measurement, and the README reports that delta for five named systems.

What the re-scoring table actually shows

The README reports re-scoring the same system predictions under the refined judge and tabulating the absolute drop against the original LoCoMo judge. EverMemOS falls 22.07 points to 58.25%, MemOS falls 17.30 points to 63.60%, MemPalace falls 15.78 points to 58.68%, and Mem0 falls 15.56 points to 48.91%. MemoraX AI is listed at 82.65% with no drop recorded. The spread is the interesting signal: a uniform tightening would produce roughly similar drops, and these range from 15.56 to 22.07 points. A system that leans on temporal answers or list answers should lose more, because those are the two categories the refined judge singles out. Treat the ranking as the project's own claim, since the predictions were not re-generated, only re-scored, and the README does not describe how those predictions were produced.

The cleaned QA set and its answer schema

The dataset ships at data/raw/locomo_refined.json with 1,382 questions, and data/public/questions.jsonl holds the public question list. Of those, 337 samples were revised after AI-assisted screening and review by 5 human annotators, covering ambiguous wording, reversed subject-object relationships, and time information inconsistent with the original conversations. The schema detail that will bite you is the answer field: it is a list of acceptable gold answers, and each list item is a complete correct answer candidate. If a question needs multiple facts, those facts belong together inside one answer string rather than split across list items as a required set. Anyone who writes a scorer against this file assuming the list is a conjunction will mis-grade multi-fact questions, and the README flags that explicitly.

Running the two metric paths

Environment setup assumes Python 3.11 or newer and the openai and tenacity packages. The README gives a conda path, creating an environment with conda create -n locomo-refined python=3.11 -y, then pip install openai tenacity, then exporting LOCOMO_PYTHON_BIN to the interpreter path. Predictions are read from ./outputs/predictions.jsonl by default, one JSON object per line with qa_id and predicted_answer keys, and the qa_id values must match ./data/public/questions.jsonl. Lexical scoring runs as ./scripts/run_eval.sh --metrics f1 bleu and writes predictions_scored.jsonl, predictions_scored_summary.json, and predictions_scored_summary.md into ./outputs. The judge path adds environment variables: EVALUATOR_MODEL set to qwen3-14b, an optional EVALUATOR_API_BASE for a custom OpenAI-compatible endpoint, and an optional EVALUATOR_API_KEY. The full command is ./scripts/run_eval.sh --metrics llm f1 bleu --llm-judge refined. Accepted aliases for EVALUATOR_MODEL include qwen3-14b, qwen3_14b, Qwen/Qwen3-14B, qwen/qwen3-14b, and vendor-prefixed variants ending with those. A non-Qwen model triggers a warning and requires confirmation, which is a deliberate guard against silently swapping the judge and reporting the resulting number as a LoCoMo Refined score.

Where the project constrains you

The judge is a 14B model, so every scored prediction is an API call against an endpoint you supply, and the README documents no batching, caching, or local inference path. The runtime file exists but the README does not describe retry or rate-limit behaviour beyond the tenacity dependency. The licence is the sharper constraint. The repository metadata reports NOASSERTION while the README badge and LICENSE.txt reference point to CC BY-NC 4.0, which is a non-commercial licence; if your evaluation feeds a commercial product claim, that is a question for your own counsel and not something this article can settle. There are no tagged releases retrieved, so version pinning means tracking the main branch or the v1.0.0 tag the badge links to. The technical report is marked coming soon, which means the human-annotation protocol, the annotator agreement figures, and the AI screening criteria are not yet published in full.

Against the original LoCoMo judge

The alternative is not a different benchmark but the original LoCoMo judge, which this repository retains as the selectable original option. The difference is in strictness, not in data. The original judge is reported to agree with human labels 43.67% of the time; the refined judge 86.33%. Running both over the same predictions is the comparison the tooling is built for, and it is why keeping the original prompt matters. If you want a benchmark whose judge accepts paraphrases and partial list coverage, the original is the looser instrument and will produce higher scores. If you want scores that track whether the memory system got the date right, the refined judge is the one to run. The cost of the switch is visible in the drop column: the same systems score 15 to 22 points lower, so any historical LoCoMo number you compare against is not on the same scale.

Who should adopt it, and what to check first

Adopt LoCoMo Refined if you already report on LoCoMo and want a judge that penalises time drift, missing facts, and unsupported additions, and if you can run a Qwen3-14B endpoint at temperature 0.0. Do not adopt it if you need a permissively licensed dataset, if you cannot route judge calls to an approved endpoint, or if you need a benchmark with a published annotation protocol today, since the technical report is listed as coming soon. Before you cite a number, three checks. Confirm your EVALUATOR_MODEL alias resolves to the official judge rather than a substitute that only warns. Confirm every qa_id in your predictions file exists in data/public/questions.jsonl. And re-score your existing predictions with --llm-judge refined and --llm-judge original over the same file, so the delta you report is measured rather than assumed. The 337 revised samples and the 1,382-question total are the fixed surface; everything else in the repository is still moving on main.

Editorial conclusion

Adopt LoCoMo Refined if you report memory results on LoCoMo and want the judge to stop accepting answers that are topically close but wrong on time or detail. Do not adopt it if you need a permissively licensed dataset or a vendor-neutral judge, because the repository badge says CC BY-NC 4.0 and the official judge is Qwen/Qwen3-14B. Before publishing numbers, verify your EVALUATOR_MODEL alias resolves to the official judge, confirm your qa_id values match data/public/questions.jsonl, and re-score your existing predictions rather than re-running your system.

Official sources

1. Issues
2. mem-eval-suite/LoCoMo_refined on GitHub
3. README