PaperGuru-Benchmark: Lifecycle-Aware Memory for Long-Horizon LLM Agents

The gap PaperGuru claims: memory as a missing primitive

The README frames AI infrastructure as three commodity primitives (compute, model weights, retrieval) and argues a fourth is missing: long-term memory with lifecycle semantics. The stated failure mode is that every long-horizon system reinvents this badly. The four LAM axioms are versioned content, structural multi-hop relevance, bounded query cost under unbounded archive growth, and provenance-grounded composition. Axiom one is the one worth pausing on. It says a statement that was correct must become stale after revision, deprecation or retraction, and the memory layer must know. Vector stores do not model that transition. They return the nearest embedding regardless of whether the source was superseded. Axiom three is the operational constraint: the archive grows daily, so routing cost cannot grow with it. The intended audience is engineers building multi-day agents (software sessions touching hundreds of files, literature assistants drafting 200K-token surveys, paper-to-code reproduction agents, clinical-evidence readers) rather than teams adding a RAG call to a chatbot.

Capital Chunk Memory: two surfaces and a temporal graph

The mechanism is a split between chunk heads and chunk contents. Heads form a compact, bounded routing surface, one per artifact. Contents are the unbounded raw text, pulled lazily. A central capital chunk indexes all heads and routes over a temporal artifact graph carrying two edge classes. Structural edges are cites, benchmarked-on, introduced-by, implements. Historical-causality edges are discussed-in, deprecated-by, retracted-by, superseded-by. The second class is what makes axiom one executable: a retraction is an edge, not a deletion, so a query can traverse from a live claim to the artifact that killed it. Query-time context is built by a route-first, expand-second, distill-last pipeline producing evidence cards, which the README calls the single data structure the rest of the system operates on. The Reason stage of the pipeline runs a Compose, Critique, Mutate cycle. The README's own comparison is blunt: MemGPT-style tiers, Ebbinghaus forgetting and knowledge-graph wrappers each handle one or two axioms, and CCM is claimed to satisfy all four without per-task hand-tuning. That last clause is the load-bearing one and the README does not show the experiment that would settle it.

Reported results and why they are hard to audit from here

The README reports 66.05% on PaperBench in the badge and 65.95% in the results table, against a best published baseline of 35.74%, plus 20 of 23 papers above a 41% human ML-PhD bar versus 4 of 23. SurveyBench is reported at 94.66% content score against 80.60%, and 43.76% composite richness against 20.36%. Ten peer-reviewed acceptances are listed across FSE 2026, ICML 2026, TOSEM, AEI and ICoGB. Two things should give a careful reader pause. First, the badge and the table disagree by 0.10 percentage points on the same benchmark, which suggests two different runs or two different aggregation rules; the README does not say which. Second, no releases were retrieved for this repository and the primary language is TeX. The repository layout points at paper/PaperGuru-CCM.pdf and PaperBench/ and SurveyBench/ directories, and the README has a section titled Reproducing the figures, but nothing in the supplied material shows a training script, an evaluation harness or a dependency manifest. Treat the numbers as claims made by the authors in their own document, not as measurements you can check yet.

Getting it running, and the honest answer about setup

There is no installation path in the material. No pip install line, no conda environment, no Dockerfile, no API key configuration, no config keys. The README's reproduction section exists but its contents were truncated before the code blocks, so the only concrete paths visible are paper/PaperGuru-CCM.pdf, PaperBench/, SurveyBench/, assets/figures/, and assets/demos/pipeline_animated.svg. The TeX badge points at paper/PaperGuru-CCM.pdf, which means the paper is checked into the tree rather than fetched from arXiv. If you want to evaluate PaperGuru today, the realistic workflow is: clone the repository, read the PDF, then inspect PaperBench/ and SurveyBench/ to see whether they contain runnable evaluation code or only result artifacts. A repository whose primary language is TeX and whose README is written as a launch page is usually a paper companion, not a library. That is not a defect by itself, but it changes what you can do with it this week.

Where lifecycle memory is the wrong tool

Lifecycle semantics cost something. Maintaining retracted-by and superseded-by edges means every ingested artifact needs a versioning pass, and that pass has to run again whenever a source changes upstream. For a corpus that does not change, this is pure overhead. A support-bot reading a frozen product manual, a code assistant indexing one repository at a single commit, or a pipeline that re-embeds everything nightly from scratch gains nothing from temporal edges, because nothing in the archive ever goes stale. The second limitation is scope. The README's own examples are research artifacts: papers, surveys, citation graphs, trial records. Those have formal revision and retraction events. A chat log, a Slack archive or a stream of user tickets has corrections but rarely has a canonical retraction edge, so the historical-causality class has little to attach to. The third is the bounded-cost axiom. Route-first, expand-second, distill-last is a routing discipline; whether it actually holds cost flat as the archive grows depends on the capital chunk staying small, and the README does not describe what happens when the head count outgrows that index.

How this differs from a vector store plus a graph wrapper

The closest conventional alternative is Pinecone-class retrieval, which the README itself names as the third commodity primitive. A vector store answers one question: which chunks are nearest to this query embedding. It has no notion of a chunk being wrong now, and multi-hop relevance has to be bolted on with a separate graph database and a query planner you write yourself. The difference in approach is that PaperGuru puts the edge classes inside the memory layer and makes routing capital-first rather than similarity-first. In a vector store, a retracted paper and its correction compete on cosine distance and may both surface. In CCM, the retracted-by edge is traversable at route time, so the correction is reachable in one structural hop rather than one similarity hop. Whether that advantage survives contact with a real corpus is exactly what the PaperBench and SurveyBench numbers are meant to demonstrate, and it is why the missing harness matters more than the missing install instructions.

Maintenance, licence and what the repository actually commits you to

The last push is dated 2026-06-08 and the repository is not archived. No releases were retrieved, so there is no version to pin and no changelog to read. For a paper companion repository that is normal; for a dependency it is disqualifying. The licence field reads NOASSERTION, which means GitHub could not map the licence file to a recognised identifier. That is not the same as having no licence, and it is not the same as having a permissive one. Until you open the licence file and read it, you do not know whether the TeX sources, the figures under assets/, or any code under PaperBench/ can be reused in a commercial product. The TeX-heavy layout also means the maintenance surface is mostly document build, not software: figure regeneration, PDF compilation, and whatever the Reproducing the figures section specifies. If the benchmark directories turn out to hold only result files, the practical maintenance cost for an adopter is zero because there is nothing to maintain, and also zero because there is nothing to run.

Editorial conclusion

Adopt PaperGuru as a reading and design reference if you are building a long-horizon agent whose archive gets revised, retracted or superseded over time, because the four LAM axioms and the capital-first routing scheme are a concrete alternative to flat vector retrieval. Do not adopt it as a dependency today: no releases were retrieved, the primary language is TeX, and the licence is NOASSERTION. Before committing engineering time, verify three things in the repository itself: whether runnable code exists under PaperBench/ and SurveyBench/, what the licence file actually grants, and whether the 65.95% and 94.66% figures can be regenerated from the scripts the README points to.

Official sources

1. AutoTrustAI/PaperGuru-Benchmark on GitHub
2. Issues
3. README