EverMind-AI/MSA: routing top-k compressed documents inside attention, and where the 100M-token claim stops

Routing happens inside attention, not in a retrieval service in front of the model

MSA folds retrieval and generation into one differentiable loop. Document latent states, the K, V and a routing key called Kr, are compressed by chunk-mean pooling. A router projector computes relevance by cosine similarity, mean-pooled over heads and then taken as a token-wise maximum. It selects the top-k documents and concatenates their compressed K/V with the query's local K/V for autoregressive decoding.

Routing is applied only in the upper layers. The lower layers keep independent document processing, which the project says supports hierarchical alignment. The consequence of this design is that the retrieval step is trained with the model rather than bolted on. That is the point of the method, and also its cost: you cannot swap in a different embedding model or reranker the way you can with a RAG stack, because the router is part of the network you train.

Document-wise RoPE resets every document to position zero

Two position schemes work together. With parallel, document-wise RoPE, each document starts its positions from 0. The README says this prevents position drift between training on short inputs and inferring on long ones, and that it lets 64k training extrapolate to 100M tokens.

Global RoPE handles the active context. The query's starting index is offset by k, the number of retrieved blocks, so causal order is kept as background, then query, then generation.

The limit is that the 64k training length is the only training length stated in the sections above. The 100M-token behaviour is therefore extrapolation by design, not something the model was trained at. Anyone expecting to fine-tune on their own data should plan for the 64k regime and treat anything beyond it as inherited from the position scheme.

Memory must be encoded offline before a query can reach it

Inference runs in three stages. First comes global memory encoding, done offline: a forward pass over the corpus caches the chunk-pooled K, V and Kr. Second is online routing and context assembly, where the query is projected to Qr, matched against the cached Kr to pick the top-k, and only the selected K/V are loaded and joined with local context. Third is sparse generation over that assembled context.

Because stage one is a pass over the whole corpus, a document has to go through the model before it can be found. A knowledge base that changes constantly means repeating that encoding for new material. The README also describes an optional Memory Interleave mode for multi-hop reasoning, alternating generative retrieval, context expansion and generation across rounds, which adds more model calls per question.

The 100M-token setup depends on host memory as well as two GPUs

The headline hardware figure is 100M-token throughput on 2xA800 GPUs. The mechanism is tiered storage under a design called Memory Parallel. The routing keys Kr are sharded across GPUs, with the query broadcast, scored locally and reduced globally. The content K and V stay in host DRAM and are fetched asynchronously when selected.

That split balances VRAM against throughput, but it means system memory bounds the size of the memory bank. Two GPUs is the GPU count, not the full bill of materials. Someone with a single consumer card and modest RAM cannot assume the 100M-token setting is reachable.

The pinned requirements give the software side: torch==2.6, torchvision==0.21.0, transformers==4.51.3, liger_kernel==0.5.10 and accelerate==1.0.1, with tqdm and openai unpinned. Those pins mean the stack will not drop into an environment that already runs a different torch or transformers version.

Quality is measured on 9 QA sets with an LLM judge, on memory banks from 277K to 10M tokens

The evaluation setup is specific. QA uses nine datasets: MS MARCO v1, NQ, DuReader, TriviaQA at 10M, NarrativeQA, PopQA, 2WikiMultiHopQA, HotpotQA and MuSiQue, with memory banks from 277K to 10M tokens. The metric is an LLM judge scored from 0 to 5. A needle-in-a-haystack test uses RULER with 8 subtasks from 32K to 1M tokens. The backbone is Qwen3-4B-Instruct-2507, and the comparisons are same-backbone RAG and stronger RAG stacks built on KaLMv2 with large generators and an optional reranker.

The point to carry away is the range. The question-answering tables stop at 10M tokens and the needle test at 1M. The 100M figure comes from the throughput claim and from the degradation curve on MS MARCO, where some baseline curves end early because of their context limits. A reader should not assume 100M-token QA accuracy was graded the same way.

Table 1 shows MSA losing one dataset to a reranked RAG baseline

Against same-backbone Qwen3-4B RAG, the summary gives MSA an average of 3.760. That is an improvement of 16.0% over standard RAG, 11.5% over RAG with reranking and 14.8% over HippoRAG2, each taken at its best retrieval depth. On MS MARCO v1 the adaptive MSA scores 4.141 against about 3.0 for the baselines, and on 2WikiMultiHopQA it scores 4.280 where the plain baselines at top-1 sit near 1.1.

The exception is NarrativeQA, where reranked Qwen3-4B at top-1 scores 3.638 and MSA scores 3.395. The README itself says MSA leads on all but NarrativeQA within the same-backbone group. If your task is long narrative documents, that single row matters more than the average. The abstract also states less than 9% degradation from 16K to 100M tokens, a claim about how much quality falls as memory grows, not about absolute accuracy.

No releases, a blank license field, and a model on Hugging Face

The repository has no GitHub releases. Its license field is not set on GitHub, although the README carries a badge that links to the MIT license page, and the top-level entries do not include a LICENSE file. Check which terms actually apply before reusing the code.

The layout is small: README.md and README_ZH.md, QUICK_START.md, requirements.txt, and the directories assets/, paper/, scripts/ and src/. The README links a Hugging Face model named MSA-4B, the paper on arXiv at 2603.23516, and a Zenodo record. The last push was on 2026-05-06.

Also note that the abstract's claim that MSA surpasses leading long-context models is a statement from the authors, and the repository does not substitute for running the comparison on your own data.

Editorial conclusion

Read this if you work on long-context or memory-augmented models and want a trainable alternative to a RAG pipeline, with the Qwen3-4B results and the paper as your starting point. Do not read it as a ready memory layer for an existing product: there are no releases, the license field is blank on GitHub, and the quality numbers cover memory banks of up to 10M tokens while the 100M figure is about throughput and degradation. Before building on it, confirm the license terms and the 2xA800 plus host memory budget.

Frequently asked questions

Official sources

1. EverMind-AI/MSA on GitHub
2. Issues
3. Project website
4. README