Adaptive Chunking Picks a Splitter Per Document, and Ships the Metrics It Judges Them With
Adaptive Chunking: automatically select the best chunking method per document for RAG. Accepted at LREC 2026.
At a glance
- What is it?
- Ekimetrics' LREC 2026 implementation evaluates four chunking methods against five intrinsic metrics and selects the best per document. The selection machinery is the interesting part; the shipped method set is narrow.
- Who is it for?
- Adopt it if you have a heterogeneous corpus (technical, legal, sustainability reporting) and you are willing to run per-document method selection rather than commit to one splitter. Do not adopt it if your documents are uniform, or if the CC BY-NC-SA 4.0 coreference extra is incompatible with your deployment, since the filtered missing reference metric depends on it.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 132 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- ekimetrics/adaptive-chunking404 ★ · +15
| Repository | 2026-09-11 | 2026-11-15 |
|---|---|---|
| ekimetrics/adaptive-chunking | 389 | 404 |
15 Stars in the last 65 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
The problem is that one splitter is being asked to serve every document
A RAG pipeline usually fixes its chunking once. LangChain recursive splitting with a token target is the common default, and it is applied to contracts, technical manuals and sustainability reports alike. The README states the premise directly: no single chunking method works best for every document. Adaptive Chunking takes that as an empirical claim rather than a slogan, and the paper's Table 5 is the evidence offered. On 99 queries, the adaptive selection answered 65, LangChain recursive answered 49, and page splitting answered 49. Retrieval completeness is reported at 67.7 against 58.1 and 59.1. The audience is therefore narrow and specific: teams running retrieval over a mixed document set, where a single splitting policy is measurably leaving answers on the table. If your corpus is one genre, one layout and one length distribution, the premise does not apply to you.
Five intrinsic metrics, no ground-truth answers required
The mechanism that makes per-document selection possible is a scoring layer that does not need labelled questions. Five metrics are implemented in src/adaptive_chunking/metrics.py. Size Compliance counts the fraction of chunks inside the target token bounds. Intrachunk Cohesion measures the similarity between a chunk's sentences and the chunk's own embedding. Contextual Coherence compares each chunk to its surrounding context window. Block Integrity counts paragraphs, tables and lists that survive the split intact. Filtered Missing Reference Error tracks coreference chains, entity and pronoun pairs, that a boundary would break. The README describes these as intrinsic: they score a chunking output on its own terms. That is the design decision worth noticing. Selection can run at ingestion time, per document, without a query set. The cost is that the metrics are proxies. A chunk set can score well on cohesion and coherence and still retrieve badly for a particular question distribution, which is why the paper also reports the RAG evaluation separately.
Four methods ship by default, and two of them share a splitter
The default method set is Recursive at 1100 tokens, Recursive at 600 tokens, Page, and LLM Regex. The first two are the same recursive splitter at two target sizes, so the selection is partly a granularity choice rather than a strategy choice. Page splits on page breaks with post-processing to enforce size constraints. LLM Regex asks a model to generate document-specific regex split patterns, which is the only method whose behaviour changes per document before selection even begins. The recursive splitter lives in src/adaptive_chunking/splitters.py; the others are in src/adaptive_chunking/paper/splitters.py. The README states that you can register any callable taking text and returning a list of chunks, so the narrow default set is a starting point rather than a ceiling. In the paper's Table 3, adaptive selection reaches a mean of 91.07 across the five intrinsic metrics, against 89.80 for LLM regex and 88.62 for LangChain recursive.
Getting it running takes four commands and one model download
Installation is a source checkout, not a package install. The README's PyPI badge is commented out, so the documented path is git clone https://github.com/ekimetrics/adaptive-chunking.git, then cd adaptive-chunking, then pip install -e ".[dev]". Extras are split by concern: pip install -e . for the core splitter and metrics, ".[parsing]" for the PDF and Excel backends, ".[paper]" for paper reproduction, and ".[coref]" for coreference resolution. Some metrics need a spaCy model, fetched with python -m spacy download en_core_web_sm. The quick-start call is chunk_files("path/to/pdfs/", chunk_size=600, chunk_overlap=50), which parses and chunks in one step and requires the parsing extra. Each returned chunk is a dict with doc_name, chunk_index, chunk_text, chunk_pages, titles_context and chunk_len. A parser can be swapped, for example PyMuPDFParser, which the README presents as the lightweight option next to Docling (default, open source) and Azure Document Intelligence (cloud).
The coreference extra carries a non-commercial licence
This is the constraint most likely to end an adoption conversation, and it is stated plainly in the install section. The core package is MIT. The coref extra is marked CC BY-NC-SA 4.0, non-commercial. Filtered Missing Reference Error is one of the five selection metrics, and it depends on coreference resolution. The repository does ship pre-computed maverick-coref clusters under data/clair/mentions/, described as requiring no GPU, but that covers the 33 CLAIR documents used in the paper, not your corpus. Running the full five-metric selection over your own documents therefore points at the coref extra and its licence. The practical options are to evaluate whether the four remaining metrics select well enough on your data, to substitute your own coreference step, or to treat this as a research and internal-evaluation tool rather than a production component. I am not in a position to give legal advice on what non-commercial means for your organisation; that determination belongs with whoever handles licensing where you work.
Where a fixed splitter is still the right call
The honest comparison is against doing nothing special. LangChain's recursive splitter is a single dependency-light function with no selection step, no metric computation, no spaCy model and no per-document scoring pass. It scores 88.62 on the intrinsic mean and 58.1 on retrieval completeness in the paper's tables. Adaptive Chunking's advantage on the intrinsic mean is roughly two and a half points, and the retrieval completeness gap is wider at 67.7 against 58.1. Those are averages across 33 documents in three domains. If your corpus is uniform, the selection step has nothing to select between, and you pay the scoring cost for a decision that would have come out the same way. The other case against it is latency and complexity at ingestion: every document is chunked more than once and scored five ways. For a small, stable corpus this is a one-off cost. For a stream of incoming documents it is a per-document pipeline stage that LangChain recursive does not have.
Reproduction data, maintenance and what is not documented
The repository ships data/clair/ with 33 pre-parsed JSON documents and pre-computed coreference mentions, so the paper's chunking evaluation can be re-run without a GPU. The repository has no published GitHub releases and the PyPI badge is commented out, which means versioning is by commit. Upgrading is a git pull plus re-running pip install -e with the relevant extras, and any change to metrics.py or splitters.py changes selection behaviour, so a chunk set produced last month may not be reproducible after a pull. Pin a commit if you need that. Python 3.11 or newer is required. The maintenance question that matters most is whether the coref extra's licence is ever relaxed, because that single dependency gates the full five-metric path.
Editorial conclusion
Adopt it if you have a heterogeneous corpus (technical, legal, sustainability reporting) and you are willing to run per-document method selection rather than commit to one splitter. Do not adopt it if your documents are uniform, or if the CC BY-NC-SA 4.0 coreference extra is incompatible with your deployment, since the filtered missing reference metric depends on it. Before committing, clone the repository, run the quick-start chunk_files call on a sample of your own PDFs, and inspect the per-metric scores to confirm that the winning method differs across your document types. If the same method wins every time, you do not need this framework.
Frequently asked questions
What does Adaptive Chunking actually do?
It evaluates multiple chunking strategies against intrinsic quality metrics and automatically selects the best one for each document, on the premise that no single method suits every document. Both dimensions are modular: you can register your own chunking callables and define your own scoring functions.
Which metrics does Adaptive Chunking use?
Five intrinsic metrics that require no ground truth: size compliance, intrachunk cohesion, contextual coherence, block integrity, and filtered missing reference error, which counts coreference entity and pronoun pairs not broken across chunk boundaries.
How does Adaptive Chunking score against simpler splitters?
It reports a mean of 91.07 across domains, ahead of an LLM regex method at 89.80 and a recursive splitter at 88.62. It leads on size compliance, block integrity, and reference error, but its intrachunk cohesion is below both the sentence splitter and the LLM regex method.
How do I chunk a folder with Adaptive Chunking?
Import the single-call helper and pass a directory or one file with a chunk size and overlap. Each returned chunk is a dictionary carrying the document name, index, text, source pages, titles context, and length, and a different parser backend can be passed in.
What are the licence terms for Adaptive Chunking?
The project is MIT, but the optional coreference dependency used by one metric is under a non-commercial licence, so that metric path carries different terms. The repository also ships a software bill of materials document.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/ekimetrics-adaptive-chunking)Star history chart for your README
[](https://hysenlabs.com/projects/ekimetrics-adaptive-chunking)Was this useful?