What is Chunking in RAG?

Why documents are cut up at all

A retrieval system needs a unit. It cannot index a document and also index a sentence, so it chooses one, and that choice is called a chunk. The unit has to satisfy two constraints pulling in opposite directions. It must be small enough that its embedding is specific, because an embedding of a whole long document is an average of everything in it and matches nothing sharply. It must be large enough to contain the answer, because a chunk that stops in the middle of an explanation cannot answer a question about the end of it.

Everything else in the pipeline sits on top of that decision. A document is parsed into elements, the elements are grouped into chunks, the chunks are embedded and indexed, a query retrieves some of them, and those chunks are assembled into the context the model reads. Chunking is the second step of five, and it is the only one that decides what the retrieval step is even capable of finding.

That is why its errors are unusually quiet. A parser that cannot read a PDF produces an empty index, which is visible. A chunker that cuts through a numbered list produces chunks that embed perfectly well, rank sensibly against a query, and contain half an answer. The retrieval metrics do not collapse; they are merely mediocre, and the reason is a few hundred characters in a file you stopped reading after the first version.

The second reason chunking is under-considered is that it is a one-time decision that silently becomes permanent. Changing it means re-embedding the corpus, which on a large index is a job with a cost measured in money and in time.

The strategies and what each assumes

Five families are in common use, and each one is making a different assumption about your documents.

Fixed size with overlap. Split every N tokens, overlapping by some amount so a sentence near a boundary is not lost. It needs no understanding of the document, which is its whole appeal, and it is the reason many production systems have chunks that start mid-sentence. Overlap also creates near-duplicate chunks in the index, which occupy top-k positions that a second, different chunk could have used.

Structure-aware splitting. Cut at headings, sections, list boundaries, or HTML tags. This is dramatically better when the document is well structured and roughly useless when it is not, and the hard case is PDFs, where the structure has to be inferred from layout and is frequently inferred wrong. Our analysis of Ontos-AI/knowhere describes it as turning PDFs and slide decks into hierarchy-native chunks with citations, and calls the dual-track design the interesting part of it.

Recursive splitting. Split on the most semantic separator available, such as paragraphs, then sentences, then words, until the piece fits the budget. It degrades gracefully, which is why most libraries default to something like it.

Semantic chunking. Embed every sentence, group consecutive sentences whose vectors are similar, and cut where the similarity drops. This finds topic boundaries that no fixed size will find, and it costs a full embedding pass over the document before you embed the chunks, and its results move when your embedding model moves. isaacus-dev/semchunk is the compact version of this idea in Python: our analysis describes it as splitting text into token-bounded chunks while keeping local semantic context and working with any tokenizer.

Parent-child indexing. Index small chunks for precision, then return the larger section they came from to the model. The retrieved unit and the read unit become different things, which fixes both constraints at once at the cost of a mapping to maintain.

When chunking decides the result, and when it does not

Chunking matters most when documents are long and structurally varied, because that is where a single fixed size has to be wrong for something.

Long documents are the case the technique exists for. A manual split into sections often outperforms a clever algorithm on a well-organised manual, and no algorithm rescues a document whose structure was never there.

Mixed corpora are the case where one strategy cannot fit. Product documentation, tickets, contracts and transcripts have different shapes, and a chunker tuned for one is a poor fit for the others. Our analysis of ekimetrics/adaptive-chunking is the clearest statement of the problem: it scores four chunking methods against five intrinsic metrics and keeps the winner for each document, accepted at LREC 2026, shipped as a Python package from source rather than from PyPI.

Granularity is a question about the task. A question that needs one fact wants a small, specific chunk. A question that needs a summary wants the whole section, and handing it a paragraph produces a confident answer built from half the context.

It does not decide anything when the documents are small enough to embed whole, when there is no retrieval step at all and the whole document fits the model's context, or when the retrieval stage is broken for a different reason. Teams often tune chunk size for a week while the real defect is an embedding model that has never seen this domain, and a change of chunk size will not reveal it.

It is also the wrong lever when the answer needs the document as a whole. Splitting is a bet that the useful information is local, and that bet is false for questions of the form summarise this contract.

The traps

Six of them, in rough order of how much time they cost.

Counting the wrong unit. A chunk size expressed in characters is not a chunk size. Tokenizers differ, and code, tables and CJK text tokenize at very different rates, so a fixed character count produces chunks that overflow the embedding model's limit on some documents and are needlessly small on others. Sizing against the tokenizer is the only version that means anything.

Cutting structures in half. A numbered list, a table, a code block and a definition with its caveat are all units. Cut anywhere else and the chunk is a fragment, and the embedding of a fragment is a fragment.

Losing the metadata. The section path, the page number, the file name and the document id have to travel with the chunk, because they are what lets the model cite a source and what lets a user open the right document. They are also what every filter in a real system runs on, and they are usually dropped first because nothing fails without them.

Overlap without thought. Overlap protects the boundary sentence and creates near-duplicates that crowd the result list. Both effects are real and the trade is worth measuring rather than guessing at.

Chunks that overflow the reader. The chunk has to fit the cross-encoder in a reranking stage, and it has to fit the context the model is given alongside other chunks. Chunks sized for the embedding model alone produce a system that indexes fine and truncates at generation time.

A one-time decision. The chunking that worked with last year's embedding model is not automatically right for this year's, and the size that suited a small corpus may be wrong once the corpus is ten times larger and holds more boilerplate.

How the projects divide

The work splits into three groups, and the group tells you which half of the problem you are solving.

Parsers that own the chunking. Unstructured-IO/unstructured is the largest of these, and our analysis describes it as a Python library and optional ETL pipeline service that converts PDFs, Word documents, HTML, images and more than sixty other file formats into clean text elements a language model can consume, with the open-source library running locally and a commercial platform and an MCP server extending it to production workflows. Ontos-AI/knowhere is the same problem with hierarchy as the organising principle, producing chunks that carry citations; our analysis also notes its weak spot, that the README never explains how to install the server yourself, which is exactly the kind of thing to check before adopting a document pipeline. Dicklesworthstone/llm_aided_ocr sits at the hardest end of the range: our analysis describes a Python pipeline that runs Tesseract and then sends the text through a local or API model to correct OCR errors, chunk it and emit markdown, which is useful when the OCR output is nearly right and costs you a second model pass over every chunk.

Libraries that do only the cutting. isaacus-dev/semchunk is the compact Python option. feyninc/chonkiejs ports the Python chonkie library to TypeScript with a zero-dependency core, and our analysis of it is that it fits when you need chunkers inside a Node or browser app and fits poorly if you need the full Python feature set, which is the usual trade for a port.

Research on the choice itself. ekimetrics/adaptive-chunking is the one to read if you are deciding between strategies rather than implementing one, because it is built around scoring methods against intrinsic metrics per document.

And one adjacent case worth knowing. headroomlabs-ai/headroom compresses tool outputs, logs, files and RAG chunks before they reach a model, as a library, a local proxy or an MCP server. Our analysis notes the savings on JSON are large and that the reversible cache and the wrapped-agent setup are the parts to check before adopting. It does not improve retrieval, and it improves the context you pay for, which is a different problem and sometimes the one that is actually binding.

In practice

Chunking decides what a retrieval system is capable of finding, and the failure mode is a plausible answer built from half an idea, which is why it deserves more attention than a default size gets. Size chunks in tokens rather than characters, keep document metadata attached, cut at structure when the structure exists, and measure the result on your own queries rather than assuming a smaller chunk is a better one. Read isaacus-dev/semchunk for a compact implementation, Unstructured-IO/unstructured for the parsing layer that has to happen first, and ekimetrics/adaptive-chunking for the argument that no single strategy fits every document. And record the chunking configuration next to the embedding model, because you will change one of them eventually.