Splitting documents into chunks

The unit RAG retrieves isn't a whole document but the small pieces, or chunks, the document is split into. A page about timeouts may have several sections; when a user asks "how do I disable timeouts", you want back just the short passage about disabling them, not the whole page.

How to split looks like a minor technical detail, but it has a big effect on how well RAG works. Chunks that are too large dilute the relevant content with lots of irrelevant text at retrieval time; chunks that are too small break a complete idea apart, so what comes back is half a sentence. This lesson compares several ways of splitting on the real httpx documentation.

Method 1: fixed length

The simplest approach: ignore the content and cut every 800 characters. To avoid slicing a sentence exactly in two, adjacent chunks overlap by 100 characters.

def split_fixed(text, size=800, overlap=100):
    """办法一：每 size 个字符切一刀，相邻两块重叠 overlap 个字符。"""
    chunks, start = [], 0
    while start < len(text):
        chunks.append(text[start:start + size])
        start += size - overlap
    return chunks

Here is what it does to timeouts.md, the page about timeouts. This is chunk 2:

le.com/api/v1/example", timeout=None)
```

## Setting a default timeout on a client

You can set a timeout on a client instance, which results in the given
`timeout` being used as the default for requests made with this client:

```python
client = httpx.Client()              # Use a default 5s timeout everywhere.
client = httpx.Client(timeout=10.0)  # Use a default 10s timeout everywhere.
client = httpx.Client(timeout=None)  # Disable all timeouts by default.
```

## Fine tuning the configuration

HTTPX also allows you to specify the timeout behavior in more fine grained detail.

There are four different types of timeouts that may occur. These are **connect**,
**read**, **write**, and **pool** timeouts.

* The **connect** timeout specifies the maximum amount of time to wait until
a socket 

It starts with a truncated URL, le.com/api/v1/example", and ends mid-sentence at "until a socket". In between it spans two subsections, half about a client's default timeout and half about the four kinds of timeout. The meaning of this chunk is muddled: if a user asks "what are the four kinds of timeout", it holds only half the answer; if they ask "how do I set a default timeout on a client", it brings a pile of irrelevant text along.

Fixed-length chunking is simple, works on any text, and produces evenly sized chunks. But it pays no attention at all to the document's structure.

Method 2: by heading

The httpx docs are Markdown, already divided into sections by ## and ### headings. Each section covers one thing, which makes it a natural unit for chunking. So split by heading: every heading starts a new chunk.

The most direct way is a regular expression that cuts before every line starting with #:

def split_by_heading_naive(text):
    """办法二（有问题的版本）：凡是以 # 开头的行都当成标题切开。"""
    parts = re.split(r"(?m)^(?=#{1,3} )", text)
    return [p.strip() for p in parts if p.strip()]

Here are the chunks it makes from timeouts.md, showing only the first line of each:

有问题的按标题切法，timeouts.md 的各块开头：
  [ 153 字符] HTTPX is careful to enforce timeouts everywhere by default.
  [  93 字符] ## Setting and disabling timeouts
  [  87 字符] # Using the top-level API:
  [ 186 字符] # Using a client instance:
  [  87 字符] # Using the top-level API:
  [ 127 字符] # Using a client instance:
  [ 424 字符] ## Setting a default timeout on a client
  [1386 字符] ## Fine tuning the configuration
  [ 207 字符] # A client with a 60s timeout for connecting, and a 10s timeout elsewhere.

# Using the top-level API: isn't a heading; it's a Python comment inside a code block. A Python comment looks exactly like a Markdown level-one heading: # followed by a space. So the code block gets cut in the middle, the "Setting and disabling timeouts" section is left with 93 characters of explanation, and the code examples all end up in other chunks.

This kind of bug is well hidden. The program doesn't raise an error and the number of chunks looks reasonable; you only notice by looking at them one by one. Across the whole documentation set, the buggy version produced 236 chunks, 60 more than the correct one, and every extra chunk was a fragment of chopped-up code like this.

The fix is to track whether we're inside a code block (toggle the state on every ```) and never treat a # inside a code block as a heading:

def split_by_heading(text):
    """办法二（修正版）：同样按标题切，但跳过代码块里以 # 开头的注释行。"""
    chunks, current, in_code = [], [], False
    for line in text.splitlines():
        if line.startswith("```"):
            in_code = not in_code
        if not in_code and re.match(r"#{1,3} ", line) and current:
            chunks.append("\n".join(current).strip())
            current = []
        current.append(line)
    if current:
        chunks.append("\n".join(current).strip())
    return [c for c in chunks if c]

After the fix:

修正后的按标题切法，timeouts.md 的各块开头：
  [ 153 字符] HTTPX is careful to enforce timeouts everywhere by default.
  [ 586 字符] ## Setting and disabling timeouts
  [ 424 字符] ## Setting a default timeout on a client
  [1594 字符] ## Fine tuning the configuration

Four chunks, each a complete section, with the explanation and the code examples together.

The lesson from this trap: after chunking, always print a few chunks and look at them. Every document format has its own traps: HTML has navigation bars and footers, PDFs have headers, page numbers and broken tables, code has function boundaries.

Method 3: by heading, with a length cap

Splitting by heading has a problem too: some sections are very long. Across the httpx docs, the longest chunk produced by heading is 5,530 characters. A chunk that long covers a mix of things, which hurts retrieval; and as Module 01, Lesson 5 explained, embedding models like bge-small-zh-v1.5 read at most 512 tokens and simply drop the rest.

So add one more step: split any chunk over the cap further at blank lines (that is, by paragraph), and put its section heading in front of each smaller piece, so every piece knows what it's about.

def split_by_heading_capped(text, max_size=1500):
    """办法三：先按标题切；太长的块再按空行（段落）切开，并在每一小块前面补上所属的标题。"""
    chunks = []
    for section in split_by_heading(text):
        if len(section) <= max_size:
            chunks.append(section)
            continue
        title = section.splitlines()[0] if section.startswith("#") else ""
        current = ""
        for para in section.split("\n\n"):
            if current and len(current) + len(para) > max_size:
                chunks.append(current.strip())
                current = title + "\n\n" if title else ""
            current += para + "\n\n"
        if current.strip():
            chunks.append(current.strip())
    return chunks

Comparing them

Statistics for the four methods over the entire httpx documentation (full code in code/04-rag/chunking.py; it makes no API calls, so your run should give exactly the same numbers):

固定长度：179 块，平均 739 字符，最短 12，最长 800
按标题（有问题）：236 块，平均 493 字符，最短 8，最长 5530
按标题（修正）：176 块，平均 662 字符，最短 8，最长 5530
按标题+限长：196 块，平均 596 字符，最短 8，最长 2193

With the cap, the longest chunk drops from 5,530 to 2,193. That's still over the 1,500 cap, because this chunk contains a long code block with no blank lines, and my code only cuts at blank lines, never inside a code block. That's a deliberate trade-off: better a slightly long chunk than code cut in half.

The shortest chunk is only 8 characters: sections that have a heading and no content. They are almost worthless for retrieval; in a real project you can merge them into the next chunk or just drop them.

The remaining lessons all use the "by heading + cap" method.

How big should a chunk be

There's no standard answer, but a few guidelines:

• Don't exceed the embedding model's limit. bge-small-zh and multilingual-e5-small both take 512 tokens. In English, one token is roughly 4 characters, so 1,500 characters is about 400 tokens, within the limit.
• Ideally one chunk covers one thing. Splitting by the document's own structure achieves this more easily than splitting by length.
• Think about how the chunks will be used after retrieval. If you put 5 chunks into the prompt at a time and each is 600 characters, that's 3,000 characters, about 750 tokens, not much. At 5,000 characters per chunk, 5 chunks is over ten thousand tokens.

The final chunk size should be decided by the evaluation in Lesson 6: run the evaluation set once for each of several sizes and see which retrieves best.

Do you need overlap

With fixed-length chunking, overlap keeps a sentence cut in two from being incomplete on both sides. With structure-based chunking, boundaries already fall on paragraphs or headings, so overlap usually isn't needed.

Overlap has a cost too: the same content appears in two chunks, and both may be retrieved together, taking up valuable slots.

Exercises

1. Change split_fixed's size to 300 and 2000 and see what timeouts.md becomes. Which do you think works better?
2. Modify split_by_heading_capped to merge chunks that have only a heading and no content (say, shorter than 50 characters) into the chunk that follows.
3. Take a document of your own (a project README, a wiki export from work, text converted from a PDF), split it with all three methods, and look through the chunks for anything that was cut badly.

Self-check

1. What's the biggest problem with fixed-length chunking?

It ignores the document's structure, often cutting in the middle of a sentence, a piece of code or a URL, and a single chunk can mix two unrelated subsections. Such chunks are incomplete and not about one thing, which hurts both retrieval and answers.

2. When splitting by Markdown heading, why do code blocks need special handling?

Python comments inside code blocks start with # and a space, exactly like a Markdown heading. Without special handling, comments are treated as headings, code blocks are cut in the middle, and the explanation and code examples land in different chunks.

3. When capping chunk length, why put the section heading in front of each smaller piece?

Once a long section is split into several pieces, the later pieces may contain only body text with no indication of what they're about. Adding the heading gives every piece its topic, so it's easier to match correctly at retrieval, and when it goes into the prompt the model knows its context.