# text-splitter: cutting documents into chunks that respect sentence and code boundaries

> The text-splitter crate splits long text into chunks sized by characters or tokens, preferring to break at semantic boundaries such as Unicode grapheme clusters, word and sentence breaks, Markdown structure and tree-sitter parse nodes. It suits a team building a retrieval pipeline over documents, and its token-aware sizing needs a tokenizer you supply.

**benbrandt/text-splitter** — Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.

- Repository: https://github.com/benbrandt/text-splitter
- Stars: 636 · Forks: 33
- Language: Rust
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/benbrandt-text-splitter

## The problem: documents that do not fit in a context window

The README states the problem in its first technical paragraph. Large language models can be used for many tasks, but often have a limited context size that can be smaller than documents you might want to use. To use documents of a longer length, you often have to split your text into chunks to fit within that context size.

That is the whole reason this crate exists, and it is worth naming the difficulty that is easy to underestimate. Splitting a document is trivial; splitting it well is not. A naive split at a fixed character count cuts sentences in half, separates a heading from the paragraph it introduces, and breaks a function signature from its body. Every one of those chunks is worse for retrieval, worse for a human reading the output, and worse for a model asked to answer a question about it.

The crate's stated approach is to provide methods for splitting longer pieces of text into smaller chunks, aiming to maximize a desired chunk size, but still splitting at semantically sensible boundaries whenever possible.

So the audience is a team with a document pipeline and a model with a context limit. That is a very common situation now, and the interesting part of this project is how it decides where the sensible boundaries are.

The package is published on crates.io as text-splitter with docs on docs.rs, licensed under MIT with the file at the repository root, and the last push was on 2026-10-22 with the repository not archived. The release history is steady: v0.31.0 on 2026-06-04, v0.32.0 on 2026-06-16 and v0.33.0 on 2026-09-24, with the workspace version in the manifest matching the last tag.

## The core idea: pick the highest semantic level that still fits

The Method section of the README explains the algorithm, and it is the part to understand before trusting the output.

The goal is to preserve as much semantic meaning within a chunk as possible, and each chunk is composed of the largest semantic units that can fit in the next given chunk. Each splitter type defines a set of semantic levels, and the steps are the same for all of them.

First, split the text by increasing semantic levels. Then check the first item for each level and select the highest level whose first item still fits within the chunk size. Finally, merge as many of these neighbouring sections of that level or above into a chunk, to maximise chunk length. The README notes that boundaries of higher semantic levels are always included when merging, so the chunk does not inadvertently cross semantic boundaries.

That third step is where the quality comes from, and it is also the part with a subtlety. The selection is made on the first item of each level, not on every item. If the first sentence of a paragraph does not fit, the paragraph level is discarded in favour of the sentence level, and the merging then works at sentence granularity for the whole text. The text is not split into a mix of levels.

For the plain TextSplitter the levels in ascending order are: characters; Unicode grapheme cluster boundaries; Unicode word boundaries; Unicode sentence boundaries; and ascending sequence length of newlines. The newline rule is the clever one. A newline is a carriage return and line feed pair, a line feed, or a carriage return, and each unique length of consecutive newline sequences is treated as its own semantic level, so a sequence of two newlines is a higher level than a sequence of one.

That is how paragraphs and sections become distinguishable from soft line breaks, without the library needing to know anything about the document format. And the README is explicit about the floor: splitting never occurs below the character level, otherwise you could get partial bytes of a character, which may not be a valid unicode string.

## Sizing by character, by range, or by a tokenizer you supply

The simplest use is the default implementation, which uses character count for the chunk size. The whole API is four lines:

```rust
use text_splitter::TextSplitter;

// Maximum number of characters in a chunk
let max_characters = 1000;
// Default implementation uses character count for chunk size
let splitter = TextSplitter::new(max_characters);

let chunks = splitter.chunks("your document text");
```

The capacity can also be a range, which is the feature most people miss. You specify a range and the chunk is filled until its length falls within that range:

```rust
use text_splitter::{ChunkConfig, TextSplitter};

// Maximum number of characters in a chunk. Will fill up the
// chunk until it is somewhere in this range.
let max_characters = 500..2000;
// Default implementation uses character count for chunk size
let splitter = TextSplitter::new(max_characters);

let chunks = splitter.chunks("your document text");
```

The README is careful about what a range means. Once a chunk has reached a length that falls within the range it will be returned, but it is always possible that a chunk may be returned that is less than the start value, since adding the next piece of text may have made it larger than the end capacity.

So a range is a target, not a guarantee, and the reason is structural: the algorithm merges whole semantic units, and a unit that pushes past the end cannot be split to fit.

The third sizer is a tokenizer, and it is where the crate earns its keep for real model pipelines. A chunk of 1000 characters is a different amount of context depending on the language and the model, and a chunk of 1000 tokens is what a context window actually counts.

The design here is that the tokenizer implements a sizer trait from the crate, so the splitter does not care which tokenizer it is. The README comments note that anything else implementing the ChunkSizer trait works too. The token-based examples come next.

## Token sizing with Hugging Face and Tiktoken, and the truncation trap

Two tokenizer integrations are documented, and each needs a feature flag and a dependency.

For a Hugging Face tokenizer you add the tokenizers feature and the tokenizers crate, and the example using from_pretrained also needs the http feature:

```sh
cargo add text-splitter --features tokenizers
cargo add tokenizers --features http
```

The code loads a pretrained tokenizer and passes it as the sizer:

```rust
use text_splitter::{ChunkConfig, TextSplitter};
// Can also use anything else that implements the ChunkSizer
// trait from the text_splitter crate.
use tokenizers::Tokenizer;

let mut tokenizer = Tokenizer::from_pretrained("bert-base-cased", None).unwrap();
// If your tokenizer has truncation enabled, disable it before passing it to
// the splitter. Otherwise chunk sizes can be capped by the tokenizer's
// truncation limit.
tokenizer.with_truncation(None).unwrap();

let max_tokens = 1000;
let splitter = TextSplitter::new(ChunkConfig::new(max_tokens).with_sizer(tokenizer));

let chunks = splitter.chunks("your document text");
```

The truncation comment is the single most important line in that example. A tokenizer configured with truncation will silently cap the length of the text it reports, so every chunk looks like it fits while the actual content is cut at the tokenizer's limit. Disabling truncation is what makes the chunk sizes mean what the caller asked for. If you take one thing from this library, take that.

The second integration is tiktoken-rs, with its own feature flag and dependency, and the same shape of code:

```sh
cargo add text-splitter --features tiktoken-rs
cargo add tiktoken-rs
```

```rust
use text_splitter::{ChunkConfig, TextSplitter};
// Can also use anything else that implements the ChunkSizer
// trait from the text_splitter crate.
use tiktoken_rs::cl100k_base;

let tokenizer = cl100k_base().unwrap();
let max_tokens = 1000;
let splitter = TextSplitter::new(ChunkConfig::new(max_tokens).with_sizer(tokenizer));

let chunks = splitter.chunks("your document text");
```

Choosing the right tokenizer for the model you are actually targeting is the decision that makes the token counts meaningful.

## Markdown and code get their own splitters with the same interface

The same chunking API is available for two structured formats, each behind a feature flag.

For Markdown, enabling the markdown feature gives you a MarkdownSplitter used in the same ways as the TextSplitter:

```sh
cargo add text-splitter --features markdown
```

```rust
use text_splitter::MarkdownSplitter;
// Maximum number of characters in a chunk. Can also use a range.
let max_characters = 1000;
// Default implementation uses character count for chunk size.
// Can also use all of the same tokenizer implementations as `TextSplitter`.
let splitter = MarkdownSplitter::new(max_characters);

let chunks = splitter.chunks("# Header\n\nyour document text");
```

For code, the code feature enables a CodeSplitter that works on anything parsable with tree-sitter, and you add the tree-sitter parser for your language:

```sh
cargo add text-splitter --features code
cargo add tree-sitter-<language>
```

```rust
use text_splitter::CodeSplitter;
// Maximum number of characters in a chunk. Can also use a range.
let max_characters = 1000;
// Default implementation uses character count for chunk size.
// Can also use all of the same tokenizer implementations as `TextSplitter`.
let splitter = CodeSplitter::new(tree_sitter_rust::LANGUAGE, max_characters).expect("Invalid tree-sitter language");

let chunks = splitter.chunks("your code file");
```

The pattern is consistent across all three: the same capacity semantics, the same tokenizer support, the same chunks method, with a different notion of what a semantic boundary is. The MarkdownSplitter has its own set of semantic levels for document structure, and the CodeSplitter derives them from the tree-sitter parse, so a chunk boundary lands between top-level items rather than in the middle of a function.

Note that the code splitter takes a language constant rather than a parser instance, and it returns a Result, so a wrong language constant is a runtime error rather than a compile error.

The repository also carries a docs directory, a benches directory, a bindings directory, a conf.py for the documentation build, a clippy.toml, and a large tests directory whose snapshot and input files are excluded from the published package because they are large. The workspace lints are strict, with the clippy pedantic group enabled and missing documentation treated as a warning, which is a decent signal of care for a library whose output other systems depend on.

## Where text-splitter is the wrong tool

The first wrong-tool case is a pipeline that needs to put chunks back in their source context. This crate returns text. It does not return offsets, character positions or node identifiers, so if you are building a retrieval system where each chunk has to be traceable back to a page, a paragraph or a line in the original document, you will need to solve that separately, and the alignment between chunks and source positions is exactly the kind of thing that goes wrong silently.

The second is a project that needs fixed-size chunks. The algorithm maximises chunk size and prefers semantic boundaries, and the README is explicit that a chunk can come out shorter than the start of your range because the next semantic unit would have overshot the end. If your pipeline requires every chunk to be exactly N tokens, for example because of a strict batching or truncation step downstream, this library gives you chunks near N, not chunks of N.

The third is a setup where you cannot depend on a tokenizer or on a tree-sitter grammar. Every token-based feature is a flag and an extra dependency, and the code feature needs a parser for the specific language. If your document types are unusual, or your environment forbids native parser builds, you are left with character counting, which is a reasonable default but not what you wanted.

The fourth is a use of the Python binding. The bindings exist and are published as semantic-text-splitter on PyPI, but the README notes that the same package name could not be acquired, so the Python distribution is called something different from the Rust crate. A team that has standardised on one package name across languages will trip over that.

The alternative worth weighing is doing it yourself with a plain split on a character count, which is a few lines and has no dependencies. The difference in approach is that this library spends effort on where to break rather than on how to count, and that effort is only worth paying if your downstream task actually benefits from semantically intact chunks, which is the case for retrieval and question answering, and much less clear for a summarisation pipeline that will compress everything anyway.

## Conclusion

text-splitter fits a Rust or Python team that needs to feed documents into a model with a limited context window and wants chunks that break at sentences, Markdown structure or syntax tree nodes rather than mid-word. It is the wrong choice if you need embeddings placed back in their source context, since it returns text chunks only, and it is the wrong choice if you cannot supply or depend on a tokenizer, since the token-based sizing is a feature flag away. Before adopting it, decide which sizer matches the model you are targeting, and if you use a Hugging Face tokenizer, disable its truncation as the README example does, or your chunk sizes will be silently capped.

## FAQ

### What does the text-splitter crate do?

It splits longer pieces of text into smaller chunks, aiming to maximise a desired chunk size while splitting at semantically sensible boundaries whenever possible, and it can size chunks by characters or by tokens.

### How do I add text-splitter to a Rust project?

Run cargo add text-splitter, then create a TextSplitter with the maximum chunk size and call its chunks method on your document text.

### Which tokenizers does text-splitter support?

It documents Hugging Face tokenizers through the tokenizers feature and tiktoken through the tiktoken-rs feature, and the splitter accepts anything implementing the ChunkSizer trait.

### Why does the text-splitter Hugging Face example disable truncation?

The README warns that if your tokenizer has truncation enabled, chunk sizes can be capped by the tokenizer's truncation limit, so the example disables it with tokenizer.with_truncation(None).

## Sources

- [benbrandt/text-splitter on GitHub](https://github.com/benbrandt/text-splitter)
- [Issues](https://github.com/benbrandt/text-splitter/issues)
- [License: MIT](https://github.com/benbrandt/text-splitter/blob/main/LICENSE)
- [README](https://github.com/benbrandt/text-splitter/blob/main/README.md)
- [Releases](https://github.com/benbrandt/text-splitter/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/benbrandt-text-splitter