Writing a BPE tokenizer by hand

As Module 01, Lesson 1 explained, a model doesn't understand text; what it sees is a sequence of token ids. This module builds a GPT from scratch, and the first step is deciding how to cut text into tokens.

This lesson writes by hand the method used by the GPT series, DeepSeek and Qwen alike: BPE (Byte Pair Encoding). Once it's written, you'll understand two things Module 01 could only ask you to take on trust: why the same passage has a different number of tokens in different models, and why a rare character gets split into several tokens.

Preparing the data

The whole module uses Tang poetry as its data. It comes from the open-source project chinese-poetry (MIT licence), which contains more than fifty thousand poems from the Complete Tang Poems.

uv add torch opencc
python prepare_poems.py

The raw data is in traditional characters, and the script converts it to simplified with OpenCC. OpenCC converts by phrase, which is more accurate than replacing one character at a time. Then only poems in strict regulated form are kept: five or seven characters per line, four lines (a quatrain) or eight (a regulated verse), containing only Chinese characters, commas and full stops.

《全唐诗》共 57607 首，留下格律整齐的 35135 首，共 1557776 个字符，6288 个不同的字符
  五言绝句：3656 首
  七言绝句：10087 首
  五言律诗：13865 首
  七言律诗：7527 首
前三首：
  闲却白云居，行踪出去初。窗中聊取笔，架上独留书。日背林光冷，潭澄岳影虚。长闻得药力，此说复何如。
  秋溪南岸菊霏霏，急管烦弦对落晖。红叶树深山径断，碧云江静浦帆稀。不堪孙盛嘲时笑，愿送王弘醉夜归。流落正怜芳意在，砧声徒促授寒衣。
  东城晓出静尘埃，紫画神旗向日开。锦袖半攘争捧辔，银鞍不下小传杯。马盘草上朱弓满，鴈落云中白羽回。晚向三通残皷尽，北原千骑卷行来。

One poem per line: 35,000 poems and 1.56 million characters. Look closely at the third poem: "鴈" and "皷" weren't converted to "雁" and "鼓". They're variant forms, not traditional/simplified pairs, so OpenCC's traditional-to-simplified conversion leaves them alone. Data cleaning is rarely 100% clean; this has little effect on training later, so it stays.

Starting from bytes

At the lowest level, text in a computer is bytes. In UTF-8, an English letter is one byte and a Chinese character is three:

一个汉字在 UTF-8 里是 3 个字节：'月' → e6 9c 88

BPE starts from bytes, so its initial vocabulary has only 256 entries (the values 0–255 one byte can represent). The benefit is that any text can be represented, and "this character isn't in the vocabulary" never happens; at worst it's split into bytes.

Then it does one thing over and over: find the pair of adjacent tokens that occurs most often in the training data, and merge it into a new token.

def train(text, n_merges):
    ids = list(text.encode("utf-8"))  # 从字节开始：词表一开始就是 0～255 这 256 个字节
    merges = {}  # (a, b) -> 新编号
    vocab = {i: bytes([i]) for i in range(256)}  # 编号 -> 它代表的字节串
    for k in range(n_merges):
        pairs = count_pairs(ids)
        pair, count = pairs.most_common(1)[0]  # 出现最多的一对相邻的词元
        new_id = 256 + k
        ids = merge(ids, pair, new_id)
        merges[pair] = new_id
        vocab[new_id] = vocab[pair[0]] + vocab[pair[1]]

count_pairs counts every adjacent pair with a Counter, and merge replaces every occurrence of the pair with the new id; each is only a few lines (see code/09-transformer/bpe.py).

Each merge adds one token to the vocabulary and makes the training data a little shorter. The number of merges determines the final vocabulary size, and is a number you choose.

What it learned

Train 1,500 merges on 2,000 poems (pure Python is slow; with more data you'd be waiting a long time):

训练数据：2000 首诗，89863 个字符，265591 个字节

第    1 次合并：          [80] + [82]           → [80 82]         （出现 6396 次），训练数据变成 259195 个词元
第    2 次合并：          [ef] + [bc]           → [ef bc]         （出现 6394 次），训练数据变成 252801 个词元
第    3 次合并：       [ef bc] + [8c]           → ，               （出现 6394 次），训练数据变成 246407 个词元
第    4 次合并：          [e3] + [80 82]        → 。               （出现 6394 次），训练数据变成 240013 个词元
第    5 次合并：          [e4] + [b8]           → [e4 b8]         （出现 3702 次），训练数据变成 236311 个词元
第    6 次合并：             。 + ↵              → 。↵              （出现 1999 次），训练数据变成 234312 个词元
第    7 次合并：          [e4] + [ba]           → [e4 ba]         （出现 1819 次），训练数据变成 232493 个词元
第    8 次合并：          [e5] + [a4]           → [e5 a4]         （出现 1749 次），训练数据变成 230744 个词元
……
第 1001 次合并：[ef bc 8c e8 a1] + [8c]           → ，行              （出现 26 次），训练数据变成 106108 个词元
第 1500 次合并：       [e7 a5] + [96]           → 祖               （出现 15 次），训练数据变成 96124 个词元
训练 1500 次合并，用了 27 秒，词表大小 1756

What's in square brackets are bytes that don't yet form a complete character. The first few merges are interesting:

• The first 4 merges assemble the comma and the full stop. Every poem has them, so they occur most. Merges 1 and 4 together first join the last two bytes of the full stop, 80 82, then add the leading e3.
• Merge 6 combines "full stop + newline" into one token, because every poem ends with a full stop followed by a newline.
• Merges 5, 7 and 8 combine the first two bytes of Chinese characters' UTF-8 encoding. Many common characters share their first two bytes; those beginning e4 b8, for example, include "不", "与", "世", "东" and more. So these "half characters" occur more often than any single complete character.

After 1,500 merges, the training data shrank from 266,000 bytes to 96,000 tokens, still slightly more than its 89,900 characters.

1500 个新词元里：883 个正好是一个完整的字符，172 个是两个字符以上，其余 445 个是半个汉字、或者跨了字的边界
两个字符以上的，最早学到的 30 个：
  。↵ ，不 ，一 。不 ，何 ，山 人。↵ ，风 万里 ，春 ，应 ，天 ，江 。何 ，无 千里 何处 ，清 ，白 。自 。莫 人间 ，秋 ，寒 ，月 。↵一 ，相 。一 ，云 ，日

It learned a few real words: "万里" (ten thousand li), "千里" (a thousand li), "何处" (where), "人间" (the human world). But more of them are "comma + one character", such as "，不", "，何", "，春". That's because in Tang poetry the first character of every line is always preceded by punctuation, so these combinations are very frequent. BPE only looks at frequency and knows nothing about language; it doesn't know the punctuation mark has nothing to do with the character after it.

Tokenizers used in practice first split the text roughly with rules before training, for example cutting punctuation, spaces and digits apart, and BPE merges only within the resulting pieces. GPT-2 uses a regular expression for this. That way it never learns tokens like "，不" that straddle punctuation.

Encoding and decoding

What training produces is a table of merge rules. To encode new text, first convert it to bytes, then apply the merges in the order they were learned:

def encode(text, merges):
    ids = list(text.encode("utf-8"))
    while len(ids) >= 2:
        # 在所有相邻的对里，找最早学到的那个合并规则先用上：和训练时的顺序一致
        pair = min(set(zip(ids, ids[1:])), key=lambda p: merges.get(p, float("inf")))
        if pair not in merges:
            break
        ids = merge(ids, pair, merges[pair])
    return ids


def decode(ids, vocab):
    return b"".join(vocab[i] for i in ids).decode("utf-8", errors="replace")

Why in order? Because later merge rules build on earlier ones. The token "，行", for example, is made from "the comma's first two bytes plus the first two bytes of '行'" plus 8c; if those earlier merges aren't done first, the later ones have nothing to merge.

Decoding is much simpler: join the bytes each token stands for and decode them as UTF-8.

On poems it hasn't seen

Test on 500 poems not seen in training:

在训练时没见过的 500 首诗上：
  68179 个字节，23059 个字符，BPE 分成 25078 个词元，平均每个词元 0.92 个字符
  训练用的 2000 首诗里有 3655 个不同的字符；测试诗里的字符，1.0% 在训练数据里一次都没出现过

例子：自君入城市，北邙无新坟。始信壶中药，不落白杨根。如何忽告归，蕣华还笑人。玉笙无遗音，怅望缑岭云。
  切成：自 | 君 | 入 | 城 | [e5 b8] | [82] | [ef bc 8c e5] | [8c] | [97] | [e9 82] | [99] | 无 | 新 | [e5 9d] | [9f] | [e3 80 82 e5] | [a7 8b] | 信 | [e5 a3] | [b6] | 中 | 药 | ，不 | 落 | 白 | 杨 | 根 | 。如 | 何 | 忽 | [e5 91] | [8a] | 归 | [ef bc 8c e8] | [95] | [a3] | 华 | 还 | 笑 | 人 | 。玉 | [e7 ac] | [99] | 无 | 遗 | 音 | [ef bc 8c e6 80] | [85] | 望 | [e7 bc] | [91] | 岭 | 云 | 。
  解码回去和原文完全一样

The result isn't pretty: more tokens than characters, an average of only 0.92 characters per token. Characters like "市", "北", "邙", "坟" and "始" were all split into two or three fragments.

The reason is too little data. The 2,000 poems contain only 3,655 distinct characters, and 1,500 merges assembled only 883 of them into complete characters. A character that appears only a few times never gets its turn to be merged. "市" (market) is a common character, but it didn't appear often enough in these 2,000 poems, so it was still split.

It does one thing well, though: decoding gives back exactly the original text. However finely it's split, no information is lost. It handles English too, just falling back to one byte per token:

  一句含英文的：月 | 落 | 乌 | 啼 | 霜 | 满 | 天 | 。 | H | e | l | l | o

Real tokenizers

Real LLM tokenizers are trained the same way, only on several orders of magnitude more data: tens or hundreds of GB of text in all kinds of languages, with over a hundred thousand merges. As of September 2026, most mainstream models' vocabularies are between one hundred thousand and a little over two hundred thousand. With enough data, common Chinese characters are all merged into complete tokens, and common words become single tokens too.

This also explains what we saw in Module 01:

• Different models' tokenizers are trained on different data, so the same passage has a different number of tokens. Models with more Chinese training data use fewer tokens for Chinese.
• Rare characters appear little in the training data and aren't merged, so they're split into several byte-level tokens.

Which tokenizer this module uses

Our GPT doesn't use this BPE; it uses the simplest character-level tokenization: one character is one token.

class CharTokenizer:
    """字符级分词器：一个字符就是一个词元。换行符表示一首诗结束。"""

    def __init__(self, text):
        self.chars = sorted(set(text))
        self.index = {c: i for i, c in enumerate(self.chars)}

    def encode(self, text):
        return [self.index[c] for c in text]

    def decode(self, ids):
        return "".join(self.chars[i] for i in ids)

The reason is what we just saw: on this little data, BPE actually splits more finely than characters do. And the linguistic unit of Tang poetry is primarily the character, so one token per character is natural. All 35,000 poems together contain 6,288 distinct characters, a modest vocabulary, and every character appears many times in training.

The cost of character-level tokenization: it can't handle characters outside its vocabulary. That doesn't affect our experiments, because the vocabulary is built from all the poems. But it can't handle arbitrary text outside the training data, which is why real LLMs all use byte-level BPE.

Exercises

1. Increase the training data from 2,000 poems to 5,000 (it'll be a bit slower). How many characters per token does the test set average now?
2. Before training, split off commas, full stops and newlines, and count and merge only within each line. Which tokens of two or more characters does it learn now?
3. Byte-level BPE starts with a vocabulary of 256 bytes. What would be the advantages and disadvantages of starting from characters instead (with an initial vocabulary of every distinct character in the training data)?

Self-check

1. What does each step of BPE training do, and when does it stop?

It counts how often each pair of adjacent tokens occurs in the training data, merges the most frequent pair into a new token, adds it to the vocabulary and replaces it throughout the data. This repeats until a predetermined number of merges is reached, that is, until the vocabulary is the desired size.

2. Why must encoding apply the merges in the order they were learned?

Later merge rules were learned on the results of earlier merges, and what they merge may be new tokens those earlier merges produced. Out of the original order, later rules find nothing to merge, and the result differs from what training produced.

3. Why did this lesson's BPE produce more tokens than characters on the test set? Why don't real tokenizers have this problem?

The training data was only 2,000 poems, and many characters didn't appear often enough to be merged into complete characters, so they were split into two or three byte-level fragments. Real tokenizers are trained on massive data with over a hundred thousand merges, so common characters and common words all become complete tokens.