Tokens: text as the model sees it

Large language models charge by the token, measure context windows in tokens, and set output limits in tokens. In the previous lesson you saw prompt_tokens and completion_tokens in usage. But what is a token, exactly? Is one Chinese character one token? What about an English word?

This isn't only about the bill. Models have some odd failings, such as sometimes being unable to count the characters in a sentence, and those also come down to tokens. In this lesson we cut text apart and look.

Models can't read text, only numbers

A neural network can only process numbers. So before text reaches the model, it goes through a program called a tokenizer: it has a fixed vocabulary, cuts the text into pieces that are in the vocabulary, and maps each piece to a number. The model sees a sequence of numbers and outputs numbers too, which the tokenizer then turns back into text.

Those pieces are tokens. A token may be one character, one word, half an English word, a punctuation mark or a few spaces. DeepSeek's vocabulary has 129,280 tokens; the o200k_base vocabulary used by OpenAI's GPT-4o generation has 200,019.

The vocabulary is worked out from a large amount of text before the model is trained: character combinations that often appear together are merged into one token. Lesson 1 of module 09 implements this algorithm from scratch. For now just remember the conclusion: common combinations become a single token; rare ones get broken up.

Cutting some text

DeepSeek publishes its tokenizer, and you can download it to use locally. Download it here; unzipping gives a deepseek_v4_tokenizer directory. Then install two packages:

uv add tokenizers tiktoken

tokenizers loads DeepSeek's tokenizer files, and tiktoken is OpenAI's tokenizer. The script below gives the same few passages to both tokenizers and prints every piece they cut:

import tiktoken
from tokenizers import Tokenizer

deepseek = Tokenizer.from_file("deepseek_v4_tokenizer/tokenizer.json")
openai_enc = tiktoken.get_encoding("o200k_base")  # GPT-4o 等 OpenAI 模型用的分词器

samples = [
    "人工智能正在改变软件开发的方式，越来越多的程序员开始用大模型写代码。",
    "Artificial intelligence is changing how software is built, and more programmers now write code with large language models.",
    "学而时习之，不亦说乎？有朋自远方来，不亦乐乎？",
    "def add(a, b):\n    return a + b\n",
    "3.1415926535897932384626",
    "😀",
]


def deepseek_pieces(text):
    ids = deepseek.encode(text, add_special_tokens=False).ids
    return [deepseek.decode([i]) for i in ids]


def openai_pieces(text):
    pieces = []
    for i in openai_enc.encode(text):
        raw = openai_enc.decode_single_token_bytes(i)
        # 一个汉字或表情可能被拆成几个字节，单独拿出来不是合法的 UTF-8
        pieces.append(raw.decode("utf-8", errors="replace"))
    return pieces


for text in samples:
    ds, oa = deepseek_pieces(text), openai_pieces(text)
    print(f"原文（{len(text)} 个字符）：{text!r}")
    print(f"  DeepSeek {len(ds):3d} 个词元：{ds}")
    print(f"  OpenAI   {len(oa):3d} 个词元：{oa}")
    print()

The output:

原文（34 个字符）：'人工智能正在改变软件开发的方式，越来越多的程序员开始用大模型写代码。'
  DeepSeek  15 个词元：['人工智能', '正在', '改变', '软件开发', '的方式', '，', '越来越多的', '程序员', '开始', '用', '大', '模型', '写', '代码', '。']
  OpenAI    21 个词元：['人工', '智能', '正在', '改变', '软件', '开发', '的', '方式', '，', '越来越', '多', '的', '程序', '员', '开始', '用', '大', '模型', '写', '代码', '。']

原文（122 个字符）：'Artificial intelligence is changing how software is built, and more programmers now write code with large language models.'
  DeepSeek  20 个词元：['Artificial', ' intelligence', ' is', ' changing', ' how', ' software', ' is', ' built', ',', ' and', ' more', ' programmers', ' now', ' write', ' code', ' with', ' large', ' language', ' models', '.']
  OpenAI    20 个词元：['Artificial', ' intelligence', ' is', ' changing', ' how', ' software', ' is', ' built', ',', ' and', ' more', ' programmers', ' now', ' write', ' code', ' with', ' large', ' language', ' models', '.']

原文（23 个字符）：'学而时习之，不亦说乎？有朋自远方来，不亦乐乎？'
  DeepSeek  22 个词元：['学', '而', '时', '习', '之', '，', '不', '亦', '说', '乎', '？', '有', '朋', '自', '远方', '来', '，', '不', '亦', '乐', '乎', '？']
  OpenAI    21 个词元：['学', '而', '时', '习', '之', '，不', '亦', '说', '乎', '？', '有', '朋', '自', '远', '方', '来', '，不', '亦', '乐', '乎', '？']

原文（32 个字符）：'def add(a, b):\n    return a + b\n'
  DeepSeek  12 个词元：['def', ' add', '(a', ',', ' b', '):\n', '   ', ' return', ' a', ' +', ' b', '\n']
  OpenAI    12 个词元：['def', ' add', '(a', ',', ' b', '):\n', '   ', ' return', ' a', ' +', ' b', '\n']

原文（24 个字符）：'3.1415926535897932384626'
  DeepSeek  10 个词元：['3', '.', '141', '592', '653', '589', '793', '238', '462', '6']
  OpenAI    10 个词元：['3', '.', '141', '592', '653', '589', '793', '238', '462', '6']

原文（1 个字符）：'😀'
  DeepSeek   2 个词元：['�', '�']
  OpenAI     1 个词元：['😀']

Several things on that screen deserve a closer look.

Modern Chinese: DeepSeek cuts coarser. "人工智能" (artificial intelligence), "软件开发" (software development) and "越来越多的" (more and more) are each a single token in DeepSeek's vocabulary, so the same sentence takes 15 tokens with DeepSeek and 21 with OpenAI. Vocabularies are built from the training corpus, and DeepSeek's corpus has more Chinese, so more Chinese phrases were merged. For the same Chinese content, fewer tokens means less money and more content fitting into the context at once.

English: the two are identical. Common English words are essentially one token each, and the space belongs to the following word: ' intelligence' has a space in front of it. 122 characters become 20 tokens, about one per word.

Classical Chinese: nearly one token per character. "学而时习之" (to learn and practise it in due time) is five single characters in both vocabularies. Modern Chinese is full of combinations like "学习" (to study) and "时间" (time); combinations such as "时习" and "不亦" are too rare to have been merged. For the same 23 characters, classical Chinese uses twice as many tokens as modern Chinese.

Spaces in code cost money too. Of the four spaces of indentation, three became a token of their own. The deeper Python code is indented, the more tokens the spaces take.

Numbers are cut into groups of three digits. 3.1415926... is cut into chunks like 141, 592 and 653. The model doesn't see a whole number but a few "three-digit fragments". That's one reason models get multi-digit arithmetic wrong: column arithmetic needs digits lined up one by one, and the input isn't digit by digit at all.

Emoji can be split into bytes. DeepSeek cut 😀 into two tokens, and decoded on its own each one is the garbled �, because each token is only some of the bytes of that emoji's encoding; only together do they form the full character.

Estimating token counts for cost

The two accurate ways are to count locally with the official tokenizer, or to look at the usage the API returns.

When neither is convenient, you can estimate roughly. DeepSeek's official docs give this conversion: 1 English character is about 0.3 tokens and 1 Chinese character about 0.6 tokens. I measured a few passages through the API, and the results came out lower than the official figures:

Here's how I measured: set max_tokens=1 and look only at prompt_tokens. First send a message containing only the letter a and note its input token count as the baseline (the overhead of the formatting markers); then send the text to measure. Subtract the two and add one, and you have the tokens of the text itself.

def count_tokens(text):
    r = client.chat.completions.create(
        model="deepseek-flash",
        messages=[{"role": "user", "content": text}],
        max_tokens=1,
        extra_body={"thinking": {"type": "disabled"}},
    )
    return r.usage.prompt_tokens


base = count_tokens("a")  # 格式标记加上字母 a 本身，我测到的是 5
print(count_tokens("学而时习之，不亦说乎？有朋自远方来，不亦乐乎？") - base + 1)

The official figures are on the cautious side, which makes them good for budgeting, since you're unlikely to overspend. But remember that token counts depend on the content: classical Chinese, rare characters, emoji and less common languages are all "pricier" than everyday Chinese.

Why models can't count characters

There's a famous example: ask a model "how many r's are in strawberry" and many answer 2, when the right answer is 3.

Look at how the tokenizer handles the word: DeepSeek cuts a standalone strawberry into three chunks, st, raw and berry, while strawberry with a leading space (which is how it appears in the middle of a sentence) is one whole token. The model receives a single number (79430); it doesn't directly "see" the letters inside. It knows how many r's the word has only from similar discussions it met during training.

That example is so famous, though, that most current models have seen it. I had deepseek-flash answer several versions of the question, asking each three times with thinking off and three times with it on:

Asked in English, the old question is now answered correctly every time. Ask it in Chinese, use a different word, or switch to counting Chinese characters, and with thinking off it occasionally goes wrong. With thinking on everything is right, because in its thinking the model writes the word out letter by letter before counting, which amounts to splitting the token apart itself.

In real development, anything that needs an exact count, such as word counts, string lengths or truncating text, shouldn't be left to the model; do it in code. len("秋天的叶子一片片落下") is always 10.

Common questions

Can token counts from different models be compared directly? No. For the same Chinese passage, DeepSeek's and OpenAI's token counts differ by about 30%. To compare two models' prices, take the same piece of real text from your use case, work out its token count for each, and multiply by each one's unit price; comparing unit prices alone isn't enough.

Can I count DeepSeek tokens in my code with tiktoken? For English and code it's about the same; for Chinese it overcounts. For an accurate count, use DeepSeek's own tokenizer.

Exercises

1. Give your name, the name of your city and a sentence in a dialect or another language to the two tokenizers in tokens.py, and see how each one cuts them.
2. Measure with count_tokens: the same content written in Chinese and in English, which uses more tokens? Try it with a real passage from your work.
3. Take some JSON data (say, an API response) and count its tokens. Then compress it to one line with all whitespace removed and count again. How many fewer tokens? This is useful when you need to feed a lot of data to a model.

Self-check

1. DeepSeek uses fewer tokens than OpenAI's tokenizer for the same modern Chinese sentence. Why, and what does that mean for you?

A tokenizer's vocabulary is worked out from its training corpus. DeepSeek's corpus has a higher share of Chinese, so more Chinese phrases were merged into single tokens. For you it means the same Chinese content costs less on DeepSeek, and more of it fits into the context at once. When comparing model prices, work out the token counts on real text and compare those.

2. Why do models easily miscount how many times a letter appears in a word?

The model sees token numbers, not letters. A common word is often a single token, so the model can't directly "see" the letters in it and has to answer from what it remembers from training. With thinking on, it writes the word out letter by letter before counting, and accuracy goes way up. When you need an exact count, use code, not the model.

3. Why is classical Chinese "pricier" than the same number of characters of modern Chinese?

The vocabulary merges character combinations that are common in the training corpus. Everyday modern Chinese words ("人工智能" for artificial intelligence, "程序员" for programmer) appear often and were merged into single tokens; the combinations in classical Chinese ("时习", "不亦") are rare and weren't merged, so it goes one token per character. More tokens means a higher bill.