tiktoken-rs: OpenAI token counting inside Rust, with a vendored BPE core

What tiktoken-rs actually solves for Rust callers

Token counts drive two practical decisions in an OpenAI-backed application: whether a prompt fits the model's context window, and how large the completion budget can be. The README frames the library around exactly that, describing use cases that cover tokenizing and counting tokens in text inputs. The crate is a Rust binding layer over the tiktoken library, with what the README calls additional features and enhancements for ease of use with Rust code.

The audience is narrow by design. The README states the scope plainly: this crate is focused on OpenAI tokenizers, and non-OpenAI models such as Llama, Gemini and Mistral should use the HuggingFace tokenizers crate instead. That is a useful boundary. If your service routes between OpenAI and a self-hosted model, you will end up with two tokenizer dependencies and two sets of counts, and tiktoken-rs will only cover one side of that split.

The supported model list in the README runs from the GPT-5 series and the o1, o3 and o4-mini reasoning models through gpt-4o, gpt-4.1 and the gpt-oss models. Encoding coverage is what makes that possible: o200k_harmony for gpt-oss-20b and gpt-oss-120b, o200k_base for the GPT-5 and o-series families, cl100k_base for gpt-4 and the text-embedding models, and the older p50k_base, p50k_edit and r50k_base encodings for legacy completion and edit models.

The encoding mechanism and the vendored upstream core

The library exposes byte-pair encoding tokenizers under names that match the encoding identifiers: o200k_base, cl100k_base and the rest. Calling one of those functions returns a CoreBPE value, and CoreBPE::encode_with_special_tokens takes a string and returns the token vector, so tokens.len() is the count. Special tokens such as <|endoftext|> are handled by that method rather than being split into ordinary pieces.

There is a second path. CoreBPE::encode mirrors upstream tiktoken and returns a Result, along with a last_piece_token_len value. The README shows it being called with an explicit special-token set obtained from bpe.special_tokens(), which means the caller decides which special tokens are permitted in the input rather than accepting a default. The generic encode_as and count helpers also return Result. That change is worth noting because it is a breaking one for callers written against an earlier API: code that previously treated encode as infallible now has to propagate or unwrap.

Underneath, the repository has a vendor directory and a .gitmodules entry at the top level, and the README says the Rust 1.85 minimum comes from the vendored upstream code rather than from a direct dependency MSRV. So the version floor is a consequence of how the upstream BPE implementation is carried, not of an independent crate in the dependency tree. If you are pinned to an older toolchain, this is the constraint that blocks you, and there is no feature flag mentioned that would lift it.

Installing tiktoken-rs and counting your first prompt

The README gives a single install step through Cargo. Running this adds the dependency to your manifest at the current published version:

cargo add tiktoken-rs

The first real use is a token count. The README's example loads the o200k_base encoder, encodes a sentence with special tokens enabled, and prints the length:

use tiktoken_rs::o200k_base;

let bpe = o200k_base().unwrap();
let tokens = bpe.encode_with_special_tokens(
  "This is a sentence   with spaces"
);
println!("Token count: {}", tokens.len());

The unwrap is doing real work. Constructing the encoder can fail, so the function returns a Result, and the README's example resolves it immediately. In a request path you would want to handle that error rather than panic. Note also that the sample string contains repeated spaces, which is a reminder that the encoder sees the exact bytes you pass, not a normalized version.

For repeated calls, the README recommends the singleton to avoid re-initializing the tokenizer:

use tiktoken_rs::o200k_base_singleton;

let bpe = o200k_base_singleton();
let tokens = bpe.encode_with_special_tokens(
  "This is a sentence   with spaces"
);
println!("Token count: {}", tokens.len());

The singleton call has no unwrap in the README's example, which is the visible difference from the constructor form. If you are counting tokens per request in a long-running service, the singleton variants are the ones the README points you toward.

Budgeting a chat request with get_chat_completion_max_tokens

Counting a raw string is only half the problem. A chat completion request carries per-message overhead that a plain string count does not capture, and that overhead differs by model. tiktoken-rs ships get_chat_completion_max_tokens, which takes a model name and a vector of ChatCompletionRequestMessage values and returns the remaining completion budget as a Result.

The README's example builds a three-message vector with system and user roles and calls the helper for o1-mini:

use tiktoken_rs::{get_chat_completion_max_tokens, ChatCompletionRequestMessage};

let messages = vec![
    ChatCompletionRequestMessage {
        content: Some("You are a helpful assistant that only speaks French.".to_string()),
        role: "system".to_string(),
        ..Default::default()
    },
];
let max_tokens = get_chat_completion_max_tokens("o1-mini", &messages).unwrap();
println!("max_tokens: {}", max_tokens);

There is a second variant for async-openai users, which requires enabling the async-openai feature in Cargo.toml and imports from tiktoken_rs::async_openai. That version takes async_openai::types::chat message types instead of the crate's own struct, so the two helpers are not interchangeable in a codebase that already standardized on one set of message types.

The context-size table is the other half of the budgeting picture, and it is worth reading before you trust a computed number. gpt-5.4 and gpt-5.4-pro are listed at 1,050,000 tokens, gpt-4.1 at 1,047,576, the gpt-5 family at 400,000, the o1 and o3 families at 200,000, gpt-4o at 128,000, gpt-3.5-turbo at 16,385 and gpt-4 at 8,192. Those figures come from the README's table, and they are the ceiling the helper is implicitly working against.

Where tiktoken-rs is the wrong dependency

The scope note is the clearest limitation in the README, and it is not a small one. If your application talks to Gemini, Llama or Mistral, this crate does not tokenize those models, and the README says to use the HuggingFace tokenizers crate instead. A multi-provider gateway therefore cannot rely on tiktoken-rs alone for its accounting.

The Rust 1.85 floor is the second constraint. The README attributes it to the vendored upstream code, and it applies to the crate as a whole rather than to an optional feature. Projects on an older toolchain cannot simply avoid the affected code path.

The third issue is API churn visible in the release history. The README has a section titled "Upgrading `encode` calls" that explains encode now returns a Result, and that encode_as and count do the same. Between v0.10.0 and v0.11.0, both released on 2026-04-08, and then v0.12.0 on 2026-06-02, the crate moved through three releases in roughly two months. That cadence is fine for a library tracking upstream OpenAI encodings, but it means the migration burden lands on you whenever upstream changes shape.

Finally, the README does not document rollback, deprecation windows or a compatibility policy for the Result-returning signatures. It tells you the signatures changed and shows the new form. If you need a documented deprecation path before upgrading a production service, that information is not in the README.

tiktoken-rs against the Python, JavaScript and HuggingFace options

The obvious comparison is the Python tiktoken package, which is the upstream this crate wraps. The difference is not in the tokenization result but in what you get around it: tiktoken-rs adds the chat max_tokens helper, the async-openai integration behind a feature flag, and the singleton accessors for repeated encoding in Rust. If you are already in Python, there is no reason to reach for the Rust crate; the README positions it as a Rust library built on top of tiktoken, not as a replacement for it.

Against js-tiktoken, the split is the same shape in a different runtime. Both are ports that bring the same encodings to a non-Python language, and the choice follows your service language rather than any tokenizer difference.

Against HuggingFace tokenizers, the difference is categorical rather than incremental. tokenizers is a general tokenization framework that loads model-specific tokenizer definitions, which is why the README points Llama, Gemini and Mistral users at it. tiktoken-rs ships fixed encodings tied to OpenAI model families and a context-size table to go with them. If you need one abstraction over many model families, tokenizers is the fit. If you need to know whether a specific OpenAI prompt fits, tiktoken-rs is the narrower and more direct tool, and the narrowness is the point.

Licence, maintenance and the cost of keeping up

The repository is MIT licensed, with a LICENSE file at the top level. For most Rust projects that is a permissive fit and requires only that the copyright notice and permission notice travel with copies or substantial portions of the software. This is not legal advice, and if you redistribute the crate in a product with its own compliance review, the vendored upstream code under vendor/ is the part worth confirming with your own counsel, since vendoring is how the upstream BPE implementation is carried.

The repository is not archived, and the last push was on 2026-07-01. The most recent release listed is v0.12.0 on 2026-06-02, preceded by v0.11.0 and v0.10.0 on 2026-04-08. The README does not state a support window for older releases, so an upgrade plan has to be built from the release notes rather than from a published policy.

The practical upgrade cost concentrates in two places. The first is the toolchain: the Rust 1.85 floor comes from vendored code, so it moves when upstream moves. The second is the Result-returning signatures on encode, encode_as and count. Any code that calls those will need a compile-time fix on upgrade, and the README's upgrading section is the reference for the new form. Because the crate tracks OpenAI encodings, staying current also means staying aligned with the encoding table when new model families appear.

Editorial conclusion

Adopt tiktoken-rs if you are writing Rust and need local token counts for OpenAI models, or if you want the chat max_tokens helper without reimplementing message overhead arithmetic. Do not adopt it for Llama, Gemini or Mistral tokenization, since the README directs those users to the HuggingFace tokenizers crate, and do not treat it as a client for any provider API. Before you commit, check that your toolchain is Rust 1.85 or newer, confirm the encoding name for the model you call against the encoding table, and decide whether you want the per-call constructors or the singleton variants for repeated encoding.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. zurawiki/tiktoken-rs on GitHub