huggingface/tokenizers on the road to 1.0: an editable Tokenizer, a compiled encoder, and a Python API still being rebuilt

pip install --pre pulls a release candidate that does not ship every 0.23 feature

The project is midway through a jump from 0.23 to v1.0.0. The Python install line in the README asks pip for pre-releases:

pip install --pre tokenizers

That flag matters. Without it you stay on the 0.23 line, which still receives Python-only releases such as v0.23.3 on 2026-10-09. With it you get the v1.0.0 release candidates, the newest being v1.0.0-rc.3 on 2026-10-19 after rc.2 on 2026-09-21. The maintainers say plainly that the current library does not ship all features and that a blog post about breaking changes is coming. If a CI job or a requirements file carries --pre without anyone noticing, an environment rebuild can quietly move you onto the release candidate and break code that relied on missing APIs.

Tokenizer is the editable config, CompiledTokenizer does the encoding

The v1 Python API separates two objects. Tokenizer holds the tokenizer.json config and can be edited, for example by setting tokenizer.truncation. Calling compile() builds a CompiledTokenizer, and that is the object that encodes and decodes:

>>> from tokenizers import Tokenizer
>>> tokenizer = Tokenizer.from_pretrained("meta-llama/Llama-3.1-8B")   # or .from_file(path)
>>> compiled = tokenizer.compile()
>>> compiled.tokenize("Hello")
['<|begin_of_text|>', 'Hello']

The encode call returns an Encoding with ids, type_ids and attention_mask, and passing a tuple such as ("What is this?", "A pair.") produces type_ids of 0 for the first sequence and 1 for the second. The practical consequence of the split: code ported from 0.23 that calls encode directly on a Tokenizer has to add the compile step, and edits made to the Tokenizer after compiling belong in a fresh compile().

Training, offsets, saving and decode_stream are still missing from Python

The v1 roadmap is explicit about what the Python bindings cannot do yet. Training has to come back, and the maintainers want the performance gains to arrive with it. Offset output is listed as something to bring back for people training. tokenizer.save, which persists a modified or trained tokenizer to disk, is not there. Neither are windowing or stride in encode, decode_stream, the role_to_token override, a batch encoding API with zero-copy views on encoded ids, or a dlpack API for torch, JAX and numpy interop. Composition is half done: the editable Tokenizer spec and compile() have landed, while per-component handles such as assigning tokenizer.normalizer are next. If your pipeline touches any of these, the release candidate is not a drop-in upgrade, and you should pin a 0.23 version until they return.

The Rust crate makes training and Hub downloads opt-in

For Rust, the dependency line is:

[dependencies]
tokenizers = "1.0.0"

Training (the train feature) and downloading from the Hub (the http feature) are both opt-in, so a default build cannot train a tokenizer or fetch one from the Hub until you enable them. Underneath, the library is split into sub-crates. tk-encode is the inference engine: the BPE, Unigram, WordPiece and WordLevel models plus the Normalizer, PreTokenizer, Model, PostProcessor and Decoder pipeline. Keeping it apart from training means a serving binary never links a trainer, a corpus reader or legacy converters. tk-serialize provides from_json_file, which turns a canonical tokenizer.json into a PipelineTokenizer, and tk-train holds the Trainer trait and the concrete trainers.

Old tokenizer.json files go through tk-convert before the reader accepts them

The v1 reader expects a canonical tokenizer.json. Files written by older versions of the library are handled by tk-convert, whose canonicalize_file rewrites a legacy tokenizer.json into the canonical form. It is a pure JSON to JSON rewrite that depends on nothing but std::path and serde_json. The maintainers made it separate so every published config stays readable without the runtime carrying years of compatibility branches, and because the legacy code made up a large share of the final crate size. The trade-off for on-device builds: disabling tk-convert makes the binary smaller, but then the build can only read configs that are already canonical, so convert legacy files ahead of time if you strip it out.

bitcannon runs pre-tokenization as a bitstream program

Pre-tokenization lives in its own crate, bitcannon. It classifies Unicode characters into tags and runs pre-tokenization as a bitstream program, following the MICRO'25 paper Interleaved Bitstream Execution for Multi-Pattern Regex Matching on GPUs, with an algorithm of the project's own so no parabix engine is needed. Tags come from a sparse table generated when bitcannon is compiled, which removes any dependency on the heavy Unicode tables. Each tag is a u8 with a high and low nibble, carrying refinement where needed, and using u8 tags means fewer bitstreams to process. It ships byte-exact grammars for gpt2 / ByteLevel, cl100k, o200k, tekken, deepseek and kimi-k2. The maintainers say that, as far as they know, it is the fastest implementation for every pretokenizer regex on CPU, but they also note the regex unrolling may have missed one or two common patterns.

ModernBERT and OLMo tokenizers hit a 64-bit hash collision

One open item on the Rust core list is a load failure for ModernBERT and OLMo, recorded as a "64-bit hash collision". Until that is fixed, those tokenizers cannot be assumed to load under the release candidates, and anyone serving those models should keep them on 0.23. The same list carries other unfinished core work that changes behaviour or output shape: bitnorm, an optimized normalization that will bring breaking changes and is meant to land before v1, windowing as truncation with stride, offset tracking, padding as a 2D flat buffer, and better training and decode performance. The safe habit during the rc phase is to encode a sample of your own text with both versions and diff the ids before deploying.

Rust, Python, Node.js and WebAssembly work today; C and the rest are planned

The bindings table marks Rust as the reference implementation, with Python under bindings/python, Node.js under bindings/node and WebAssembly published as tokenizers-wasm on npm. C, C++, Java, Go and others are marked planned. C and C++ bindings for loading, encode and decode are on the v1 list, while Java, Go and Swift bindings come after 1.0, built on top of the C bindings. Ruby support exists only as a community project in an external repository, ankane/tokenizers-ruby, so its release timing is not tied to this one. For a Java or Go service, that means there is no first-party binding to adopt now. Also after 1.0 is tk-devices, an exploratory GPU encoding and batch decoding component still subject to prototyping and measurement.

Editorial conclusion

Move to the tokenizers v1.0.0 release candidates if you only need to load a tokenizer.json or a Hub tokenizer and encode or decode with it, especially in Rust where the split crates keep serving binaries small. Stay on 0.23 if your Python code trains tokenizers, reads offsets, saves modified tokenizers or uses decode_stream, since those are still on the v1 roadmap. Before switching, load your own model's tokenizer under the release candidate and compare its output, because ModernBERT and OLMo load failures are still listed as open.

Frequently asked questions

Official sources

1. huggingface/tokenizers on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases