Gigatoken: A Rust tokenizer that trades Python compatibility for raw GB/s throughput

The problem Gigatoken targets

Tokenization is a bottleneck in large-scale language model training. The README positions Gigatoken as a drop-in replacement for HuggingFace tokenizers and tiktoken, claiming to tokenize text at gigabytes per second. The intended user is someone preparing massive text corpora, like the 11.9 GB OpenWebText file used in their benchmarks, where every second of preprocessing matters. The project is written in Rust, and the README notes that both HuggingFace tokenizers and tiktoken already run multithreaded Rust underneath. So the speedup does not come from merely switching to a compiled language; it comes from a different design that avoids Python overhead and reads data directly from disk.

How the native API achieves speed

The fastest path is the Gigatoken API, which takes a HuggingFace model name and a file source. The example shows `gt.TextFileSource(["owt_train.txt"], separator=b"<|endoftext|>")` and then `tokenizer.encode_files(file_source)`. The README explains that this lets the Rust implementation read data directly, skipping as much overhead as possible while allowing maximum parallelism. The key mechanism is that the native API operates on file sources, not on Python objects. Passing Python data structures still incurs overhead from reading from Python, so the design pushes users toward file-based batch processing. The separator parameter suggests the tokenizer handles document boundaries within a single file, which is typical for pretraining corpora where special tokens separate documents.

Compatibility mode: a measured trade-off

For existing users, Gigatoken offers a compatibility mode. You wrap an existing HuggingFace tokenizer with `gt.Tokenizer(hf_tokenizer).as_hf()` and then call `encode_batch` as you would with the original. The README is explicit: a substantial amount of effort went into matching outputs exactly, but this comes at a non-negligible cost to performance. You still get faster speeds, but not the 1000x you get with the native API. This is a honest admission. The compatibility mode is not a free lunch. It exists to let you test Gigatoken without rewriting your pipeline, but the performance advantage shrinks. The README does not quantify how much slower compatibility mode is, so you cannot predict whether it will still be worth the switch.

What the benchmark tables actually show

The README includes three benchmark tables, each on a different CPU: a dual-socket AMD EPYC 9565 with 144 cores, an Apple M4 Max with 16 cores, and an AMD Ryzen 7 9800X3D with 16 cores. The headline numbers are striking: on the EPYC, GPT-2 tokenization runs at 24.53 GB/s versus 24.8 MB/s for HuggingFace, a 989x speedup. But the tables reveal wide variation. For Gemma 1 on the EPYC, the speedup is only 7.3x. For Mistral 7B v0.3 it is 10x. On the Ryzen, GPT-2 shows 106x, not 1000x. The 1000x figure only appears on the high-core EPYC and M4 Max for certain tokenizers. The pattern suggests that tokenizers with smaller vocabularies or simpler merge rules benefit most, while others like Gemma and Mistral have more complex tokenization that limits parallel speedup. The README notes that the benchmark file is representative of CommonCrawl-extracted text, but it does not show results for short strings or interactive use.

Getting started: installation and first run

Installation is a single pip command: `pip install gigatoken`. The usage examples are short. For the native API, you instantiate `gt.Tokenizer("Qwen/Qwen3-8B")`, which accepts HuggingFace model names. Then you create a `TextFileSource` with a list of file paths and a separator byte string. The separator is likely the end-of-text token used to split documents. For compatibility mode, you pass an existing tokenizer object, either from HuggingFace or tiktoken, and call `as_hf()` or `as_tiktoken()`. The README does not mention any configuration for parallelism or memory, so you rely on defaults. There are no release tags or changelog in the repository, which means you cannot pin a specific version from the material.

Where it is the wrong tool

Gigatoken is not designed for interactive or low-latency tokenization. The native API reads from files, not from Python strings, and the README warns that passing Python data structures incurs overhead. If your workload is tokenizing individual prompts in an inference server, you will not see the benchmark speedups. Also, the compatibility mode, while easier, sacrifices performance, so you could end up with only a modest speedup after paying for the integration effort. The benchmark tables show that for certain tokenizer families, like Gemma and Mistral, the speedup is in the single digits to low tens on consumer hardware. If you use those tokenizers, the 1000x claim does not apply. The README also does not mention support for decoding back to text, which is a common requirement in generation pipelines.

Alternatives and how they differ

The obvious alternatives are HuggingFace tokenizers and tiktoken, both of which Gigatoken compares against. HuggingFace tokenizers is a Rust library with a Python API that supports a wide range of tokenizers and offers an `encode_batch` method. Its design prioritizes correctness and flexibility over raw throughput, and it handles both encoding and decoding. Tiktoken is OpenAI's tokenizer, focused on their specific models, and is also Rust-based. Both are already multithreaded, according to the README, so Gigatoken's advantage comes from specialized file reading and reduced Python boundary crossing. The difference in approach: HuggingFace and tiktoken operate on Python sequences, while Gigatoken's native API operates on file sources. If you need to tokenize data that is not in files, the alternative libraries are a better fit. If you need decoding, you must check whether Gigatoken supports it, but the README does not show any decode method.

Maintenance and license considerations

The repository is MIT licensed, which means you can use it in commercial projects with few restrictions, though this is not legal advice. The last push was September 2026, and the project is not archived, so it appears active. However, there are no recent releases listed, which is a concern for dependency management. You cannot rely on a stable version tag from the material. The README does not mention a changelog or migration guide, so upgrading may be risky. The project also has no homepage, which limits documentation beyond the README. For a performance-critical library, you should verify that the maintainer responds to issues and that the build is reproducible. The benchmark code is not shown in the README, so you cannot reproduce the numbers without cloning the repository and inspecting the benchmark scripts yourself.

Editorial conclusion

Adopt Gigatoken if you are preprocessing large corpora for language model training and can tolerate its native API, which requires reading from files directly and accepts only certain tokenizer variants. Do not adopt it if you need exact parity with HuggingFace tokenizers in production, because the compatibility mode explicitly trades performance for output matching, and the README warns that matching is not guaranteed. Before committing, verify that your specific tokenizer is in the supported list, run your own benchmark on your target CPU, and confirm that the 1000x headline applies to your workload, since some tokenizers show only 7x to 20x speedups on certain hardware.

Official sources

1. Issues
2. License: MIT
3. marcelroed/gigatoken on GitHub
4. README