JTokkit: a Java tokenizer for OpenAI models, without the Python dependency

The gap JTokkit fills for JVM teams calling OpenAI

If your service is written in Java and you want to know how many tokens a prompt will cost before you send it, the obvious reference implementation is not available to you. tiktoken is a Python library, and the README says JTokkit grew out of the need for similar capacities in the JVM ecosystem. That is the whole pitch, and it is a narrow one: JTokkit does not call any API, does not manage prompts, and does not talk to OpenAI. It converts text to token IDs and back.

The audience follows from that. Backend engineers who enforce a per-request token budget, teams that truncate conversation history to fit a context window, and anyone building a cost estimate before a request leaves the process. The README frames the use case directly, describing tokenizing input text for counting required tokens in preparation of requests to the GPT-3.5 model. If your tokenization already happens in Python, or you are happy to pay for a round trip to count tokens server-side, this library adds a dependency you do not need.

How the encoding registry and BPE pipeline are wired

The entry point is an EncodingRegistry, created through Encodings.newDefaultEncodingRegistry(). From it you request either a named encoding via EncodingType or the encoding that corresponds to a model via ModelType. The README shows both paths, and notes that the two can return the same object: in its example, getEncodingForModel(ModelType.TEXT_EMBEDDING_ADA_002) returns the same encoding as EncodingType.CL100K_BASE.

Underneath, the default encodings are byte pair encoding. The README lists the supported sets as r50k_base, p50k_base, p50k_edit, cl100k_base and o200k_base. The extension path exposes that machinery: you can register an entirely new Encoding implementation, or construct a GptBytePairEncodingParams with a name, a regex pattern, an encoding map and a special token encoding map, then hand it to registerGptBytePairEncoding. That second option is the interesting one, because it means a custom vocabulary reuses the existing BPE loop rather than requiring you to write a tokenizer from scratch.

One design detail worth noting because it affects how you deploy the library: the README states that EncodingRegistry and Encoding are thread-safe and can be freely shared among components. A single registry held in a static field or injected as a singleton is the intended shape, not one registry per request.

Adding JTokkit to a Maven or Gradle build

The README gives two installation paths, both pointing at the same artifact. For Maven, add the dependency to your pom.xml. The coordinates and version below are copied from the README, and the artifact resolves from Maven Central under the group com.knuddels.

<dependency>
    <groupId>com.knuddels</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.0</version>
</dependency>

For Gradle, the README shows the equivalent declaration. Note that the README writes this in Groovy DSL syntax.

dependencies {
    implementation 'com.knuddels:jtokkit:1.1.0'
}

A first real use is counting tokens before a request. The README's getting started example creates a registry, asks for CL100K_BASE, encodes a sentence, and decodes it back. The encoded value in its example is [2028, 374, 264, 6205, 11914, 13], and decoding that list returns the original sentence.

EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding enc = registry.getEncoding(EncodingType.CL100K_BASE);
IntArrayList encoded = enc.encode("This is a sample sentence.");
// encoded = [2028, 374, 264, 6205, 11914, 13]

String decoded = enc.decode(encoded);
// decoded = "This is a sample sentence."

If you only need a count, call encode and read the size of the returned IntArrayList rather than decoding anything. The library ships no CLI and no server mode, so this is the shape of every integration: a library call inside your own process. The README points to https://jtokkit.knuddels.de/ for a fuller getting started guide.

Where the 2-3x performance claim comes from, and what it does not cover

The README claims JTokkit is between 2 and 3 times faster than a comparable tokenizer, and points at a benchmark directory in the repository and a generated chart at benchmark/reports/benchmark.svg. That is more than most libraries offer, and the benchmark directory is in the repository listing, so the harness is inspectable. But the claim is comparative without naming the baseline in the README text, and it says nothing about the input distribution, the JVM version, or whether warmup was excluded. Treat it as a reason to run the benchmark yourself on your own text, not as a number you can put in a capacity plan.

The same caution applies to the zero-dependency claim in the features list. Zero runtime dependencies is a real advantage for a library that sits in an application's hot path, because it cannot drag a conflicting transitive version into your build. It also means JTokkit has to implement everything it needs itself, and the encoding data has to come from somewhere inside the artifact.

The maintenance question: a stable release line and a moving target

The last push to the repository was on 2026-08-25, so the project is not dormant. The release history tells a different story about cadence: 1.1.0 shipped on 2024-07-19, 1.0.0 on 2024-02-10, and 0.6.1 on 2023-07-03. Between the 1.1.0 release and the most recent push there is more than a year of commits with no tagged release, which is worth knowing if you depend on release artifacts rather than snapshots. The README documents version 1.1.0 as the dependency to add, and that is the version to pin.

The upgrade cost is mostly about encodings, not API churn. The core surface is a registry, an encoding, and two methods. What changes over time is the set of encodings OpenAI defines and the vocabulary behind each one. A new model family can arrive with a new encoding, and until JTokkit ships it you are either extending the library yourself through GptBytePairEncodingParams or falling back to another tool. There is no documented deprecation policy, and the README does not describe a release schedule. Plan for upgrades as occasional and low-effort, but do not assume a new encoding will appear the week a model does.

On licensing, the README states JTokkit is licensed under the MIT License and points at the LICENSE file. MIT is permissive and imposes no copyleft obligation on your application, but it also comes with no warranty. If you are redistributing the library or bundling it into a product, read the actual LICENSE text rather than this summary; this is a description of what the README says, not legal advice.

When JTokkit is the wrong tool

The clearest failure case is a model family the library does not cover. The README lists five encodings, all of them OpenAI's. If you call a model from another vendor, or a self-hosted model with its own vocabulary, JTokkit gives you nothing out of the box, and the extension path is real work: you supply the pattern, the encoding map and the special token map yourself, and a wrong regex will silently produce different token boundaries than the model expects.

A second case is approximate counting. If you only need a rough sense of prompt size and you already make network calls, a cheap heuristic may be enough, and adding a library plus a registry to every service is overhead you can skip. The third case is non-Java stacks. The README is explicit that this exists because the JVM lacked what tiktoken provides elsewhere; if your tokenizer runs in Python, Go or Rust, JTokkit is not the answer, and the README itself names tiktoken as the counterpart it was built to match.

There is also a subtler trap. Token counts are model-specific, and the README's own example shows that a model type and an encoding type can map to the same object. Getting that mapping wrong means your budget is computed against the wrong vocabulary, and nothing in the library will tell you. Verify the mapping against OpenAI's documentation for the model you actually call.

JTokkit versus tiktoken, and versus writing your own BPE

The README names the real alternative directly: tiktoken, OpenAI's Python library. The difference is not features, it is where the code runs. tiktoken is the reference implementation and lives in the Python ecosystem, so a JVM service that wants it either runs a sidecar, calls out to a Python process, or reimplements the encoding. JTokkit removes that hop. If your stack is already Python, tiktoken is the natural choice and JTokkit buys you nothing; the trade only makes sense when the caller is Java.

The other alternative is implementing BPE yourself, which is what JTokkit's extension API is designed to prevent. Writing byte pair encoding is not conceptually hard, but matching an existing vocabulary exactly is, because the regex pre-tokenization pattern and the merge table have to agree with the source. The README's GptBytePairEncodingParams constructor takes exactly those two things plus names, which tells you the library treats them as the configurable parts and the merge loop as the fixed part. If your need is a custom encoding that follows the same BPE structure, extending JTokkit is less work than a from-scratch implementation. If your need is a genuinely different tokenization algorithm, the Encoding interface is the escape hatch, and at that point you are writing the algorithm anyway.

Editorial conclusion

Adopt JTokkit if you run JVM services that need local token counts before calling an OpenAI model, or if you want to decode token IDs without shelling out to Python. Do not adopt it if you need a tokenizer for a non-OpenAI model family, since the README lists only r50k_base, p50k_base, p50k_edit, cl100k_base and o200k_base. Before committing, verify that the encoding you need matches the model you call, confirm your build resolves com.knuddels:jtokkit:1.1.0 from Maven Central, and read the benchmark directory if throughput matters to you.

Frequently asked questions

Official sources

1. knuddelsgmbh/jtokkit on GitHub
2. License: MIT
3. Project website
4. README
5. Releases