gpt-tokenizer: a tiktoken port that runs synchronously in JavaScript

What gpt-tokenizer replaces in a JavaScript stack

Counting tokens before a request is how you avoid a 400 from the API and how you decide what to truncate. In Python that job belongs to tiktoken. In JavaScript the options have historically been either a WASM or native binding, a hand-rolled encoder, or a round trip to a service. gpt-tokenizer targets the last two cases by shipping the BPE merge tables as ordinary modules that the bundler can include. The README describes it as a port of OpenAI's tiktoken with additional features, written in TypeScript, and lists Microsoft Teams and GenAIScript, Elastic's Kibana, Effect TS, CodeRabbit and Ironclad's Rivet as users. That list is the clearest signal of the intended audience: application code, often in a browser or a serverless function, where adding a Python process is not an option. The package also covers a broader model range than a naive port would, from GPT-2 through GPT-5 and the o-series, plus audio, transcription, image and video models.

The encoding tables are the architecture

There is no server and no model in this library. Each supported encoding, r50k_base, p50k_base, p50k_edit, cl100k_base, o200k_base and o200k_harmony, is a static table of byte pairs and ranks. encode walks the input text, applies the regex splitting rules for that encoding, then repeatedly merges the highest-ranked adjacent pair until no merge applies. The output is an array of integers. decode reverses it by concatenating the byte sequences the ids map to. The README notes one implementation detail worth caring about: the library eliminates transitive arrays during encoding, which is the kind of change that matters when you tokenize a long document in a hot loop. Because the tables are data, importing a model entry pulls in only that encoding. Importing from the package root gives you o200k_base, the encoding the README says is used by GPT-5, GPT-4.1, GPT-4o and the o-series. If you need a different one you import from a per-model path instead, and the tables for the others are not loaded.

Synchronous loading, generators and the functions that are not in tiktoken

The README makes a point of synchronous loading, meaning you can call encode at module scope or inside a render without awaiting anything. That is the main structural difference from a WASM binding, which usually has to be initialised first. Around that core the package adds functions tiktoken does not expose in the same form. encodeChat takes an array of message objects with role and content and returns tokens for the whole conversation, including the per-message framing the chat format adds. isWithinTokenLimit returns false when the limit is exceeded and otherwise returns the token count, so it doubles as a check and a measurement, and the README presents it as a way to assess a limit without encoding the entire text. estimateCost computes API usage cost, and the repository keeps a model catalog in src/models.ts and src/models.gen.ts with context limits, capabilities, modalities and pricing. encodeGenerator, decodeGenerator and decodeAsyncGenerator cover streaming: decodeAsyncGenerator accepts any AsyncIterable of token ids and yields text chunks, which is what you want when tokens arrive from a streaming completion.

Installing it and picking the right import path

The install is a single command: npm install gpt-tokenizer. In a browser without a build step the README shows an ES module import from unpkg, and it explicitly says to pin a package version in production and that global UMD bundles are no longer published. That last point is a migration note for anyone upgrading from an older major. The import path is where most integration mistakes happen. The default entry uses o200k_base, so encode from gpt-tokenizer and encode from gpt-tokenizer/model/gpt-4o are not interchangeable if you are targeting an older model. To select a model you import from gpt-tokenizer/model/gpt-3.5-turbo, or from gpt-tokenizer/cjs/model/gpt-3.5-turbo when the resolver does not understand package.json exports. If the model is not in the catalog but you know its encoding, the README offers gpt-tokenizer/encoding/cl100k_base as a fallback. Special tokens are opt-in: isWithinTokenLimit accepts an options object with allowedSpecial, and the package exports ALL_SPECIAL_TOKENS for the case where you want them counted. Lazy loading is a dynamic import of the same model path inside a function, which moves the table load off the critical path.

Where the port stops being a drop-in

The honest limitation is in the word port. The README does not claim byte-for-byte parity with tiktoken on every input, and the additional features are exactly where divergence is possible: chat framing changes between model generations, and encodeChat has to encode a specific message format that the API may not match for every model in the catalog. Treat its output as an estimate for budgeting and truncation, not as a substitute for the usage field the API returns. The second constraint is coverage. Six encodings are listed. A model whose encoding is not among them is not supported by a model entry, only by the encoding path if you already know which one it uses. Third, the model catalog and pricing live in generated source files, so a new model or a price change requires a package release; nothing in the material suggests the tables are fetched at runtime. Finally, the README's performance claims are the project's own and are not reproduced here, and the browser import path means the encoding tables ship to the client, which is a payload decision you make once per encoding you import.

How it differs from tiktoken and from a WASM binding

tiktoken is the reference implementation, in Python and Rust, and it is the thing this package ports. If your token counting already happens in a Python service alongside your model calls, keeping tiktoken means one implementation and no drift, at the cost of a network hop from any JavaScript caller. gpt-tokenizer inverts that: the tables live in your bundle, so counting is local and synchronous, and the cost is bundle size plus the maintenance of a port that has to track OpenAI's releases. A WASM binding sits between the two. It gives you the upstream algorithm more directly than a reimplementation, but the README's emphasis on synchronous loading and on avoiding transitive arrays reads as a direct answer to the initialisation and allocation overhead those bindings carry. The trade is that a port can lag or diverge, and a binding usually cannot. There is also a narrower comparison inside the JavaScript ecosystem: the README calls out the original GPT-3-Encoder implementation for keeping a global cache that can leak memory, and states that this package has no global cache. If you have been bitten by a tokenizer holding onto strings between requests, that design choice is the reason to switch.

Maintenance cost, licence and what to check before you ship

The licence is MIT, which permits commercial use and modification provided the copyright notice and permission notice are retained; that is a description of the licence text, not legal advice, and your own counsel should confirm how it interacts with your distribution model. Maintenance is the real cost. The release history shows 3.3.0 and 3.4.0 within a day of each other in November 2025, then 4.0.0 in August 2026, so expect occasional majors with breaking import or packaging changes; the README's note that UMD bundles are gone is an example of the kind of change a major carries. Because the model catalog and pricing are generated files in the repository, staying current means tracking releases rather than reading a remote endpoint. Before you ship, run your own prompts through encode and encodeChat and compare the counts against the usage numbers the API returns for the same inputs, and confirm that the model path you import matches the model string you send in the request. If those two disagree, the tokenizer is measuring a different model than the one you are paying for.

Editorial conclusion

Adopt gpt-tokenizer if you are writing JavaScript or TypeScript that runs in a browser, an edge worker or a synchronous code path and needs to count tokens for an OpenAI model before sending a request. Do not adopt it if your pipeline already calls tiktoken in Python and consistency with that exact implementation matters more than removing a service, or if your model's encoding is not one of the six the package ships. Before committing, verify three things on your own machine: that the encoding you need resolves through your bundler's exports handling, that the model entry you import matches the model you actually call, and that encodeChat's token arithmetic agrees with the usage numbers returned by the API for a sample of your real prompts.

Official sources

1. License: MIT
2. niieani/gpt-tokenizer on GitHub
3. Project website
4. README
5. Releases