TONL: a compact text format for data headed into an LLM prompt

A text serialization format aimed at prompt token budgets

TONL expands to Token-Optimized Notation Language, and the intent is narrow: shrink the amount of text a JSON payload occupies when it is fed to a language model, while keeping the result readable by a person rather than binary. The README's own summary of the project calls it a data platform combining compact serialization with query, modification, indexing and streaming, and the package description repeats that framing. Two architectural choices stand out for an engineer evaluating it. The implementation is TypeScript throughout, and the README states zero dependencies in pure TypeScript, so there is no native module to compile. The package manifest also splits the library into separate entry points for schema, parser, stream, query, optimization, navigation, indexing and modification, which means a browser build can avoid pulling in the parts it does not use.

Installing and driving the format from the command line

The package installs from npm:

npm install tonl

The CLI covers the everyday conversions. Encoding takes a JSON file and writes a TONL file, and the README notes that it quotes special keys to make the round trip exact:

tonl encode data.json --out data.tonl --smart --stats

The reverse direction is a decode back to JSON, and the same tool can prettify a TONL file, validate it against a schema, answer a path query, or compare token costs between two representations:

tonl decode data.tonl --out data.json
tonl query users.tonl "users[?(@.role == 'admin')]"
tonl validate users.tonl --schema users.schema.tonl
tonl format data.tonl --pretty --out formatted.tonl
tonl stats data.json --tokenizer gpt-5

A stats subcommand also has an interactive mode with selectable themes and a file comparison mode, which is the closest thing here to a visual diff between a JSON file and its TONL counterpart.

Queries use JSONPath-like syntax over the document

The library API works on a TONLDocument built from JSON, and the accessor surface reads like JSONPath with additions. The README's quick start shows a get for a single path, a query for a wildcard, a filter expression over an array of objects, and a recursive descent:

doc.get('users[0].name');

doc.query('users[*].name');

doc.query('users[?(@.role == "admin")]');

doc.query('$..age');

On top of that sit aggregation helpers such as count, sum, avg and groupBy, a stats call returning count, sum, avg, min, max and standard deviation, and separate modules for fuzzy search, phonetic matching and temporal literals such as @now-7d. The document is also mutable, with set and push, and it can be walked or iterated entry by entry before being written back out as TONL or JSON. The formatting details of the format itself are documented in the TONL.md file and in the docs directory rather than in the README.

Streaming and optimization live in their own entry points

Two parts of the design go beyond a serializer with a query layer. The stream export point means a document can be processed incrementally rather than held whole, which matters when the input is larger than you want in memory. The optimization export point holds an AdaptiveOptimizer that picks a strategy on its own, plus a BitPacker for boolean arrays and a DeltaEncoder that encodes timestamp series by difference, the kind of data where consecutive values share most of their bytes:

const packer = new BitPacker();
const packed = packer.packBooleans([true, false, true]);

const delta = new DeltaEncoder();
const timestamps = [1704067200000, 1704067201000, 1704067202000];
const compressed = delta.encode(timestamps, 'timestamp');

For browser use the package exposes a separate browser entry point, and the README puts the gzipped bundle at roughly 29.5 KB in IIFE and UMD form, with React 18 and Vue 3 interactive demos listed among the examples.

The README's numbers are its own, and one of them disagrees with itself

The feature list in the README is written in marketing form, so it is worth separating the claims from the facts. The project claims up to 60 percent smaller output than JSON and speed figures of 10 to 1600 times faster than its targets, but the README gives no method for either, and the repository keeps its benchmarks in a separate bench directory. The test totals are also inconsistent. The release notes for v2.5.2 state 216 new tests and 698 tests across 162 suites with a full pass rate, while the feature list higher up claims more than 2,300 tests passing. Both cannot describe the same suite. Security is handled seriously on paper, with a SECURITY.md file, an eslint plugin for TONL security rules in the repository, and a note that five npm vulnerabilities were resolved in v2.5.2, but the claim that the project is production ready is a statement by the authors rather than an audited result.

No GitHub releases, and the version trail lives in the README

The repository has no GitHub releases, so the version history that most maintainers rely on is absent. The README itself carries a release section that lists v2.5.2 dated December 20, 2025, v2.5.1 dated December 11, 2025 and v2.5.0 dated December 3, 2025, and the package manifest carries the same version, 2.5.2. The repository is not archived, and the last push was on 2026-08-07, so work continues on the main branch even though no tagged release has been published there. The licence is MIT, with the LICENSE file in the repository root. For a package that calls itself production ready, the absence of tagged releases and of a published changelog entry beyond the README is the weakest part of the project, and it means consumers pin a version without a published release note to consult.

Editorial conclusion

TONL suits teams that send large JSON payloads into model prompts and want a smaller text form with querying still available on the decoded document. It does not suit a project that needs a wire format other tools already parse, since adoption means every consumer has to learn TONL. Before adopting, encode one of your own real payloads and measure the token count and round-trip behaviour yourself, because the size and speed numbers in the README are the project's own claims and the test totals stated there are inconsistent.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. tonl-dev/tonl on GitHub