# toon-format encodes JSON for token budgets rather than for storage

> toon-format is the official Python implementation of TOON, a compact human-readable encoding of the JSON data model built to save tokens in LLM prompts. It suits teams that hand structured data to a model and want it to read clearly, and it is the wrong shape for long-term storage.

**toon-format/toon-python** — 🐍 Community-driven Python implementation of TOON

- Repository: https://github.com/toon-format/toon-python
- Stars: 767 · Forks: 61
- Language: Python
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/toon-format-toon-python

## A format shaped by the token budget, not by the file system

TOON stands for Token-Oriented Object Notation, and the name says everything about the design. It is a compact, human-readable encoding of the JSON data model, and the stated purpose is to save tokens in LLM prompts. The README's advice is to keep working with dictionaries and lists in your code and to encode them as TOON only where a model reads them.

That last sentence is the whole positioning. This is not a replacement for JSON on disk, and it is not a database serialization format. It is a transport format for the moment when a prompt is assembled, and the optimisation target is the tokenizer rather than the parser.

The Python package is the official implementation, not a third-party port, and it implements TOON specification 4.4 and passes every official conformance fixture. The specification lives in a separate repository, and the package is pure Python with no dependencies, fully typed, and requires Python 3.10 or later.

The repository layout is conventional for a modern Python package: src/ for the code, tests/, docs/ with an mkdocs.yml, a pyproject.toml, a uv.lock, a .pre-commit-config.yaml, a .devcontainer/ directory, and a CHANGELOG.md. There is also an AGENTS.md and a CLAUDE.md, which tells you the project expects coding agents to work in it and has written down the conventions for them.

## pip install toon-format, then dumps and loads

Installation is a single package name:

```bash
pip install toon-format
# or
uv add toon-format
```

The import name is toon_format, and the API deliberately mirrors the standard library json module. dumps and loads work on strings, dump and load work on files, and the same keyword arguments apply where they make sense.

The quick start is a nested dictionary of the kind you would hand to a model:

```python
import toon_format

data = {
    "context": {"task": "Our favorite hikes together", "location": "Boulder"},
    "friends": ["ana", "luis", "sam"],
    "hikes": [
        {"id": 1, "name": "Blue Lake Trail", "distanceKm": 7.5, "wasSunny": True},
        {"id": 2, "name": "Ridge Overlook", "distanceKm": 9.2, "wasSunny": False},
    ],
}

text = toon_format.dumps(data)
print(text)
```

The round trip is asserted in the README with a single line, and the guarantee it rests on is losslessness: loads(dumps(data)) equals data for every JSON-compatible value. That property is what makes the format safe to use as a boundary between your code and a model, since nothing is dropped on the way in.

Pydantic users install the extra, toon-format[pydantic], which pulls in pydantic and, on Python 3.10 only, typing-extensions.

## Uniform arrays collapse into a header and one row per record

The encoded output is where the format earns its name. The same dictionary prints as:

```
context:
  task: Our favorite hikes together
  location: Boulder
friends[3]: ana,luis,sam
hikes[2]{id,name,distanceKm,wasSunny}:
  1,Blue Lake Trail,7.5,true
  2,Ridge Overlook,9.2,false
```

Two mechanisms do the work. A scalar list becomes a count and a comma separated line, so friends[3] tells you there are three entries and then names them. An array of objects with the same keys becomes a header, with the key names written once inside braces after the array length, and every object reduced to a row of values in that order.

The header line, hikes[2]{id,name,distanceKm,wasSunny}, carries the schema and the row count, so each subsequent line needs no key names at all. For a list of fifty records with the same fields, that is the difference between repeating every key on every line and writing it once.

Note what the format does not do. It does not reorder or rename your keys, it does not convert your values, and it does not invent structure for data that has none, which is why the round trip is exact rather than approximately equal.

## The saving is large on uniform rows and small on irregular data

The README gives a token comparison for the example above, and it is worth reading carefully because it also states where the benefit ends. With OpenAI's o200k_base tokenizer, that document takes 65 tokens, against 76 for compact JSON and 130 for indented JSON.

So against a compact JSON baseline the saving is about 15 percent, and against the indented JSON a human would normally paste, it is about half. The README also states that the saving grows with the number of rows, and that 100 uniform records take 38 percent fewer tokens than compact JSON.

That is the mechanism in one sentence: the win comes from tables, and the more rows share a shape, the more there is to amortise. And the README is equally direct about the other case, saying that deeply nested or irregular data gains less.

That boundary decides where the format is worth using. A list of hundreds of records with the same fields is the ideal payload, because the header is written once and the rows are almost bare values. A deeply nested object graph with varying keys is where the encoding has little to remove, and the token saving can shrink to a point where the extra encoding step is not worth the complexity.

The counterpoint to those numbers is that the comparison is against a specific tokenizer, and tokenizers differ by model family. Treat 65 versus 76 as the shape of the effect, not as a number to quote in a design document.

## Decode errors carry the line number and the offending text

Parsing is where a new format usually disappoints, because a failure tells you only that something is wrong. TOON's error type is more useful. Invalid documents raise toon_format.ToonDecodeError, which is a ValueError subclass carrying the 1-based line and the source line of the error:

```python
try:
    toon_format.loads("a: 1\ntags[3]: a,b")
except toon_format.ToonDecodeError as exc:
    print(exc.line, exc.source)  # 2 tags[3]: a,b
    print(exc)  # line 2: Declared length 3 but found 2
```

The example is a document that declares three tags and provides two, and the message says so in those words. When a model produces malformed output, an error that points at the line and quotes it is worth a great deal, because the fix is usually a prompt change rather than a code change.

The decoder also has a tolerance mode. loads accepts strict=False to tolerate wrong counts and indentation, which is the setting to reach for when the input comes from a model that may have miscounted a list. The flip side is that a tolerant decoder can accept a document that a strict one would reject, so the choice belongs where the input is known to be machine generated.

## encode and decode exist so TypeScript examples carry over

The package exposes the names used by the TypeScript reference implementation, @toon-format/toon, and by the other TOON implementations, as aliases of dumps and loads:

```python
from toon_format import decode, encode

data = {
    "users": [
        {"id": 1, "name": "Ada", "role": "admin"},
        {"id": 2, "name": "Bob", "role": "user"},
    ]
}

print(encode(data))
# users[2]{id,name,role}:
#   1,Ada,admin
#   2,Bob,user

assert decode(encode(data)) == data
```

The point is portability of examples and of muscle memory across the implementations, which matters for a format that lives in a specification with several language ports.

Options are keyword arguments in snake case, a spelling the specification allows, so the JavaScript indentSize: 4 becomes indent_size=4 and strict: false becomes strict=False. One asymmetry is called out explicitly: the streaming functions encodeLines and decodeStream, and the replacer option, have no Python counterpart in this package. Anyone porting a TypeScript pipeline that streams will have to restructure that part.

## Pydantic models work, and the package ships no command

With Pydantic installed, dumps accepts any model, and the toon_format.pydantic module adds TOON counterparts of the JSON helpers. ToonPydanticModel gives a model a validate-from-TOON entry point:

```python
from toon_format.pydantic import ToonPydanticModel


class User(ToonPydanticModel):
    name: str
    age: int


user = User.model_validate_toon("name: Ada\nage: 36")
```

There is no command in this package. The README points at the official @toon-format/cli, which runs with Node.js and needs no installation:

```bash
npx @toon-format/cli data.json -o data.toon   # JSON to TOON
npx @toon-format/cli data.toon -o data.json   # TOON to JSON
npx @toon-format/cli data.json --stats        # token savings against JSON
```

The third invocation is the interesting one for adoption, because it lets you measure the saving on your own documents rather than trusting the example numbers. The trade-off is that a Python-only team now needs Node.js on the machine to run that measurement, since the CLI lives in the TypeScript toolchain.

For a Python project that wants everything in one language, the readthedocs documentation covers usage, the Python type mapping, the choices the specification leaves to implementations, and Pydantic.

## Version 1.2.0 after a 0.9 that broke its own API

The project has moved fast and is now at version 1.2.0, with v1.0.0 tagged on 2026-10-02, v1.1.0 on 2026-10-06 and v1.2.0 on 2026-10-18, the same day as the last push to the repository. The package metadata marks the development status as Production/Stable and the licence is MIT.

The migration from the 0.9 line is the part to read before upgrading an existing integration. The distribution and the module keep their names and encode(value) and decode(text) keep working, but three things changed. Options are now keyword arguments instead of a dictionary, the token helpers and the toon command are gone, and the output follows specification 4.4. A migration guide exists in the documentation, and the project keeps a CHANGELOG.md for the detail.

The dependency picture is unusually clean. The runtime dependencies list is empty, the package is pure Python and fully typed, and the development group is what you would expect: mypy for the typing claims, pytest and hypothesis for the tests, attrs, pydantic, and prek for the pre-commit hooks.

The design constraint to keep in mind is the one from the first section. This is a prompt format, it is lossless, and it is at its best on uniform tabular data. Adopt it at the boundary where a model reads your data, and leave the rest of your stack in JSON.

## Conclusion

Adopt toon-format when you build the prompt that carries structured data to a language model and the token count matters, since the format keeps the JSON data model, stays lossless and ships with no dependencies. Do not adopt it as a storage format, since it is designed for a single prompt rather than for round-tripping through a database over time, and do not expect savings on deeply nested or irregular payloads. Before you switch, run the npx @toon-format/cli --stats comparison on your own documents, check that your team can live with the snake_case option names, and read the 0.9 migration guide, because the options and the command changed.

## FAQ

### How do I install the TOON Python package?

Run pip install toon-format, or uv add toon-format. The module is imported as toon_format, and the Pydantic integration needs the extra, toon-format[pydantic].

### Does toon-format round trip data without losing anything?

Yes. The README states that loads(dumps(data)) equals data for every JSON-compatible value, so nothing is dropped when you encode to TOON and decode again.

### How much fewer tokens does TOON use than JSON?

The README gives an example where one document takes 65 tokens against 76 for compact JSON and 130 for indented JSON with the o200k_base tokenizer, and states that 100 uniform records take 38 percent fewer tokens than compact JSON, while deeply nested or irregular data gains less.

### Why does my TOON parse fail with a line and source attribute?

Invalid documents raise toon_format.ToonDecodeError, a ValueError subclass that carries the 1-based line and the source line of the error. You can also pass strict=False to tolerate wrong counts and indentation, which is useful for model generated output.

### Does the toon-format Python package include a command line tool?

No. The README points to the separate @toon-format/cli package, which runs with Node.js through npx and can convert between JSON and TOON or report token savings with the --stats flag.

## Sources

- [Issues](https://github.com/toon-format/toon-python/issues)
- [License: MIT](https://github.com/toon-format/toon-python/blob/main/LICENSE)
- [README](https://github.com/toon-format/toon-python/blob/main/README.md)
- [Releases](https://github.com/toon-format/toon-python/releases)
- [toon-format/toon-python on GitHub](https://github.com/toon-format/toon-python)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/toon-format-toon-python