TypeLLM: schema-guaranteed output from a model you already serve

Constrained decoding instead of a schema check after the fact

Most structured output work parses free-form model output and retries when the shape is wrong. TypeLLM attacks the problem at decoding time. It brings type-safe generation to existing autoregressive LLMs without changing their architecture or weights, so a model that already serves traffic through SGLang keeps its native thinking and free-form generation while producing schema-guaranteed output through JSON Schema. The stated effect on the output side is that there are no out-of-schema hallucinations, because choices stay within the allowed values. The cost side is claimed to be negligible output-token overhead, since categorical selection takes a single token and numeric decoding is bounded, with optional thinking mode adding tokens on top. The inspiration named in the project is TypeSafe AI's Jev, and the implementation sits on SGLang rather than on a custom inference engine.

A dependency graph over fields, not one flat schema

The feature that separates TypeLLM from a plain JSON Schema request is the execution model. Independent fields run together, and a field can declare depends_on to form a dependency graph, with incremental prefix reuse so shared context is not reprocessed. That combination is what the workflow model post describes, and it is why the API is able to pause a call and resume it. A call can wait for answers from your own tools or from a person, and then continues with them; in the hosted version the answer carries a source of user, while in the self-hosted setup you pass the answers yourself in given. On top of that, conditional fields let a field be answered only if its dependencies meet a condition, expressed as a when clause such as testing a category confidence against 0.8. The result is a workflow with types at every step rather than a single constrained answer.

Permutation averaging attacks option order bias

Enum and boolean models have a known weakness: the order of the options in the prompt biases the probability assigned to each. TypeLLM handles this with permutation averaging, which the project describes as reducing option-order bias on enum and boolean probabilities with balanced, sampled or exhaustive orderings, and it is on by default. The point of the feature is the predictive distribution rather than a single sampled answer, since averaging over several orderings cancels part of the positional effect. An insights post on the topic is titled fair die, and the examples directory has a fair_die folder, so the reasoning is spelled out outside the README. For a decision system where the choice is the product, this is the difference between a probability you can trust and one that reflects where the option happened to sit in the prompt.

Self-hosted setup is a client pointed at your own SGLang server

The installation is a single pip install, and the client talks to either the hosted API or a model you serve yourself.

pip install -U typellm

For the hosted path you create a key on the API keys page and export it.

export TYPELLM_API_KEY="tl-sk-..."

For the self-hosted path you serve a compatible autoregressive model with SGLang on your own GPU server, with prefix caching enabled, and then point the client at that endpoint.

from typellm import TypeLLMClient

client = TypeLLMClient(
    "http://127.0.0.1:30000",
    model="Qwen/Qwen3.8-27B",
)

The same client handles both, which is the point of the shared request shape. The client is also the console entry point: pyproject.toml declares a script named typellm mapped to typellm.cli:main.

Per-field budgets decide how much text a field can hold

The client is where the practical limits live, and they are set as constructor arguments. The base_url default is http://127.0.0.1:30000, the SGLang server URL. The model defaults to None, meaning the model ID is discovered from the server when it is omitted, and tokenizer defaults to None as well, in which case the server's paths are used and then its served model name. text_max_tokens defaults to 128, which is the text generation budget per field. numeric_max_digits defaults to 32 and is the digit limit for open numeric fields. temperature defaults to 0, and 0 gives each field the answer with the highest probability. The consequence for a schema designer is that a free-text field is not free: it has a 128 token ceiling unless you raise it, and a numeric field has a digit ceiling. Bounded integer and number outputs are separately constrained types, while an open numeric field is the one where the digit limit matters.

Boxes extend the same schema idea to image regions

Image handling arrived in two steps in October 2026. Boxes were added on 2026-10-15 and find something in an image, returned as its edges, so an output type is a region of a picture rather than a value. An array of boxes followed the same day and finds every match in an image, described as every dog, every line item. This is not a separate vision pipeline; it is the same typed output machinery pointed at coordinates, which means a request can return an array of typed regions alongside typed properties. Image input itself came earlier, on 2026-09-24, and lets you pass images to vision-language models alongside the text context, tested with Qwen3.8-27B. The supported output types are string, integer, number, boolean, enum choice, object and array, and thinking mode, added on 2026-09-19, lets the model reason before producing the constrained answer with a per-field budget.

The hosted API is ahead of the self-hosted path on paper

The README publishes a comparison table between the hosted TypeLLM API and the self-hosted setup with the same repository and Qwen3.8-27B, and the numbers are measured by TypeLLM in October 2026. On classes in a single choice field the hosted API lists 512 against 26 for self-hosted. Thinking speed is listed as 2.5 times faster against baseline. Arrays of objects are up to 34 percent higher recall and 27 percent faster on the hosted side, with 200 items a call against 50. Tools and human input differ in shape: a call pauses and resumes on the hosted side, while self-hosted means passing answers yourself in given. The API also exposes compatible endpoints for the OpenAI Decisions API and Jev System One. Against that, self-hosted needs no GPU from the provider and costs no credit, while the hosted API is listed with none for the GPU and five dollars of free credit. Read the table as a vendor benchmark of two configurations rather than as a neutral comparison.

JevBench scores and the shape of the evaluation

The accuracy claim is tied to a named benchmark. TypeLLM was evaluated on 231 public JevBench tasks, with the full results and every per-task answer kept in the evals/jevbench directory of the repository, alongside a method and configuration document. The published figures are 195 out of 231 without thinking and 228 out of 231 with thinking, and the corresponding update entry is dated 2026-09-23. The evals directory and the tests directory are separate from the examples directory, which holds fair_die, notebooks, receipt and receipt_us sample projects. This is the evaluation layout you would want if you intend to check the claims yourself: per-task answers are published rather than a single headline number, and the method document sits next to them. The licence is Apache-2.0, and pyproject.toml records the package as typellm version 0.7.2 with httpx, jinja2 and transformers as its dependencies.

Editorial conclusion

TypeLLM fits a team that already serves open weights through SGLang and needs structured output with choices that cannot leave the allowed set, and it does not fit a project that must run the same schema against many backends or that needs long free-form text in one field without a per-field budget. Verify three things before you commit: which checkpoints are on the tested list under Supported models, what the per-field budget of text_max_tokens at 128 does to your longest string, and how the API versus self-hosted trade-off in the comparison table lands on your hardware. The licence is Apache-2.0, the package version is 0.7.2, and the last push was on 2026-10-18.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. TypeLLM/TypeLLM on GitHub