llguidance: enforcing a grammar on a language model's every token

A mask per token instead of a check after the fact

The mechanism is worth understanding before the feature list, because it explains both the speed and the trade-offs. Given a context-free grammar, a tokenizer, and a prefix of tokens already generated, llguidance computes a token mask: a set of tokens from the tokenizer that, when added to the current prefix, can lead to a valid string in the language the grammar defines. The sampler then restricts itself to that set. Mask computation takes approximately 50 microseconds of single-core CPU time for a tokenizer with 128k tokens, and while that timing depends on the exact grammar it holds for grammars derived from JSON schemas. There is no significant startup cost, which is the property that separates this approach from ones that precompute. The library is written in Rust and the numbers are per token, so the overhead is a fixed small cost added to each decoding step rather than a large cost paid once at the beginning of a request.

Four grammar formats, and the internal one is on the way out

The library supports a large subset of JSON schemas, regular expressions, and context-free grammars in a variation of the Lark format, with embedded JSON schemas and regular expressions allowed inside them. There is also an internal JSON-based format called llguidance, and the documentation is candid about its status: it is the most expressive, though the Lark-like format is catching up, and there are plans to convert the libraries to use it, so it is slowly being deprecated in favour of the Lark-like format. That deprecation is the single most important practical detail here, because choosing a format is a decision you will live with. The internal format is the one the higher level libraries generate. Guidance, the Python library for interacting with LLMs, and guidance.ts, the TypeScript one, both generate it, and the documentation expresses the hope that more will follow. If you are writing raw calls against the parser, pick the Lark-like format and avoid building on something that is on its way out.

The parser is an Earley parser over a lexer built from regex derivatives

The implementation detail that explains the performance profile is the combination of two known algorithms. The library implements a context-free grammar parser using Earley's algorithm, on top of a lexer based on derivatives of regular expressions. Mask computation is then achieved by traversing the prefix tree of all possible tokens, using highly optimized code. The prefix tree of the tokenizer is a separate crate in the workspace, toktrie, with sibling crates for the Hugging Face tokenizers, a downloader and the tiktoken tokenizers, which is how the same mask machinery is applied across the tokenizer families in common use. Grammars can also be used to speed up decoding through fast-forward tokens, a separate mechanism documented on its own, which is about skipping tokens that are unambiguous under the grammar rather than sampling them one at a time. The workspace is a Cargo workspace with ten members, and the release profile keeps debug information while stripping the debuginfo, with a separate perf profile that inherits from release and turns optimisation up to level 3.

It already runs inside the serving stacks you probably use

The integration list is the strongest argument for adopting this, because it means the constraint is available without changing how you serve. LLGuidance powers structured output for JSON Schema in OpenAI models. It is merged into Chromium, where it is used for JSON Schema enforcement for window.ai in Chromium based browsers. It is merged into vLLM in v0.8.2, into SGLang in v0.4.4, and into llama.cpp, where it is available through the -DLLAMA_LLGUIDANCE=ON option for cmake, and llama.cpp can also be used through the Guidance Python package. The SGLang route is a flag, --grammar-backend llguidance, and when passing a Lark grammar there is a detail to get right: the grammar must be prefixed with %llguidance {}, just as in llama.cpp. Other integrations listed are LLGTRT, an OpenAI-compatible REST server using NVIDIA's TensorRT-LLM, mistral.rs and onnxruntime-genai, the last of which is a Microsoft project.

Rust, C, C++ and Python, and nothing else yet

The library can be used from Rust, from C and C++ through a header, and from Python through a type stub file, and the repository holds samples for each: a minimal Rust sample, a C sample, and the Python bindings. The Python package is published under the name llguidance and its metadata shows the machinery. It is built with maturin, with a release profile, pyo3 bindings and a manylinux2014 compatibility target, and the extension is built from a separate Rust crate in the python_ext directory with its own Cargo manifest:

[tool.maturin]
profile = "release"
bindings = "pyo3"
compatibility = "manylinux2014"
python-source = "python"
module-name = "llguidance._lib"
manifest-path = "python_ext/Cargo.toml"

The package itself has no Python dependencies, requires Python 3.11 or newer, and its version is 1.9.1. The Python bindings live under a module name of llguidance._lib with the Python source in a python directory. That last detail, the leading _ character, is the usual convention for a compiled extension module inside a package, and it tells you the Python side is a thin wrapper over the Rust core rather than a port of it. The C and C++ surface is a single header, which is a much smaller commitment than a full binding layer. The Python project metadata names Michal Moskal as the author, and the contact address in the README is [email protected], which is a useful signal about who is behind the project.

The comparison with alternatives is about precompute versus on the fly

The README compares llguidance with two similar tools and one different one, and the comparison is on architecture rather than on benchmarks. LM-format-enforcer and llama.cpp grammars are similar in that they dynamically build token masks for every step of the decoding process, and both are significantly slower: the former because of clean Python code, and the latter because of the lack of a lexer and the use of a backtracking parser, which while elegant is inefficient. Outlines takes a different route, building an automaton from the constraints and then pre-computing token masks for all automaton states, which can make sampling fast but inherently limits the constraint complexity and introduces significant startup cost and memory overhead. llguidance computes masks on the fly with essentially no startup cost, and the lexer's automata are built lazily and are typically much smaller because the context-free grammar imposes the top-level structure. The detailed numbers live in MaskBench, part of the JSON Schema Bench project, and the project also maintains JSONSchemaBench with a paper.

A release history in the form of integration milestones

The project is documented through its timeline rather than a changelog alone, and the entries are the integrations. On 2025-01-07 Guidance v0.2.0 was released using llguidance as the grammar engine. On 2025-01-21 JSONSchemaBench was released, including the paper and MaskBench. On 2025-02-01 the llama.cpp integration was merged, on 2025-02-26 the SGLang integration, on 2025-03-25 the vLLM one, on 2025-04-11 the Chromium one, on 2025-05-20 LLGuidance shipped in OpenAI for JSON Schema, and on 2025-06-11 the Making Structured Outputs Go Brrr blog post came out, with the library deemed v1.0.0 on 2025-06-23. The last push to the repository was on 2026-10-14, so work continues, and the package version has moved to 1.9.1 since that v1.0.0 milestone. There are no GitHub releases to consult, so the timeline in the README and the CHANGELOG file are the record. The blog post is where the design decisions and the performance analysis are written up, and it is linked from the technical details section as the place to start.

Editorial conclusion

llguidance suits anyone who needs a model to produce parseable output rather than prose that might be parseable, because it enforces the constraint inside the decoding loop rather than checking the result afterwards, and it suits teams that already run vLLM, SGLang or llama.cpp and can flip a backend flag. It does not suit a project that cannot accept a grammar constraint, since a model that may only emit valid JSON will refuse to say anything outside it, and it does not suit a stack where you cannot swap in the native library. Before adopting it, decide which grammar format you will use, because the internal format is the most expressive but is slowly being deprecated in favour of the Lark-like one, and check the language bindings you need, since Rust, C, C++ and Python are listed and nothing else.

Frequently asked questions

Official sources

1. guidance-ai/llguidance on GitHub
2. Issues
3. License: MIT
4. README