lm-format-enforcer: filtering the token set instead of begging for JSON

The mechanism: a parser that returns the allowed tokens

Models emit text; asked for a format, they improvise one. Better instructions raise the hit rate and leave a gap. This project takes the other route and constrains the sampler instead of the prose.

The data flow has three steps. You build a character-level parser for the format you want, such as a JSON Schema, a regex or schemaless JSON mode. You adapt that parser to your inference framework with a helper function. The generated function is passed into the generation call as a prefix function, and at every timestep it returns the set of tokens the model is allowed to emit, given the characters produced so far.

Because the filter is applied per step, a token that would break the format is never sampled. There is no retry loop, no repair pass and no parsing failure to handle, which is what distinguishes this from asking nicely in a prompt.

Two properties make the approach less restrictive than it sounds. The model retains freedom over whitespace and field ordering inside a JSON object, so the output is not a rigid template with the values bolted in, and the README argues that this freedom reduces hallucinations. The parser operates on characters, not on your schema's semantics, so it validates shape and type rather than business meaning.

Nothing in the core imports a model. The parser works on strings, so the library can sit in front of any backend that exposes a token step function.

A first run with pydantic and transformers

The README's basic tutorial is complete and short. Declare a pydantic model describing the shape you want:

class AnswerFormat(BaseModel):
    first_name: str
    last_name: str
    year_of_birth: int
    num_seasons_in_nba: int

Then wire four pieces together: a transformers pipeline, the schema parser, the adapter that turns the parser into a prefix function, and the generation call:

from lmformatenforcer import JsonSchemaParser
from lmformatenforcer.integrations.transformers import build_transformers_prefix_allowed_tokens_fn

hf_pipeline = pipeline('text-generation', model='TheBloke/Llama-2-7b-Chat-GPTQ', device_map='auto')
prompt = f'Here is information about Michael Jordan in the following json schema: {AnswerFormat.schema_json()} :\n'

parser = JsonSchemaParser(AnswerFormat.schema())
prefix_function = build_transformers_prefix_allowed_tokens_fn(hf_pipeline.tokenizer, parser)

output_dict = hf_pipeline(prompt, prefix_allowed_tokens_fn=prefix_function)
result = output_dict[0]['generated_text'][len(prompt):]

Getting it installed is a single pip line, with pydantic, interegular, packaging and pyyaml pulled in as the runtime set. Reproducing the README example on a local machine needs more than that: transformers, torch, huggingface_hub, optimum, and auto-gptq for the GPTQ checkpoint the tutorial names.

The extraction step matters: the pipeline returns the prompt concatenated with the completion, so the code slices off len(prompt) characters before parsing. The commented expected result is a dictionary with the four fields populated, which is the test that the whole chain works.

One detail is easy to miss. Note that the schema still appears in the prompt. The enforcement does not replace telling the model what you want; it guarantees the answer is well formed when the model is wrong.

What the parsers cover, and what they deliberately allow

Three format families are supported: JSON Schema, JSON mode without a schema, and regular expressions. The regex path is what interegular is for, compiling a pattern into a state machine that can answer the same allowed-token question.

Inside a schema the handled shapes are required fields, optional ones, nesting, arrays, dictionaries, and classes that refer to themselves. That last one matters more than it sounds: a recursive schema produces a language the parser must handle with a stack rather than a fixed rule table, and it is exactly the case where a simpler implementation gives up.

What is on purpose not enforced is worth stating too. Semantic constraints are out of scope, since the parser can tell you a field is a string but not that it should be a plausible name. Coercion is out of scope for the same reason, and the README does not claim otherwise.

Visual model support is listed as a capability with a dedicated notebook for a Llama 3.2 vision setup, so image input and format enforcement are meant to work together rather than being two separate features you have to reconcile.

The comparison table at the end of the README is the most useful part of the documentation, because it states capabilities rather than adjectives. Optional JSON fields, recursive classes, model-controlled ordering and whitespace, batched generation and beam search are the columns where this library claims more than its alternatives, and the project invites corrections through issues when a cell is wrong.

Batches, beams and leaving your generation loop alone

The claim that shapes the integration story is that the library does not modify the high level loop of the transformers API, so it can be used in any scenario.

Practically, that means the enforcement is a callback rather than a wrapper. Your existing pipeline call keeps its arguments, its batching and its stopping criteria, and one keyword carries the constraint. There is no custom generate loop to adopt and no sampler to replace.

Batching and beam search are the two features that most token-filtering libraries give up, because a constraint function has to be evaluated per sequence rather than once per batch. Filtering happens per sequence, so each input or beam can have a different token set at every timestep, which is the correct semantics: two sequences in the same batch may be at different points in their JSON and therefore have different legal continuations.

For beam search this is what keeps multiple candidates valid rather than only the greedy path, which is why the comparison table marks beam search support where two of the three alternatives mark it unsupported.

The framework coverage follows the same principle. Rather than shipping its own inference stack, the project provides integrations for transformers, LangChain, LlamaIndex, Haystack, llama.cpp, vLLM, NVIDIA TensorRT-LLM and ExLlamaV2, with a sample notebook in the samples directory for each.

Serving it through vLLM without custom inference code

If you run vLLM, the library is integrated into the inference server rather than wrapped around it, which is the cleanest way to use it without writing inference code at all.

There are two ways in. Server-wide, with a command line parameter on the OpenAI compatible server:

python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.2 \
  --guided-decoding-backend lm-format-enforcer

Or per request, by passing a guided_decoding_backend parameter alongside the guided decoding parameters in the chat completion call, which lets one deployment serve both constrained and unconstrained traffic.

That distinction matters in production. A global backend switch means every request pays for enforcement, which is right if every caller wants structured output and wasteful if half of them do not. The per-request form costs you a decision per call site and nothing else.

The rest of the framework story is the same shape: a helper per library, an example notebook per helper, and unit tests as the reference for the transformers path. The README points at tests/test_transformerenforcer.py for the various ways to integrate, which is more useful documentation than prose would be.

How it compares to Guidance, Jsonformer and Outlines

The README includes a capability table against three alternatives, and it is more informative than most such tables because it names mechanisms rather than grades.

Guidance comes closest: it handles regex and visual models, while its JSON Schema column records a partial conversion, and the table points at a notebook where a Jsonformer approach is bridged in. On the axes that matter for batched serving, Guidance is marked as not supporting batched generation, beam search, or integration into existing pipelines.

Jsonformer is marked as supporting JSON Schema and recursive classes but not regex, and not the other four. Outlines is the strongest of the three in the table, with regex, JSON Schema, batched generation and beam search all marked, but not optional JSON fields, and not letting the model control field ordering and whitespace.

So the differentiation is narrow and specific: optional fields, recursive schemas, model-controlled layout, and dropping into a pipeline you already have. If none of those four is your problem, Outlines looks like the closer fit on the table's own terms, and the library's own README says to open an issue when a capability changes.

For teams already on vLLM, the tiebreaker is simpler: whichever backend the server already exposes.

Packaging, tooling and what a version pin means

The library is MIT licensed and packaged with Poetry, with the package name lm-format-enforcer and the import name lmformatenforcer. pyproject.toml declares version 0.11.3, while the newest v0.11.2 entry dates from 2025-08-09, with v0.11.1 a day earlier and v0.10.12 before that. A patch-level difference between the file and the newest entry is normal in a Poetry project with a tag-driven flow, but it means the version in your environment and the one in the repository are not guaranteed to match.

The classifier is Development Status 3 - Alpha, which is the author's own assessment and worth taking literally. The supported Python range is 3.8 to below 4.0, and pydantic from 1.10.8, so the project has kept compatibility with old environments rather than tracking new ones. The development tooling is thorough for a small library: flake8 with bugbear and pep8-naming, black for formatting, mypy for types, isort, tox for multi-version runs, pre-commit, and liccheck, a licence checker that is an unusual thing to find inside a project.

The timeline is the part to weigh. The repository was last pushed to on 2026-04-04, and the numbered versions stop in August 2025. For a library whose integration surface is a set of framework-specific helper functions, that matters more than it would for a self-contained package, because a framework upgrade can leave the adapter behind.

Editorial conclusion

Adopt lm-format-enforcer when you need structured output from a model you run yourself and you would rather guarantee the format than parse and retry. The guarantee is real, because at every step the sampler is limited to tokens that keep the output parseable, and it drops into an existing transformers pipeline with one extra argument rather than replacing your generation loop. Do not adopt it expecting a fast-moving dependency: the repository was last pushed to on 2026-04-04, pyproject.toml carries version 0.11.3 against a newest entry of v0.11.2 from 2025-08-09, and the package classifies itself as Alpha. Verify four things first. Install with pip install lm-format-enforcer and check what version you got, since the metadata and the newest numbered entry disagree. Confirm your integration path: a build_transformers_prefix_allowed_tokens_fn call if you use transformers, or the guided_decoding_backend parameter if you serve through vLLM. Decide whether you want the model to control field order and whitespace, because that freedom is a feature the README claims against every alternative in its comparison table. And check your Python floor, since the project declares support from 3.8 upwards with pydantic from 1.10.8. Start with JsonSchemaParser over a pydantic model and a single prompt, as in the README example.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. noamgat/lm-format-enforcer on GitHub
4. README
5. Releases