What is LLM guardrails?

What actually happens when a guardrail runs

A guardrail is a piece of code that sits between your application and a model, and it runs at one of three points.

On the input, before the model sees anything. This is where you check the shape of what arrived: a length limit, a schema, a filter for content you will not process at all, or a check for text that is trying to give the model new instructions. A request that fails here never reaches the model, which is the cheapest possible place to stop something.

On the output, after the model has answered. This is where you check that what came back is something you can use. The strongest version of this is structural rather than judgemental: you give the model a schema for its answer and the output has to conform, so a field that should be an integer is an integer or the request is rejected. A weaker version asks another model whether the answer is acceptable, which is a check with a different failure mode, because a judge model can be talked into agreeing.

On behaviour, during a conversation or an agent's work. Here the guardrail is less about content and more about scope: which topics the assistant will discuss, which sequence of turns is allowed, which tools it may call, and what it is permitted to do with a result. This is the least standardised of the three and the one that most often turns out to be application logic wearing a different name.

Across all three, the action taken on failure is what separates one guardrail from another. Block and return an error. Block and retry, usually with the failure reason added to the prompt. Block and repair, by asking the model to fix its own output against the schema. Or allow but flag, recording the violation for a human to review. The last of these is under-used and is the right choice far more often than teams expect.

Where the guardrail lives decides what it can protect

The same rule implemented in three different places behaves very differently, and choosing the place is most of the design work.

In the application, as a library call around each model invocation. This is the most flexible and the least reusable, because a developer has to remember to wrap every call, and the call they forget is the one that matters. It is the right choice when the rule depends on application state, such as which user is asking or what they are entitled to see.

In a proxy or gateway, between your services and the provider. Everything that calls a model goes through one place, so a rule applies uniformly, and configuration lives outside the code. This is the shape most of the open source tooling below takes, and it is the right choice for policies that are uniform: rate limits, cost ceilings, model allow-lists, provider routing, and blanket refusals.

In the agent's tool layer, as permissions on what the model can reach. This is not usually described as a guardrail, but it is the one that prevents the most damage, because it constrains consequence rather than content. A model that can only read a customer's own invoices cannot exfiltrate somebody else's, whatever it is convinced to do.

There is a fourth place that people forget: the model itself, through fine-tuning or a carefully written system prompt. That is not a guardrail, because it is a suggestion to something that can be persuaded. It is worth doing and it is worth not counting on.

When you need them, and when they are just another layer

Guardrails earn their place in a small number of situations, and it is worth being able to name which one you are in.

They are necessary when the output feeds code. If a model's answer is parsed, executed, used to build a query or rendered into a page, then validating it is not optional, and a schema is the cheapest form of validation there is.

They are necessary when the input is not yours. If your application reads text from users, from the web, or from another system, you have a content question and a boundary to draw. Blocking some categories outright is a product decision, and it is cheaper to make in a gateway than in every call site.

They are useful when you need an audit trail, because a guardrail that flags rather than blocks produces the record of what the model was asked to do.

And they are useful for cost, which people forget: a gateway with per-route limits and a fallback chain is doing something no system prompt can do.

Now the cases where they are not worth it. A single-user chat with no tools and no downstream parsing needs almost none of this, and a guardrail framework will mostly add latency and configuration. Internal drafting tools where a mistake costs a paragraph and not a process do not justify a validation layer either. And a rule that blocks on a false positive rate high enough to annoy people will get disabled, and a disabled guardrail is worse than none because somebody still believes it is there.

The honest test is whether a failure has a consequence outside the conversation. If it does, you need a check. If it does not, you are adding latency and a thing to maintain.

What guardrails do not do

It is useful to be blunt about the limits, because the term is used loosely and the expectation it creates is often wrong.

A guardrail does not make a model safe. It reduces the frequency of one class of failure. Nothing in the vocabulary of check, filter or block changes the fact that the component deciding what is acceptable is a language model reading untrusted text, and that component can be persuaded.

A guardrail does not stop prompt injection, and this is the most common misunderstanding. A content filter sits in the same class of component as the thing it is filtering. If a guardrail reads a document alongside the model's instructions, it can be injected into as easily, and if it is a separate call with a separate prompt, it can be injected into through the content it is judging.

A guardrail does not replace permissions. This is the one that bites hardest in agent systems. Restricting what an agent can reach is a different mechanism from restricting what it can say, and only the first one holds when the model is convinced.

The practical failures of guardrails are quieter. They add latency, which matters in an interactive path. They accumulate configuration, and a guardrail config is code that nobody reviews. They produce false positives, and the response to a false positive is always to loosen the rule, which quietly lowers the ceiling over time. And they can be bypassed by calling the model provider directly, which is why a gateway only works if everything actually goes through it.

What works better than most of this is unglamorous: narrow permissions, a schema the output has to match, and a human in front of the actions that matter.

How guardrails show up in open-source projects

The projects below fall into three groups, and the group a tool belongs to tells you what it will and will not protect.

Frameworks that wrap the call in your code. guardrails-ai/guardrails is an Apache-2.0 Python framework that wraps LLM calls with validators and Pydantic-based output schemas, which makes it the tool for rejecting or repairing model output; our analysis of it notes that a recent move away from hosted inferencing changes how validators are installed. NVIDIA-NeMo/Guardrails is a Colang-driven toolkit that wraps a call in a Python layer able to enforce dialog flows, topic boundaries and input and output checks, and our analysis of NeMo Guardrails describes it as something for teams that already run their own model endpoint rather than a drop-in hosted filter.

Gateways and proxies, where the rule is configuration rather than code. BerriAI/litellm ships as a Python SDK and as a self-hosted proxy, and our analysis of litellm is that it is worth adopting when provider sprawl rather than model quality is the bottleneck, and the wrong tool when you only ever call one vendor. maximhq/bifrost is a Go gateway putting OpenAI, Anthropic, Bedrock, Vertex and others behind one OpenAI-compatible endpoint with fallbacks and load balancing; our analysis of Bifrost notes it starts in one command but that several headline features sit behind enterprise deployments. katanemo/plano is an Apache-2.0 proxy that moves agent routing, model selection and tracing out of your application into a config file, and our analysis of Plano says it fits teams already running more than one agent or provider and is the wrong tool if you want a library to import.

Platforms that bundle the whole control plane. archestra-ai/archestra bundles an LLM gateway, an MCP registry and gateway, an agent runtime and deterministic guardrails into a single Docker image, aimed at platform teams wanting one control point across several coding tools and internal chat users, where our analysis of archestra names the trade-off as adopting a large surface area at once. future-agi/future-agi bundles tracing, evaluations, simulations, a Go gateway and guardrails into one platform you run with an install script, where our analysis of future-agi puts the trade-off at roughly seventeen containers before you have evaluated a single prompt. IBM/mcp-context-forge is the tool-facing case, unifying MCP, A2A and REST or gRPC services behind one endpoint with centralised guardrails.

And one project that is instructive by going the other way. freecodexyz/free-code is a community fork of the Claude Code CLI that strips outbound telemetry, and among other changes it removes the vendor's system-prompt guardrails and unlocks experimental feature flags. That is a reminder that guardrails are a policy choice with an owner, and that a user with root on the machine can always decide the policy is different from the one you shipped.

In practice

The useful way to hold LLM guardrails is that they are a set of decisions about what to check and what to do when the check fails, placed at the input, the output, the behaviour or the tool permission. Start with the cheapest layer that solves your actual problem: a schema on the output if the output is parsed, a block list in a gateway if the input is not yours, and narrow permissions on the agent if consequences matter. Prompt-level guardrails are worth writing but they are not a boundary. To read further, guardrails-ai/guardrails and NVIDIA-NeMo/Guardrails show the in-code approach, BerriAI/litellm and maximhq/bifrost show the gateway approach, and archestra-ai/archestra shows what it looks like when both come bundled with a control plane.