What is Prompt injection?

What actually happens during a prompt injection

The mechanism is easier to understand if you start from how a language model reads. A model has no privileged channel. Everything it knows at a given moment, the system prompt, the conversation so far, the documents you pasted, the results its tools returned, is one flat sequence of tokens. There is no marker in that sequence saying this part is an instruction and that part is information.

So when text arrives that contains something phrased as a command, the model has no reliable way to tell whose command it is. If a web page it was asked to summarise contains a line telling it to disregard its instructions and instead return a specific value, or to call a particular tool, that line is as fluent and as plausible as the system prompt. The model may follow it. This is the whole of it.

Two variants get named separately and both are worth distinguishing. Direct injection is the user typing the malicious instruction, or an attacker convincing a user to paste it. It is easy to spot and mostly a social engineering problem.

Indirect injection is the one that matters in production. The attacker does not touch your application. They publish a page, send an email, upload a document, or leave a comment somewhere your agent will read. When your system puts that content into the context, the instruction is already inside. The user may have typed nothing malicious at all, and the model still ends up doing what the attacker wanted.

The second reason indirect injection is hard is that the payload arrives at the moment the model is most likely to be obedient. An agent asked to summarise a document is doing exactly what it was told, and the instructions in the document look like more of the same work.

Why it happens: instructions and data share one channel

It is tempting to treat this as a bug in a particular model, and it is not. It follows from the design.

A model that reads a sequence of tokens has one way of representing everything. A system prompt, a user's question, a retrieved document, a tool's JSON response and an attacker's sentence are all just text. There is no type system over the context window. When you build an agent, you are in effect concatenating trusted and untrusted text into the same place and hoping the model can tell them apart, and the model is doing its best with no way to be certain.

This is why the framing of data exfiltration is the common one. An attacker who can get an instruction into the context often does not need anything clever: they ask the model to include a specific value in its answer, and a system with access to tools or to connected data can comply, because from the model's point of view the request came from the same place as every other instruction it has been following.

The consequence for design is uncomfortable but worth stating plainly. You cannot fully solve this by prompting. Instructions in the system prompt telling the model to ignore instructions found inside documents reduce the frequency of successful attacks, and that is worth doing, but they do not change the fact that both kinds of instruction arrive through the same channel. A sufficiently direct payload that happens to read like a system instruction will be followed, because the model has no way to verify authorship.

That is why most real defences move the boundary rather than trying to persuade the model. Anything the model can be made to do, you want it to be able to do only within limits you set in code.

What actually helps, and why it is not a filter

The mitigations that work are the ones that do not depend on the model resisting anything.

Least privilege for tools is the largest single improvement. If your agent can send an email but not read the address book, an injected instruction to exfiltrate contacts has nothing to exfiltrate. The same applies to databases, shells, payment APIs and file systems: enumerate what the agent can actually reach, and remove the capabilities the task does not need. Most systems have far more reach than their prompts suggest.

Treat everything the model produces as untrusted input. If the model's output is used to build a query, a path, a command or an API call, that output needs validation on the way in, exactly as a value from a web form would. This is the same discipline as any injection class, and the reason the analogy to SQL injection is a good one: the parameterisation lives in your code, not in the prompt.

Require a person for side effects that matter. Sending, deleting, paying, publishing and permission changes are the actions worth a confirmation step, and that step should show the user what is about to happen in terms they can judge rather than a summary written by the model that is itself compromised.

Separate the data from the instructions structurally where you can. Fetching a document into a clearly delimited, untrusted block, giving tools narrow schemas, and keeping credentials out of the context the model reads are all ways of making the attack harder rather than asking the model to be careful.

What does not work well is detection. Classifiers trained to spot injection attempts raise the cost of a lazy attack and do nothing about a patient one. Sanitising text by stripping strings that look like instructions is a race against phrasing, and phrasing is cheap to vary. These are worth having in a defence in depth, and they should not be the layer you are relying on.

The limits everyone runs into

It is worth being honest about where this leaves you, because most writing on the topic oversells either the danger or the fixes.

The pessimistic reading says any system that reads untrusted text and can take actions is permanently compromised. That is too strong for most practical systems, because least privilege and human confirmation narrow what a successful injection can actually accomplish, and in a lot of deployments the worst case is a wrong answer rather than a breach.

The optimistic reading says a good system prompt fixes it. That is clearly wrong, and it is wrong in a specific way: it produces false confidence. Teams who believe the model is filtering attacks stop looking at the tool permissions, and that is precisely where the real control lives.

There are also structural limits that no amount of prompting addresses. A model that has been given a document to summarise has no way to know that a sentence inside it was not part of the document's subject matter. Retrieval that pulls the wrong page has the same problem one level up. And an agent with long-running memory has a version of this where the injected instruction persists, which is a different problem from a single poisoned request and a harder one.

The practical position is that you are managing a risk rather than eliminating a bug. The work is to make the worst outcome survivable: narrow what the agent can reach, put a person in front of the actions that matter, and instrument the system so you can see what your agent was told and what it did. That last one is underrated, and it is the reason logs matter more here than in most application debugging.

How prompt injection shows up in open-source projects

The projects below split into attack material and defensive tooling, and both are worth looking at if you are building something that reads untrusted text.

Attack material, published openly. 0xk1h0/ChatGPT_DAN is a prompt collection rather than a tool, gathering role-play prompts that ask the model to answer as a second, unrestricted persona; its README dates its own DAN 12.0 note to 20230711, which is a useful reminder of how old this genre is. LouisShark/chatgpt_system_prompt is a community-assembled archive of system prompts from custom GPTs, organised for educational reference on prompt writing and for security research into prompt injection and leaking. Goochbeater/Spiritual-Spell-Red-Teaming collects prompt material aimed at language model guardrails with a short guide and a list of techniques, and its README is written in invisible Unicode tags, which says something about the project as plainly as its content does. MDX-Tom/gpt-instruct ships instruction files for a specific model family, a Python installer that edits a config file, and a three-tier evaluation gate, and our analysis of it notes that the v1 release ran the full B suite and did not clear the project's own hard threshold.

Defensive tooling. superagent-ai/superagent is an SDK that classifies prompts, redacts personal data and secrets, and scans repositories for agent-targeted attacks; the guard and redaction paths are documented, the red team module is marked as coming soon, and the runtime calls depend on a hosted API key. Tencent/AI-Infra-Guard bundles agent and skills scanning, MCP server checks, AI infrastructure vulnerability scanning and jailbreak evaluation into one Docker-deployed service, aimed at teams that already run agents and MCP tooling and want a repeatable way to probe them.

Both halves of that list are worth reading. The attack repositories show which phrasings people actually reach for, and knowing that is more useful to a defender than any amount of theory about what an attacker might try.

In practice

The useful way to hold prompt injection is that a model reads one stream of tokens and cannot verify who wrote any part of it, so the defence has to live in your code: narrow what the agent can reach, validate everything the model produces before acting on it, and put a person in front of the actions that matter. Prompt-level defences are worth having but should not be the layer you rely on. To see the problem from the other side, read 0xk1h0/ChatGPT_DAN and LouisShark/chatgpt_system_prompt for what the attack surface looks like, and superagent-ai/superagent and Tencent/AI-Infra-Guard for the tooling people are building on the defence side.