Prompt Fuzzer: an interactive attack runner for system prompts

The problem Prompt Fuzzer addresses: a system prompt nobody has attacked

Most teams write a system prompt once, ship it, and never test it against an adversary. The README frames the tool around exactly that gap: it "assesses the security of your GenAI application's system prompt against various dynamic LLM-based attacks" and produces a security evaluation from the outcome of those simulations. The intended user is whoever owns that prompt, which in practice is an application engineer or an AI security engineer rather than a penetration tester working from a network position. The tool is interactive by default and ships a Playground chat interface so you can edit the prompt and re-run after each failure. That loop (attack, read the result, rewrite, attack again) is the product. If your workflow does not include rewriting the prompt, the fuzzer has nothing to feed back into. The README also states plainly that using it consumes tokens, which is the practical constraint that shapes how often you can afford to run it.

What the fuzzer actually sends: attack model against target model

The architecture visible in the command line options is a two-model design. One model is the attacker, selected with --attack-provider and --attack-model, and it generates the adversarial prompts. The other is the target, selected with --target-provider and --target-model, and it receives your system prompt plus those generated attacks. The README describes the tests as "dynamic" and says the fuzzer "dynamically tailors its tests to your application's unique configuration and domain", which is the reason an LLM sits on the attack side rather than a static list of payload strings: the attack text is generated against the context you supplied. --attack-temperature controls sampling on the attack model, so the diversity of generated attacks is a knob you can turn. --num-attempts sets how many distinct attack prompts are produced, and --num-threads parallelizes the run. The attack catalogue is grouped in the README under Jailbreak, Prompt Injection, RAG and Vector Database Attacks, and System prompt extraction. The RAG group is the one that needs extra configuration, because it requires an embedding provider and model rather than just a chat model.

Install and first run: pip, an API key, then the interactive prompt

Installation requires Python 3.10 or newer and one command: pip install prompt-security-fuzzer. The package name on PyPI differs from the repository name, so pip install ps-fuzz is not the documented path. The README also points to a release wheel file on the GitHub releases page as an alternative. Launch is a single binary name, prompt-security-fuzzer, and before that you export a provider key. The documented example is export OPENAI_API_KEY=sk-123XXXXXXXXXXXX, and the README notes you can instead place the key in a .env file in the current directory. On start the tool asks for your system prompt, then begins testing. For unattended runs there is -batch (also written -b), which the README describes as bypassing the interactive steps. Two base URL flags matter for self-hosted or proxied setups: --ollama-base-url for a local Ollama deployment and --openai-base-url for OpenAI-compatible endpoints, which is how you would point the fuzzer at an internal gateway rather than the vendor API. The README marks Docker support as coming soon, so containerized use is not documented yet.

Provider coverage and what the key list does not tell you

The README lists sixteen providers behind their own environment variables: ANTHROPIC_API_KEY, ANYSCALE_API_KEY, AZURE OPENAI_API_KEY, BAICHUAN_API_KEY, COHERE_API_KEY, EVERLYAI_API_KEY, FIREWORKS_API_KEY, GIGACHAT_CREDENTIALS, GOOGLE_API_KEY, JINA_API_TOKEN, KONKO_API_KEY, MINIMAX_API_KEY with MINIMAX_GROUP_ID, OPENAI_API_KEY, PROMPTLAYER_API_KEY, QIANFAN_AK with QIANFAN_SK, and YC_API_KEY. --list-providers prints the same set at runtime. The list is broad, and that breadth is mostly inherited from the underlying framework rather than proven per provider here: the README does not state which of these have been exercised as attack models versus target models, and those are different jobs. An attack model has to follow instructions well enough to produce varied adversarial text, while a target model just has to answer. Treat --list-providers as a menu to test, not a compatibility guarantee. The same caution applies to the embedding flags, which are documented as required for RAG tests and accept only ollama or open_ai as the provider value, so a RAG run against, say, a Cohere target still needs one of those two embedding backends.

Where it stops: prompt hardening is not application hardening

The scope boundary is the most important thing to understand before adopting this. Prompt Fuzzer evaluates a system prompt. It does not inspect your tool-calling permissions, your retrieval filters, your output parsing, your rate limits or your logging. A prompt that resists extraction in the Playground can still sit inside an application that hands the model a database tool with no row-level restriction, and the fuzzer will report success because the prompt held. The README's own framing supports this reading: the deliverable is a security evaluation of the prompt and a Playground for iterating on it. There is a second limitation that is structural rather than scoped. The attacker is a language model, so results vary between runs. --attack-temperature exists precisely because generation is stochastic, and a single run at a low --num-attempts value is a weak signal. A clean score means the attacks that were generated in that run failed, nothing more. Anyone who treats one green run as a clearance has misread the tool. The README does not publish a fixed scoring rubric or a threshold that defines a passing prompt, so the interpretation of the evaluation is left to the reader.

What to compare it against: garak and static red-team suites

The closest open-source comparison is garak, NVIDIA's LLM vulnerability scanner. The difference is in where the attack text comes from. garak ships a large library of predefined probe families and detectors that score model output against known failure patterns, which makes runs comparable across models and time. Prompt Fuzzer generates its attacks with a second LLM at runtime, which is what lets it adapt to your domain and your prompt rather than to a fixed taxonomy. The trade is reproducibility: a static probe suite gives you the same test twice, a generative attacker does not. A second comparison point is the manual red-team spreadsheet, which is free and tailored but does not scale past a handful of prompts. Prompt Fuzzer sits between those two positions, and the honest reading is that it complements a static scanner rather than replacing one. Run garak for coverage you can diff between releases, run Prompt Fuzzer when you have changed the system prompt and want domain-aware pressure on the new wording.

Maintenance, licence and the cost of keeping it current

The project is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is a permissive licence, and it is the same licence garak uses, so licence terms are not a differentiator between the two. This is not legal advice; check the LICENSE file in the repository for the binding text. On maintenance, the release history shows 2.0.0 in August 2024, v2.1.0 in February 2026 and 2.1.1 in August 2026, so the cadence is roughly two releases a year rather than continuous churn. The repository is not archived and the last push recorded is 2026-08-19. The real ongoing cost is not the package, it is the tokens. Every run spends budget on both the attack model and the target model, and the README flags this explicitly. The practical upgrade risk is provider drift: sixteen environment variables and a dependency on external chat APIs means a vendor deprecation or an SDK change can break a run without any change on your side, so pinning the installed version and re-running a known prompt after each upgrade is the cheap way to detect that.

Editorial conclusion

Adopt Prompt Fuzzer if your security work is specifically about the system prompt and you already have an API key and a token budget for the provider you intend to test against. Do not adopt it as a general application security scanner: it exercises the prompt, not your tool permissions, retrieval filters or output handling. Before relying on a result, verify three things in your own environment: that the target provider and model in --target-provider and --target-model match production, that --embedding-provider and --embedding-model are set if you intend to run the RAG tests, and that the reported score is reproducible across two runs with the same --num-attempts value, because the attack side is a language model and sampling temperature is exposed through --attack-temperature.

Official sources

1. License: MIT
2. Project website
3. prompt-security/ps-fuzz on GitHub
4. README
5. Releases