# msoedov/agentic_security: an LLM vulnerability scanner you point at your own endpoint

> Agentic Security is an Apache-2.0 Python scanner that fuzzes LLM APIs with jailbreak and adversarial prompt datasets. It is easy to start and hard to trust blindly, because its scoring is threshold-based and its documentation leaves rollback and CI failure semantics open.

**msoedov/agentic_security** — Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

- Repository: https://github.com/msoedov/agentic_security
- Website: https://agentic-security.vercel.app
- Stars: 2,016 · Forks: 292
- Language: Python
- License: Apache-2.0
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/msoedov-agentic-security

## What agentic_security scans, and who it is written for

Agentic Security is an open-source vulnerability scanner for agent workflows and large language models. The repository describes its purpose as protecting AI systems from jailbreaks, fuzzing and multimodal attacks, and the topics list names prompt testing, LLM jailbreaks, LLM guardrails and LLM fuzzing. That framing tells you the intended user: a developer or security engineer who already has a model endpoint and wants to throw adversarial prompts at it on a schedule.

The tool is not a firewall and it does not sit in the request path. It is a client. You give it an HTTP specification for your model, it substitutes attack prompts into that request, and it records which ones produced a response it considers a failure. The distinction matters when you are deciding where this fits. Guardrail products filter live traffic; agentic_security measures how often your existing filtering fails.

The project is written in Python, licensed Apache-2.0, and published on PyPI as agentic_security. The last push to the default branch was on 2026-09-07, and the most recent release listed is 0.7.5 from 2026-06-03. Note the version mismatch: pyproject.toml declares version 1.0.0 while the release list stops at 0.7.5. That is worth checking against the tag you actually install.

## The mechanism: an HTTP spec, a dataset registry, and a threshold

Everything in agentic_security revolves around one idea. You describe your target as a plain text HTTP request, and the scanner replaces a placeholder with attack strings. The README gives this example:

```http
POST https://api.openai.com/v1/chat/completions
Authorization: Bearer sk-xxxxxxxxx
Content-Type: application/json

{
     "model": "gpt-3.5-turbo",
     "messages": [{"role": "user", "content": "<<PROMPT>>"}],
     "temperature": 0.7
}
```

The token <<PROMPT>> is the injection point. During a scan the fuzzer walks a dataset, substitutes each entry, sends the request, and classifies the answer. Datasets come from two places: CSV files with a prompt column placed in the working directory, and Hugging Face datasets referenced by name. The startup log in the README shows the local CSV path, reporting which files were found and warning when a file lacks a prompt column.

The registry is visible through the ls subcommand, which prints a table of dataset name, prompt count, source and modality. That table is the honest inventory of what a scan will cover, and it is the first thing to read before drawing conclusions from a result. A run against a 416-prompt jailbreak set and a run against an 11,191-prompt set are different experiments, and the README shows both kinds of entries side by side.

Scoring is threshold-driven rather than absolute. The generated configuration carries max_th alongside a thresholds block with low, medium and high values. Those numbers, not the raw response text, decide whether a scan is reported as a problem. That is a design choice with a cost: two teams can scan the same model and disagree about the result purely because they edited different threshold values.

## Installing agentic_security and running a first scan

Installation is a single pip command. The README gives it directly:

```bash
pip install agentic_security
```

Running the bare command starts a Uvicorn server. According to the README output, it binds to http://0.0.0.0:8718 and serves a web UI, which is where you configure the target and launch scans. You can also start it as a module or override the bind address:

```bash
python -m agentic_security
agentic_security --port=PORT --host=HOST
```

For a first real use, generate the configuration file before anything else. The init subcommand writes a default agesec.toml:

```bash
agentic_security init
```

The generated file is where the actual work happens. It contains a general section with llmSpec, maxBudget, max_th, optimize and enableMultiStepAttack, plus a modules section mapping dataset names to modules and a thresholds block. Edit llmSpec to point at your own endpoint and insert your credentials in the Authorization header. Then list what is available:

```bash
agentic_security ls
```

That prints the dataset registry table. Confirm the modules you care about are present and note their prompt counts. When you are ready, run the CI-oriented scan:

```bash
agentic_security ci
```

The README shows this command loading agesec.toml, resolving each module, and then scanning. Expect a progress line per module and a summary at the end. If a configured dataset fails to resolve, the log is where you will see it, not the final score.

## Where agentic_security stops being the right tool

The scanner assumes you can express your target as an HTTP request with a single substitution point. If your agent takes structured input, calls tools, or maintains multi-turn state, a single <<PROMPT>> slot cannot represent it. The configuration exposes enableMultiStepAttack, which the README describes as enabling multi-step attack simulations, but the documentation does not explain how a multi-step sequence maps onto an HTTP spec that carries one prompt field. Treat that flag as unverified until you read the source.

The README also marks the section on adding LLM integration templates as TBD. So the documented path is the raw HTTP spec, and anything more elaborate is undocumented. That is a real constraint for teams whose models sit behind an SDK rather than a REST endpoint.

Budget is the other hard limit. maxBudget defaults to 1000000 in the sample configuration. A scan against a large dataset will consume tokens and money, and the README does not document what happens when the budget is exhausted mid-scan, nor whether partial results are kept. There is no documented rollback or resume behaviour. If you run this against a production endpoint, set maxBudget deliberately and expect to pay for the scan.

The Dockerfile builds on python:3.14-slim and defines a health check against http://localhost:8718/health, so containerised deployment is supported. pyproject.toml requires Python >=3.14,<4.0, which is an unusually narrow floor and will rule out environments pinned to older interpreters.

## agentic_security versus garak and general-purpose fuzzers

The closest comparison in the same problem space is garak, the LLM vulnerability scanner. The pyproject.toml in this repository lists garak as a commented-out optional dependency, which suggests the author knows the overlap. The practical difference is how you describe a target. Garak is built around its own model interface layer and ships a large probe library with its own conventions. Agentic Security takes a raw HTTP spec as the primary abstraction, so anything reachable over HTTP with a bearer token can be scanned without writing an adapter.

That is a genuine advantage for internal services and self-hosted models, and a disadvantage for hosted APIs where an existing integration already exists. If your model is a well-known commercial endpoint, a tool with a maintained connector will get you running faster than hand-writing an HTTP spec.

A generic HTTP fuzzer is the other alternative, and it is the wrong comparison. A fuzzer mutates bytes; agentic_security mutates prompts using curated jailbreak and adversarial datasets, and it reports against a security threshold rather than a crash. The datasets are the product. If you already have a prompt corpus and only need delivery and scoring, a general load-testing tool plus your own classifier would cover the same ground with more assembly required.

## Maintenance, licensing and the cost of upgrading

The repository is not archived and the last push was on 2026-09-07, so the project is being worked on. The release cadence visible in the published releases is uneven. 0.7.3 and 0.7.4 landed eight days apart in May 2025, and 0.7.5 arrived on 2026-06-03, roughly a year later. There is no published support window or deprecation policy in the README.

The dependency list is heavy and will drive most of your upgrade cost. FastAPI, Uvicorn, pandas, Hugging Face datasets, scikit-learn, scikit-optimize, matplotlib, Anthropic and OpenAI SDKs all appear in pyproject.toml. A scanner that pulls in matplotlib and scikit-optimize is doing more than sending HTTP requests, and each of those packages carries its own upgrade cycle. The Python >=3.14 floor compounds this: you are tracking a recent interpreter plus a wide dependency graph.

On licensing, the project is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant. That is a permissive licence and the practical implication is that you can embed the scanner in an internal pipeline without publishing your changes. Apache-2.0 does require you to preserve notices and state significant changes if you redistribute. This is a description of the licence text, not legal advice; check with your own counsel if redistribution is on the table. Note that the datasets pulled from Hugging Face carry their own licences, which the README does not discuss, and that is the licensing question most likely to surprise you.

## Conclusion

Adopt agentic_security if you already have an HTTP endpoint for your model or agent and want a repeatable jailbreak and fuzzing pass rather than a one-off manual probe. Skip it if you need a hosted platform with SLAs, or if your target is a closed API you cannot point a scanner at. Before trusting the output, run agentic_security init, read the generated agesec.toml, confirm maxBudget and max_th match your risk appetite, and verify which dataset modules actually loaded. The threshold values low, medium and high in that file decide what counts as a failure, so they are the first thing to change.

## FAQ

### What is agentic security?

In this project, it is an open-source vulnerability scanner for agent workflows and large language models, distributed on PyPI as agentic_security under Apache-2.0. It probes LLM endpoints with jailbreak, fuzzing and multimodal attack datasets and reports results against configurable thresholds.

### How do I secure agentic AI with agentic_security?

Point the scanner at your model by editing the llmSpec field in agesec.toml so its <<PROMPT>> placeholder targets your endpoint, then run agentic_security ci. It measures how often your existing defences fail; it does not filter live traffic, so it complements runtime guardrails rather than replacing them.

### What are the dangers of agentic AI that agentic_security looks for?

The README names jailbreaks, fuzzing and multimodal attacks across text, images and audio, and the repository topics list prompt injection, prompt leakage and the OWASP LLM Top 10. The scanner ships datasets for these categories and reports a failure rate against the thresholds block in agesec.toml.

### Can you give an example of an agentic system that agentic_security can scan?

The README's own example is an HTTP chat completions request with a messages array and a <<PROMPT>> placeholder. Any target expressible as a single HTTP request with one substitution point fits the documented model; the README does not document how to scan a target that needs structured or multi-turn input.

### What does agentic stand for in agentic_security?

The README and pyproject.toml do not define the term. The project describes itself as a scanner for agent workflows and LLMs, and the topics include agent-framework and agent-security, but no expansion of the word appears in either file.

## Sources

- [License: Apache-2.0](https://github.com/msoedov/agentic_security/blob/main/LICENSE)
- [msoedov/agentic_security on GitHub](https://github.com/msoedov/agentic_security)
- [Project website](https://agentic-security.vercel.app)
- [README](https://github.com/msoedov/agentic_security/blob/main/README.md)
- [Releases](https://github.com/msoedov/agentic_security/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/msoedov-agentic-security