agentic_security: a prompt-dataset scanner you drive from a TOML file
Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪
At a glance
- What is it?
- Agentic Security is an Apache-2.0 Python scanner that fires jailbreak and fuzzing prompt datasets at an LLM endpoint defined in a plain HTTP spec, then gates a CI run on a failure threshold. It is a harness for people who already have an endpoint and a dataset, not a hosted product.
- Who is it for?
- Adopt it if you already have an HTTP endpoint that accepts a prompt field and you want jailbreak and fuzzing datasets replayed against it from a CI job with a numeric failure gate. Do not adopt it if you need a hosted dashboard, a fixed pass/fail verdict from the vendor, or a scanner that understands your agent's tool-call graph rather than its final text response.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 5 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The endpoint-shaped hole in LLM security testing
Most teams shipping an LLM feature can describe their endpoint and cannot describe their attack surface. Agentic Security targets that gap. It is a Python package that takes a list of prompts from a dataset, substitutes each one into an HTTP request template, sends it to whatever endpoint you name, and records which responses look like failures. The README frames the goal as protecting AI systems from jailbreaks, fuzzing and multimodal attacks, and lists multi-step jailbreaks, comprehensive fuzzing, API integration and stress testing, and RL-based attacks as the feature set. The audience implied by the workflow is developers, researchers and security teams who own an endpoint and want repeatable probing rather than a one-off manual session. Nothing in the material suggests the project hosts your data or runs the target model for you; the target is always an address you supply.
llmSpec is the whole integration surface
The core mechanism is a text block the README calls a plain text HTTP spec. You write the method, URL and headers, then a JSON body containing the literal token <<PROMPT>>. The README's example posts to https://api.openai.com/v1/chat/completions with an Authorization: Bearer header, a model field, a messages array containing a user message whose content is <<PROMPT>>, and a temperature. During a scan, the scanner replaces <<PROMPT>> with the current attack vector. That is the entire contract between the tool and your system. It does not need an SDK, a specific vendor, or a schema. It needs an HTTP endpoint that accepts a prompt somewhere in a request body. The consequence is that anything speaking HTTP is in scope, including a local server the project itself can start, and the same consequence means the tool has no idea what your endpoint does with the prompt. It cannot see tool calls, retrieved documents, or downstream side effects. It sees a response body and classifies it, and the material does not document the classifier in detail, which is the first place a careful reader should slow down.
Datasets arrive as CSV rows or Hugging Face names
Prompts come from two places. Local CSV files with a prompt column are loaded at startup; the README's log line shows the loader finding one CSV file and naming it prompts.csv, and a later example shows it finding two files and warning that issues_with_descriptions.csv does not contain a prompt column. That warning is useful behaviour: a malformed file is reported rather than silently skipped. The second source is Hugging Face Datasets, referenced by name in the config. The agentic_security ls command prints a Dataset Registry table with columns for Dataset Name, Num Prompts, Tokens, Source, Selected, Dynamic and Modality, and the sample output shows entries such as simonycl/aya-23-8B_advbench_jailbreak with 416 prompts and acmc/jailbreaks_dataset_with_perp... with 11191 prompts, both marked as Hugging Face Datasets, text modality, and Selected set to false. The Modality column implies the multimodal claim is expressed per dataset rather than per scanner mode. The Selected column implies you choose which registered datasets participate, though the README does not show the flag that flips it.
Running a scan: init, ls, ci
Installation is one command, pip install agentic_security. Running the bare binary starts a Uvicorn server on http://0.0.0.0:8718 by default, and the README also gives python -m agentic_security, agentic_security --help, and agentic_security --port=PORT --host=HOST. For CI use the flow is three commands. agentic_security init writes a default agesec.toml, and the log line confirms the filename. agentic_security ls prints the dataset registry table so you can see what is available before committing to a module list. agentic_security ci loads agesec.toml, prepares the prompts, and scans the listed modules. The sample config has a [general] table holding llmSpec, maxBudget, max_th, optimize and enableMultiStepAttack; a [modules.<name>] table per dataset with a dataset_name key and an optional [modules.<name>.opts] sub-table; and a [thresholds] table with low, medium and high values of 0.15, 0.3 and 0.5. The default llmSpec in the generated config points at http://0.0.0.0:8718/v1/self-probe, which is the scanner probing itself, so the first thing you edit is that block.
max_th is the gate, and the material does not define the denominator
The config comment describes max_th as the maximum failure threshold percentage and the sample sets it to 0.3. The [thresholds] table separately labels 0.3 as medium. A CI job that exits non-zero when the failure rate crosses max_th is the obvious intent. What the material does not state is how a failure is counted: whether a refusal counts as a pass, whether a partial compliance counts as a failure, or who decides. Any team relying on this gate in a release pipeline needs to read the scanner source or run it against a known-good and known-bad endpoint and compare the reported numbers. A threshold whose numerator is defined by an undocumented classifier is a number you cannot defend in a review. This is the single largest gap between what the README promises and what an adopter can verify from the README alone.
Where it stops being the right tool
The scanner talks to one endpoint over HTTP and judges the response text. That design excludes a large class of real incidents. If your agent calls a tool that writes a file, sends an email, or queries a database, a jailbreak that produces a harmless-looking final answer while triggering a destructive tool call will not be visible to a scanner that only reads the response body. Multi-step attack simulation is behind an enableMultiStepAttack flag that the default config sets to false, and the README does not document what enabling it changes in the request sequence. The optimize flag is likewise named and not explained. The README also carries a section titled Adding LLM integration templates whose body is simply TBD, so the documented path for integrating a non-OpenAI-shaped API is the raw llmSpec block and nothing more. Budget control exists via maxBudget, but the README does not say whether it counts tokens, requests or currency, which matters before you point it at a paid endpoint with an 11191-prompt dataset.
What a general-purpose fuzzer does differently
A conventional HTTP fuzzer such as ffuf or a property-based testing library generates inputs from a grammar or a mutation strategy and looks for crashes, timeouts and status-code anomalies. Agentic Security inverts that. Its inputs are curated prompt datasets, often published jailbreak corpora, and its signal is the content of a model response rather than a transport error. That inversion is the reason to pick it: a generic fuzzer will never contain an advbench jailbreak prompt, and it has no concept of a refusal. The cost of the inversion is that the tool is only as good as its datasets and its response classifier, and it cannot find a class of bug that has nothing to do with prompt content, such as an authentication flaw in the endpoint you point it at. If your concern is transport and input validation, a general fuzzer is the better instrument. If your concern is whether a specific model or system prompt holds up against known jailbreak families, the dataset-driven approach is the one that maps to the question.
Licence, releases and the cost of keeping up
The repository is Apache-2.0, which permits commercial use and modification with the usual notice and patent-grant terms; that is a summary of the identifier, not legal advice, and anyone embedding it in a product should read the licence text. Release cadence visible in the material is uneven: 0.7.3 in May 2025, 0.7.4 later that month, and 0.7.5 in June 2026, with the last push to the default branch in September 2026. That pattern suggests maintenance is active but not on a schedule you can plan a quarterly upgrade around. The practical upgrade cost is low if you stay inside the config surface, because your integration lives in agesec.toml and your datasets live in CSV files or Hugging Face names. It rises if you depend on the undocumented parts, namely the failure classifier, the optimize and enableMultiStepAttack flags, and the response parsing for non-OpenAI APIs. Pinning the version in your CI image and diffing agesec.toml between upgrades is the cheap insurance.
Editorial conclusion
Adopt it if you already have an HTTP endpoint that accepts a prompt field and you want jailbreak and fuzzing datasets replayed against it from a CI job with a numeric failure gate. Do not adopt it if you need a hosted dashboard, a fixed pass/fail verdict from the vendor, or a scanner that understands your agent's tool-call graph rather than its final text response. Before wiring it into a pipeline, run agentic_security init, point the llmSpec block at your endpoint, and check that the modules you list actually load prompts and report failures, because a dataset registered with zero prompts will still appear in agentic_security ls.
Community notes