Rogue: an evaluator and red team harness for AI agents
AI Agent Evaluator & Red Team Platform
At a glance
- What is it?
- Rogue runs two kinds of tests against a deployed agent: policy evaluation against expected behaviour, and adversarial red teaming with CVSS-style scoring. It speaks A2A, MCP and plain Python, and ships as a Go TUI plus a non-interactive CLI for CI.
- Who is it for?
- Adopt Rogue if you already have an agent reachable over A2A or MCP, or a Python call_agent function you can point a file path at, and you want policy regression checks plus adversarial scans from the same tool. Skip it if your agent only speaks a bespoke HTTP schema, since the documented protocols do not cover that, and skip it if you cannot supply a judge LLM key, because --judge-llm is listed as required.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 43 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap Rogue is aimed at: agents that pass unit tests and still misbehave
An agent that returns well-formed JSON can still ignore a refund policy, leak a system prompt, or hand a user more authority than the business rules allow. Rogue is built for that gap. The README frames the two halves plainly: automatic evaluation checks the agent against business policies and expected outcomes, and red teaming simulates adversarial attacks to find vulnerabilities. The first is regression testing with a judge model in the loop; the second is closer to a penetration test.
The intended user is an engineer who owns an agent endpoint and needs evidence about its behaviour, not a researcher studying model internals. The README's own split is explicit: automatic evaluation is described as best for regression testing, behaviour validation and policy compliance, while red teaming is aimed at security audits, penetration testing and compliance reporting. If you have no deployed agent to point at, there is nothing here for you yet.
Client-server split, and the three ways Rogue reaches your agent
Rogue is a client-server system. The server holds the evaluation and red team logic. Two clients sit on top: a TUI written in Go with Bubble Tea, and a CLI for non-interactive use in CI/CD pipelines. That split matters operationally, because it means the TUI and the CI job drive the same backend rather than two divergent implementations.
Reaching the agent under test happens over one of three protocols. A2A uses HTTP and follows Google's Agent-to-Agent protocol. MCP is supported over SSE and STREAMABLE_HTTP transports, and the README notes it is invoked through a send_message tool. The third option is Python: a direct function call with no network protocol involved. For that path you write a file exposing a call_agent function that takes a list of messages with role and content keys and returns the agent's response as a string. The README supplies a stub that echoes the last user message, and points to examples/python_entrypoint_stub.py as a fuller reference.
The Python entrypoint is the most interesting design choice here. It sidesteps protocol work entirely, which lowers the cost of trying Rogue on an agent that was never built to be tested. The trade-off is that you are now testing a function, not the deployed service, so anything in the real transport layer (auth, serialisation, rate limits) falls outside the test.
Getting a first scan running with uvx
Rogue is distributed for execution through uvx, so the prerequisites are uvx itself, Python 3.10 or newer, and an API key for OpenAI, Anthropic or Google. The README recommends the TUI as the default entrypoint:
uvx rogue-ai
For CI, the non-interactive form is uvx rogue-ai cli. There is also a self-contained demo that starts both Rogue and a sample T-shirt store agent in one command: uvx rogue-ai --example=tshirt_store. The README instructs you to point the agent URL at http://localhost:10001 and pick either Automatic Evaluation or Red Teaming in the UI.
The CLI carries the configuration that matters. The documented options are --config-file for a config JSON, --evaluated-agent-url for the agent endpoint (marked required), --judge-llm for the evaluation model (also required), --business-context or --business-context-file for the policy text, --input-scenarios-file for scenarios in JSON, and --output-report-file for the report path. A representative invocation from the README passes the agent URL, a judge model such as openai/gpt-4o-mini, and a business context file at ./.rogue/business_context.md.
The Python protocol adds two more flags: --protocol python and --python-entrypoint-file pointing at your module. Note that the judge LLM is a separate model from the agent under test, and it is doing the grading, so its quality bounds what the pass/fail reports can tell you.
Inside a red team scan: categories, techniques and the 0-10 score
The red team side is the more developed half on paper. The README claims 75 or more vulnerabilities across 12 security categories, paired with 20 attack techniques. Attack categories listed are encoding (Base64, ROT13, leetspeak), social engineering (roleplay, trust building), injection (prompt injection, SQL injection), semantic (goal redirection, context poisoning) and technical (gray-box probing, permission escalation).
Scans come in three sizes. Basic covers 5 curated vulnerabilities with 6 attacks and is quoted at roughly 2 to 3 minutes. Full covers 75 or more vulnerabilities with 40 or more attacks and is quoted at roughly 30 to 45 minutes. Custom lets you select both. Those durations are the README's estimates; they will move with your agent's latency and the judge model you choose.
Each finding gets a 0-10 risk score in the CVSS style, computed from four inputs the README names: impact, exploitability, human factor and complexity. Mapping findings onto compliance frameworks is a first-class feature rather than an afterthought: OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF, ISO/IEC 42001, the EU AI Act, GDPR and OWASP API Top 10 are all listed. The README also mentions reproducible scans via random seeds, though the supplied text cuts off mid-sentence at that point, so the seed flag name cannot be confirmed from this material.
Where Rogue is the wrong tool
The protocol list is the first hard boundary. A2A, MCP and a Python function are the documented paths. If your agent is exposed only through a bespoke REST schema with its own request shape, none of the three applies directly, and your options are to wrap the agent in a Python call_agent shim or to stand up an A2A or MCP facade. Both are real work, and the shim approach means the thing under test is your wrapper, not your production endpoint.
The second boundary is cost and nondeterminism. Every evaluation and every red team run needs a judge LLM, which means a per-run bill and a per-run source of variance. A Full scan is quoted at 30 to 45 minutes, so it does not fit a per-commit gate in most pipelines. The Basic scan at 2 to 3 minutes is the more plausible CI candidate, but the README does not describe a pass/fail threshold policy for CI, so you would be parsing --output-report-file yourself to decide whether a build should fail.
The third is the state of the material. The README is truncated mid-sentence in the section on reproducible scans, the repository carries a NOASSERTION licence rather than a recognised identifier, and there is no changelog summary in the supplied data beyond three patch releases in April 2026. None of that makes the tool unusable, but it does mean you are reading a project whose documentation is still being filled in.
How Rogue differs from pytest-style agent testing
The obvious alternative is writing your own harness: pytest with recorded conversations, assertions on the agent's output, and a separate script for adversarial prompts. That approach is deterministic, cheap, and lives next to the rest of your test suite. Its weakness is coverage and maintenance. You have to invent the adversarial cases yourself, and prompt injection techniques move faster than most teams' test files.
Rogue's difference is that the attack corpus and the scoring model are the product. You supply an endpoint and a judge model; the tool supplies the 12 categories, the 20 techniques, the CVSS-style 0-10 scoring and the mappings to OWASP, MITRE, NIST, ISO/IEC 42001, the EU AI Act and GDPR. That is a different division of labour, and it is why the report format matters more here than in a homegrown suite: the value is in the framework mapping, not in the raw pass/fail. The cost is the judge LLM dependency and the loss of determinism that comes with it. A pytest suite gives you the same answer every time. Rogue gives you a graded answer that depends on the grader.
Maintenance surface and what the licence field does not tell you
Running Rogue through uvx means you are not vendoring a source tree; you are invoking a published package, so upgrades are a version bump rather than a merge. That keeps the maintenance cost low, but it also means the server, the TUI and the CLI move together. The release history in the supplied material shows three patch releases inside April 2026 (v0.6.2, v0.6.3, v0.6.4), with v0.6.3 and v0.6.4 landing the same day. A cadence that dense suggests active churn, and pinning a version for CI is the safer default.
The larger open question is licensing. The repository reports NOASSERTION, which means GitHub could not match the licence file to a known identifier. That is not a statement about what the licence permits; it means you cannot tell from this metadata alone. If you plan to run Rogue against client systems or fold its reports into a compliance submission, read the licence file in the repository before you build a process around it. Nothing here is legal advice, and the metadata does not resolve the question either way.
Editorial conclusion
Adopt Rogue if you already have an agent reachable over A2A or MCP, or a Python call_agent function you can point a file path at, and you want policy regression checks plus adversarial scans from the same tool. Skip it if your agent only speaks a bespoke HTTP schema, since the documented protocols do not cover that, and skip it if you cannot supply a judge LLM key, because --judge-llm is listed as required. Before committing, verify the licence terms, since the repository reports NOASSERTION rather than a named licence, and check what the default scan actually costs in tokens and wall-clock time on your own agent, because the README's time estimates (about 2 to 3 minutes for Basic, 30 to 45 for Full) are the only figures given and they are not broken down by model or provider.
Community notes