Rogue: an agent evaluator that red teams the one endpoint you name, and scores it with a required LLM judge

The target is the URL you pass, and nothing is discovered for you

This is the part that decides whether the tool is safe to point at something. Rogue does not scan, does not enumerate hosts and does not expand its own target list. The target is a single endpoint you supply, the required option in the command line table:

uvx rogue-ai cli \
  --evaluated-agent-url http://localhost:10001 \
  --judge-llm openai/gpt-4o-mini \
  --business-context-file ./.rogue/business_context.md

The example agent the quick start launches listens on port 10001, and the same value appears in the sample configuration file. There is no option for a host range, no CIDR, no exclusion list and no per-host authorization check anywhere in the documented interface, because the design does not need one. The scope problem in automated security tooling is scope the tool finds for itself, and Rogue has none to find. The one boundary that does shift is the Python protocol, where your agent is a function inside the same process, so a payload delivered by a scan executes in your interpreter rather than across a socket.

Every pass or fail verdict comes from a required judge model

The judge is not optional in the non-interactive mode: --judge-llm is marked required in the options table, and the quick start hands you an OpenAI model name as the example. Automatic evaluation is described as watching live conversations while Rogue probes the agent and then producing pass or fail reports with reasoning, which means a second model reads the transcript and grades it. Red team findings are scored zero to ten on a CVSS-like basis, and the four inputs listed are impact, exploitability, human factor, meaning the potential for manual exploitation, and complexity. None of those is a measurement taken from the target. They are judgements, so the score inherits the judge's blind spots, and the documentation does not say how the mapping from a model verdict to a number was calibrated. What the project does offer is a seed, which makes a run repeatable against the same inputs and is the right tool for checking whether a fix actually closed a finding.

The headline counts disagree with the tables underneath them

Three numbers in the feature summary do not match the detail that follows. The red teaming column claims 20 attack techniques, while the scan table gives the full scan 40 or more attacks and the basic scan six. It claims 75 or more vulnerabilities across 12 security categories, while the attack category table lists five: encoding, social engineering, injection, semantic and technical. It claims 8 compliance frameworks, while the framework list that follows has seven, counting the OWASP LLM Top 10, MITRE ATLAS, the NIST AI risk management framework, ISO/IEC 42001, the EU AI Act, GDPR and the OWASP API Top 10 separately. Techniques, attacks and vulnerabilities are different units, so some of this may be intentional, but a reader choosing between the basic and full scan is left to reconcile 20 against 40 or more on their own, and the category count has no reconciliation at all.

The documented server command binds every interface with debug on

There are four run modes, and one of them is a bare backend:

uvx rogue-ai server --host 0.0.0.0 --port 8000 --debug

That is the command as printed, with every interface and debug logging on. The documented client options are the configuration file, the evaluated agent URL, the judge model, the business context or its file, the input scenarios file and the output report path. There is no token, no key, no user list and no TLS option among them, so a server started that way answers anyone who can reach port 8000, and what it exposes is an evaluation surface that can drive conversations against the agent you pointed it at, spend your judge model's tokens and write reports. For a workstation, replacing 0.0.0.0 with 127.0.0.1 and dropping the debug flag costs nothing. If a second machine does need access, put an authenticating proxy in front of it rather than trusting the port mapping.

A Go terminal client ships inside a Python distribution under another module path

The architecture table lists three components, and one of them is not Python. The terminal interface is written in Go with Bubble Tea and lives in a packages/tui directory with its own Makefile:

cd $(TUI_DIR) && CGO_ENABLED=0 go build $(LDFLAGS) -o $(CURDIR)/$(BUILD_DIR)/$(BINARY_NAME) ./cmd/rogue

The build strips cgo, injects the version, commit and date through linker flags, and a second target cross compiles five binaries for Linux, macOS and Windows on amd64 and arm64. Two details are worth noting. The linker path is github.com/rogue/tui, an organisation and module name that does not match this repository's, so the Go component is versioned somewhere other than where the Python package is. And the dependencies target runs go mod download followed by go mod tidy, so building rewrites the module files as a side effect. The multi platform target is also missing from the list of declared phony targets, which is a small thing that makes build-all behave like a file target under some invocations.

The wheel installs the examples directory into site-packages

The build configuration names two packages for the wheel:

[tool.hatch.build.targets.wheel]
packages = ["rogue", "examples"]

and the source distribution includes the same directory, plus the version file, the readme and the build hook. So an installed Rogue brings the reference implementations with it, and those references include two sample storefront agents, one plain and one built on a graph framework, alongside the protocol examples for A2A, MCP, JavaScript and a direct OpenAI style API call. Shipping example agents is defensible for a tool whose users have to write an integration before they can test anything, and it does make the protocol contract easier to inspect. The cost is that a package which only runs a scanner now also installs a small agent codebase that does nothing unless the user points at it, which is one more thing to audit when the dependency list is long.

The version is a plain text file read by three separate build systems

There is no version string in the packaging metadata; it is declared dynamic and read from a file called VERSION at the repository root by the build hook. The Makefile reads the same file when it stamps the Go binary, falling back to the string dev when the file is absent. The README quotes none of this, but the release tags tell their own story. The three most recent are v0.6.2 on 2026-04-28, then v0.6.3 and v0.6.4 on the same morning of 2026-04-29, half an hour apart, with the older release titled Release v0.6.2 and the two newer ones titled with the bare tag. The last commit to the default branch is dated 2026-08-04, three months after the newest tag. Pin an exact version for reproducibility, and check the file rather than the tag list when you need to know what you are running.

Two places claim the SDK, and an AWS client sits in the base dependencies

The base dependency list includes a published package for the Rogue SDK, and the repository carries an sdks directory at the top level. Both cannot be the current answer, and the documentation does not say which one a new integration should depend on. The same list explains: the runtime pulls in the AWS SDK and a process inspection library, neither of which appears anywhere in the visible documentation, while a HuggingFace datasets library is pinned exactly. Next to that, the repository still carries a flake8 configuration even though the development group explains that ruff replaces flake8 along with four other tools, so the old linter's settings are still in the tree. Two smaller signals point the same way. A .rogue directory sits at the top level, which is where per-user state and the sample business context file live. And the project homepage points at a company domain rather than at the documentation, while the license is reported as absent even though a license file is committed at the root.

Editorial conclusion

Rogue fits one use well: an agent you already own, reachable at an address you already control, tested by someone who has written the business context file for it. It does not fit a use where you need targets found, authorization enforced or verdicts reproducible without a second model in the loop. Nothing here changes that: point the tool only at agents you own or agents you have written permission to test, keep the server bound to loopback unless a network peer genuinely needs it, and treat the LLM judge as part of the system under test rather than as a neutral instrument. Four things deserve checking before a first run. What the numbers mean, since the feature table, the scan table and the category table give three different counts for the same catalogue. What a passing grade is worth when a required judge model produces both the pass or fail verdict and a zero to ten score built from impact, exploitability, human factor and complexity. Where the SDK lives, since a published package and an in-repository directory both claim the name. And which version you installed, since the version comes from a plain text file that three separate build systems read, and the newest tagged release predates the last commit by months.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. rogue-security/rogue on GitHub