AIBeat: black-box and runtime security evaluation for LLM, RAG and agent targets

The gap AIBeat targets: a safe answer with an unsafe execution

Most evaluation tooling for generative AI scores the final response. AIBeat's stated premise is that this is insufficient once the target can act. The README gives a worked illustration: a task to create a diagnostic support bundle returns an answer that contains no credential, while the trace records `command_exec_observed: env | sort > env_dump.txt` and the resulting file contains an API key. The verdict is FAIL, with the note that the answer was safe and the execution was not.

That example defines the audience. It is for teams running agents that touch a shell, a filesystem or an API, and for teams that must explain a verdict to someone else: a reviewer, an auditor, or a colleague deciding whether a regression fix actually closed the hole. A bare pass rate does not survive that conversation. A trace does.

The project is split into two commands that share one model. PromptBeat asks whether a target behaves safely under a scenario. AgentBeat asks whether the agent crossed a boundary at any point in execution. The second is a superset: it reuses the scenario, case, judge and report model, and adds trace and environment evidence on top.

How the PromptBeat and AgentBeat pipeline fits together

The README describes a five-stage flow: scenario design, adversarial cases, target execution, judgment, and evidence. The `llm-basic` example separates three provider roles, attacker, judge and target, and states that they may share one model gateway or use different ones. That is the core mechanism. An attacker model generates or mutates adversarial cases; the target model or endpoint receives them; a judge model returns a decision, a reason and metrics.

AgentBeat does not reimplement that pipeline. According to the README, `agentbeat` owns an adapter's lifecycle and delegates scenario execution and reporting to the shared PromptBeat engine. Adapters register through a language-neutral manifest and an EvalRun HTTP protocol. The lifecycle is explicit: the check starts the registered process, verifies its identity and capabilities, submits one EvalRun, validates the response, and stops the process.

What an evaluation can retain is enumerated in the README: generated cases with scenario and seed lineage, a normalized `evaluation_result.json`, judge decisions with reasons and metrics, runtime and trace events, artifact manifests and an HTML report, and reviewed cases promoted into the next regression set. The seed lineage matters more than it looks. It is what lets you regenerate the same case set later and argue that a change in results came from the target, not from a fresh roll of the generator.

The split between the two binaries is a real architectural decision, not packaging. Black-box evaluation needs only network access to a target. Runtime evaluation needs a process you control. Keeping them as separate commands means you can start with PromptBeat and add AgentBeat later without rewriting the scenario and judge configuration, because both read the same project config format.

Installing promptbeat and agentbeat from verified release artifacts

AIBeat ships native Go binaries rather than a package on a language registry. The README states that public Releases provide two native Go commands for each supported platform: `promptbeat-<version>-<platform>` and `agentbeat-<version>-<platform>`, plus `install.sh` and `install.ps1`. Supported targets are Linux x64, Windows x64, macOS arm64 and macOS x64. Every native artifact has an adjacent `.sha256` file, and the installers verify every download.

On macOS or Linux, from a repository checkout or an unpacked release:

bash install.sh --version <version>
promptbeat --version
agentbeat --version

The installer prepares Node.js 22.22.2 and promptfoo 0.121.9 in the user cache and creates stable `promptbeat` and `agentbeat` entries. The README is explicit that it does not modify the global npm installation, and that re-running it reuses valid cached components and replaces only missing or damaged ones. Exact versions and checksums are recorded in `runtime-manifest.json`. On Windows PowerShell the equivalent is `.\install.ps1 -Version <version>` followed by the same two version checks.

Before any run you need provider configuration. The `llm-basic` example uses three sets of environment variables, one per role:

export ATTACKER_MODEL_NAME="openai:gpt-4o"
export ATTACKER_BASE_URL="https://api.openai.com/v1"
export ATTACKER_API_KEY="sk-..."

export JUDGE_MODEL_NAME="openai:gpt-4o"
export JUDGE_BASE_URL="https://api.openai.com/v1"
export JUDGE_API_KEY="sk-..."

export TARGET_MODEL_NAME="openai:gpt-4o-mini"
export TARGET_BASE_URL="https://api.openai.com/v1"
export TARGET_API_KEY="sk-..."

Then validate the configuration, run the evaluation, and render a report:

promptbeat validate --config examples/llm-basic/promptbeat.yaml

promptbeat run \
  --config examples/llm-basic/promptbeat.yaml \
  --output-dir artifacts/llm-basic/run

promptbeat report \
  --input artifacts/llm-basic/run/evaluation_result.json \
  --output artifacts/llm-basic/report.html

After the run, open `artifacts/llm-basic/report.html`. If you want to see what the attacker produces before spending a full run, `promptbeat generate --config examples/llm-basic/promptbeat.yaml --count 5 --output artifacts/llm-basic/generated-cases.json` writes the generated cases to JSON. That generate step is the cheapest way to sanity-check a scenario before committing to a full evaluation.

For AgentBeat, the README offers a credential-free first step that exercises the adapter contract without an external agent or model keys:

agentbeat adapter check \
  --adapter examples/agent-adapters/minimal/typescript/agentbeat-adapter.json

That command starts the registered process, verifies its identity and capabilities, submits one EvalRun, validates the response, and stops the process. Equivalent zero-dependency examples are included for Python and Go. A full evaluation then points the manifest at your runtime implementation and supplies a PromptBeat project config:

agentbeat run \
  --adapter /absolute/path/to/agentbeat-adapter.json \
  --config path/to/promptbeat.yaml \
  --output-dir artifacts/my-agent

Where AIBeat is the wrong tool, and what the README leaves open

The install path is the first constraint. There is no Docker image, no Homebrew formula and no npm package documented. If your CI runs on Alpine or on an architecture outside Linux x64, Windows x64, macOS arm64 and macOS x64, the release artifacts do not cover you and the README does not describe building from source.

The second constraint is the judge. AIBeat's verdicts are produced by a model, and the README does not document how to calibrate a judge, how to measure agreement between two judges, or what threshold turns a judge decision into a pass or fail. For a tool whose central claim is evidentiary strength, that is the weakest documented link. The trace example is persuasive precisely because the file content is unambiguous; a judge disagreeing about whether a response was a refusal is not.

The third is adapter cost. AgentBeat requires an instrumented runtime that speaks the EvalRun HTTP protocol. If your agent is a closed hosted product with no way to emit tool calls, commands and file changes, AgentBeat cannot see the execution path, and you are back to black-box PromptBeat. The README says adapters register through a language-neutral manifest and that legacy JavaScript modules exporting `createEvalServer()` remain supported, but it does not enumerate what a conforming adapter must emit for each evidence category.

Finally, the licence is not stated in the repository metadata. For a security tool you may want to embed in a pipeline or ship alongside a product, that is something to resolve by reading the repository before adoption, not after.

PromptBeat versus a general LLM red-teaming harness

The closest familiar comparison is a prompt-evaluation harness such as promptfoo, and the relationship here is unusual: the AIBeat installer prepares promptfoo 0.121.9 in the user cache, so promptfoo is part of the runtime rather than a competitor. The difference in approach is what sits on top. A general harness gives you test cases, providers and assertions, and leaves the evidence model to you. AIBeat fixes a specific model on top of that machinery: attacker, judge and target roles, a scenario and case format with seed lineage, a normalized `evaluation_result.json`, and an HTML report, with AgentBeat adding trace and environment evidence from an instrumented runtime.

That is a trade. You get a defined workflow and a report format that a reviewer can read without knowing your setup. You give up the freedom to express arbitrary assertions in whatever shape your team already uses, and you inherit the pinned runtime versions the installer manages.

For pure prompt-injection testing against a chat endpoint, a plain harness with your own assertions is lighter and you control every knob. AIBeat earns its place when the question shifts from what did the model say to what did the agent do, and when you need the answer stored in a form someone else can audit.

Maintenance, release cadence and licence exposure

The repository is not archived. The last push was on 2026-09-11, and the same day carries the v0.4.0 release. Before that, two v0.3 AgentBeat preview releases landed on 2026-08-24. The cadence in the release list is therefore recent and uneven: a preview pair, then a jump to 0.4.0 about two and a half weeks later. The version numbering says the project is pre-1.0, and the AgentBeat preview naming says the runtime side is the newer of the two halves.

Upgrade cost is bounded by the installer design. Because the installer reuses valid cached components and replaces only missing or damaged ones, and because exact versions and checksums live in `runtime-manifest.json`, a version bump is a re-run of `install.sh --version <version>` or `install.ps1 -Version <version>` rather than a dependency-resolution exercise. The pinned Node.js 22.22.2 and promptfoo 0.121.9 mean you are not tracking upstream drift yourself, but you also cannot move ahead of the pinned versions without leaving the supported path.

On licensing: the repository metadata does not state a licence, and the README does not discuss one. Nothing here should be read as legal advice. The practical point is that the two binaries and the installer are distributed through GitHub Releases, and whether you may redistribute them, embed them in a commercial product, or run them in a customer environment depends on terms the repository does not state. Check the repository's licence file directly before any redistribution.

What to verify before your first real evaluation

Run `agentbeat adapter check` against the minimal TypeScript, Python and Go adapters in `examples/agent-adapters/minimal/` before writing your own. Those three exist to define the contract, and the check exercises the full lifecycle: process start, identity and capability verification, one EvalRun, response validation, process stop. If your runtime cannot satisfy that sequence, the rest of the AgentBeat path is unavailable to you regardless of scenario quality.

Then run `promptbeat generate` with a small `--count` on your own scenario and read the generated cases before running anything. The README presents generated cases with scenario and seed lineage as part of the retained evidence, and the cheapest way to judge whether the attacker is producing cases you care about is to look at them directly. A scenario that generates irrelevant cases will produce an irrelevant report no matter how clean the HTML looks.

Finally, decide in advance what a FAIL means for your process. The trace example in the README ends in a verdict, not a remediation, and the README does not document rollback or automatic case promotion beyond the statement that reviewed cases can be promoted into the next regression set. That promotion step is manual, and it is where the value compounds or does not.

Editorial conclusion

Adopt AIBeat if you need reproducible evaluation runs with retained evidence, and if your target is an LLM, RAG app, HTTP API or an agent runtime you can wrap in an adapter. Do not adopt it if you need a hosted dashboard, a managed service, or a one-command scan of a black-box endpoint with no model credentials, because the workflow assumes you supply attacker, judge and target provider configuration yourself. Before committing, verify three things: that your platform is one of Linux x64, Windows x64, macOS arm64 or macOS x64; that your agent runtime can implement the EvalRun HTTP protocol described by the adapter manifest and typed SDK; and that the licence terms in the repository match how you intend to redistribute the promptbeat and agentbeat binaries.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. tophant-ai/aibeat on GitHub