pentest-ai (ptai): an AI pentester that has to prove every finding before it reports it

The problem ptai solves is triage, not scanning

Scanner output is not a finding list. It is a queue. ZAP, in the project's own benchmark table, produced 593 findings on a single OWASP Juice Shop instance with a 47% false positive rate; Nuclei produced one. Neither number tells you what to fix. The README describes the failure mode directly: a scanner that dumps 500 possibles is a second job.

ptai's answer is to separate detection from proof. The model proposes and probes, and a finding stays a candidate until a named machine oracle re-runs the exploit N out of N times against a control that must fail. A trusted-header bypass only verifies if privileged content leaks with the header and is denied without it. A leaked credential only verifies if it authenticates as the real secret and fails as a corrupted twin. An endpoint that returns 200 to everything earns nothing.

That is the audience: application security engineers and bug bounty hunters who already have too many raw alerts and want a list they can defend in a report. It is not aimed at people who want a one-click exploit against a target they have not been authorized to touch. The README puts the authorization requirement in a blockquote near the top and states that out-of-scope hosts are refused at tool-invocation time.

How the pipeline works: LLM plans, probes detect, oracles prove

The architecture diagram in the README is a left-to-right flow. Recon feeds auth, which feeds web, which branches into ad, cloud and api. All of them write into a findings database that the README labels scope-guarded. From there the path is verify (oracles, N/N), then chain, validate, detect, and finally report in markdown, HTML, PDF, SARIF or JUnit.

The division of labour is stated plainly: the LLM drives, the probes detect, the oracles prove. The model picks the next probe and reads what came back, but it cannot promote its own guess. That is the design decision worth noting, because it means swapping models does not change what counts as a detection. The README says detection does not change if you swap models. You can run Anthropic, OpenAI, or a local model through Ollama, and the verification layer stays the same.

The numbers section is honest about how much of the library is actually wired up. There are 65 probes, but only 34 can earn a verdict, and 64 of the 65 target HTTP. There are 203 registered tool wrappers, but only 20 parse tool output into findings today. Third-party scanner output, when it comes in through a wrapper, stays unverified until one of the oracles re-proves it. Those gaps are the real shape of the tool: a deep HTTP verifier with a wide but shallow wrapper layer around it.

Installing ptai and running a first engagement

The primary install path is to drop ptai into an MCP-capable client such as Claude Code, Cursor or Codex, and let that client be the LLM. Install the package and run the MCP setup command:

pip install ptai
ptai setup --mcp

Restart the client and the README states that 52 tools are available. From there you call start_engagement against a target you are authorized to test. If you prefer a standalone CLI with your own API key, export a provider key and point ptai at a URL:

export ANTHROPIC_API_KEY=sk-...        # or OPENAI_API_KEY
ptai start https://target.example.com

Spend is capped at $10 per engagement by default through PTAI_PRICE_LIMIT. To run entirely against a local model, set the provider to Ollama:

export PENTEST_AI_LLM_PROVIDER=ollama

If you have no target of your own, the README offers a self-contained demo that prints findings, replays a proof capsule, then runs the same routes hardened and prints zero:

pip install ptai && ptai demo

The bundled demo capsule is also in the repository as ptai-demo-capsule.json, so you can inspect the artifact format without running an engagement. Optional extras exist for the surrounding surface: ptai setup --tier recommended installs the scanner binaries, ptai serve exposes a REST and WebSocket API, and ptai playbook run prints the plan. A docker-compose file provides profiles for the MCP server, a dev mount with hot reload, an interactive menu, and a serve profile on port 8888 with a dashboard at /dashboard/.

The CI gate is the part most teams will actually feel

The README's CI example is short and worth reading closely, because it is where the verification model pays off operationally. The workflow installs ptai, runs an engagement against a staging URL with the CI flag, and fails the build only on an oracle-proved finding:

- run: pip install ptai
- run: ptai start ${{ vars.STAGING_URL }} --ci --fail-on verified --sarif pentest.sarif
- uses: github/codeql-action/upload-sarif@v3
  with: { sarif_file: pentest.sarif }

The flag --fail-on verified is the interesting one. A conventional scanner gate fails on severity, which means it fails on noise, which means teams learn to ignore it or to raise the threshold until it catches nothing. Gating on the verified set inverts that: the build breaks only when an oracle has re-run the exploit and the control failed. The output is SARIF, so it lands in the same code scanning view as everything else. GitLab and Jenkins templates are referenced under docs/ci-cd.md.

The cost of this design is coverage. Only 34 probes can earn a verdict, so a build gate set to verified will miss anything those probes cannot prove. That is a deliberate trade: fewer findings, each one defensible.

Where ptai is the wrong tool

Read the counts before you read the marketing. The README states that 64 of the 65 probes target HTTP, and that the single non-HTTP probe, ad.asrep_roast, talks Kerberos. Active Directory has one replayable class. Cloud and API appear as branches in the architecture diagram, but the probe distribution says where the depth is. If your engagement is primarily AD, cloud posture, or a non-web protocol, ptai is not the tool for the job yet, and the repository says so in its own numbers.

The second limitation is the wrapper layer. 203 tool wrappers are registered but only 20 turn tool output into findings. Wrapping a tool is not the same as understanding it. If you were hoping ptai would orchestrate your existing scanner suite and verify all of it, the parse coverage is roughly a tenth of the registered surface.

The third is the benchmark itself. The comparison table is labelled n=1 and attributes the sweep to ptai 0.13.0, not the current release. A single Juice Shop instance is a useful sanity check and a weak basis for generalization. The 1.4.0 figure of 81 of 90 in-scope HTTP challenges is a different measurement on a clean instance, and the README notes that OSINT, Web3 and UI-only keys stay off the board. Treat these as scoped claims, not as a general accuracy rate.

Finally, the guardrails that reduce risk are off by default. intensity=safe, which skips state-mutating probes, respect_rate_limits, which honours 429 responses, and strict_scope, which refuses off-host requests, are all described as worth turning on. A default run can mutate state on your target. On a staging environment that is fine; on anything shared, it is not.

How ptai differs from PentestGPT and from plain scanners

The obvious comparison is PentestGPT, which people also search for by name. The difference in approach is where the trust sits. A conversational pentest assistant produces a transcript: the model reasons about the target, suggests commands, and you decide what is real. ptai keeps the model in the planner role but inserts a deterministic layer between the model's guess and the report. The output is not a chat log; the README makes that contrast explicitly. In practice that means you can hand a ptai report to someone who was not in the session and they can re-run the proof.

The comparison with ZAP and Nuclei is a different axis. Those are signature and fuzzing engines: deterministic, fast, no model in the loop, and in the benchmark table they produced 593 and 1 findings respectively on the same box. ptai is slower and costs money per engagement, with a default cap of $10, in exchange for a smaller set of findings that carry a control. If you want breadth and you have a human to triage, a scanner plus a person is cheaper. If you want a report where every line has been re-executed against a negative control, that is what the oracle layer buys.

The proof capsule is the concrete artifact that separates it. Every VERIFIED finding ships a portable capsule, and the README is candid that capsules are unsigned. That matters: ptai replay is how you decide whether to believe a capsule, not a signature check. Anyone can hand you a capsule, so the replay step is the trust boundary, not the file.

Licence and the cost of keeping up with releases

ptai is MIT licensed, and the README states it is free forever. MIT is permissive: you can use it commercially, modify it, and ship it inside a product, provided the copyright notice and licence text travel with it. The repository also carries a SECURITY.md and a .gitleaks.toml, and installing is described as accepting an acceptable use policy and terms hosted at pentestai.xyz. Those are separate from the code licence and are worth reading before you install, because the AUP is what governs how you are permitted to point the tool at things, not the MIT grant. None of this is legal advice; if the AUP terms matter to your organization, have someone read them.

On maintenance, the last push to main was on 2026-09-05, and the most recent tagged release, v1.4.0, dates from 2026-09-02. The project is not archived. The release cadence visible here is three releases across roughly two and a half weeks in August and September 2026, which suggests active work, though a short window is a short window.

Upgrade cost is shaped by two things in the repository. First, the Python requirement is >=3.10,<3.15, and pyproject.toml carries a comment saying the upper bound tracks the newest interpreter CI actually tests and should be lifted in lockstep when the CI matrix gains 3.15. If you run a newer interpreter, pip will refuse the install until that bound moves. Second, the dependency list is not small: impacket, scapy, paramiko, bloodhound, fastmcp and cryptography all come in with the base package, so a fresh install pulls a meaningful amount of offensive and network tooling. The docker-compose file keeps engagements in a mounted ./engagements directory and the dev profile sets PENTEST_AI_DB_PATH to /data/pentest_findings.db, which is the practical answer for keeping history across upgrades. The CHANGELOG.md at the repository root is where the actual breaking changes will be listed; the README does not document a rollback path.

Editorial conclusion

Adopt ptai if you run authorized web application tests and you have been burned by scanner output that nobody triages. The candidate-then-oracle model is the whole point, and the CI gate (ptai start --ci --fail-on verified) is the sharpest thing in the repository. Do not adopt it as an Active Directory or cloud pentest tool: the README is explicit that 64 of the 65 probes target HTTP and that AD has a single replayable class. Before you point it at anything, confirm that the model provider is configured the way you expect (PENTEST_AI_LLM_PROVIDER, and the matching API key), that the target is in scope, and that you are prepared to read the capsule rather than the badge. The badge is a claim; ptai replay is the evidence.

Frequently asked questions

Official sources

1. 0xSteph/pentest-ai on GitHub
2. License: MIT
3. Project website
4. README
5. Releases