ptai: an AI pentest driver that makes oracles prove each finding before it ships
Open-source AI pentester that proves every finding. Machine oracles re-run each exploit; verified bugs ship a proof capsule you can replay yourself.
At a glance
- What is it?
- ptai (0xSteph/pentest-ai) is an MIT-licensed Python tool that lets an LLM drive a web app engagement while machine oracles re-run each exploit N/N times against a control that must fail. This article covers the verification model, the install paths, the probe and oracle counts, and where the project is honestly thin.
- Who is it for?
- Adopt ptai if you run authorized web app tests and want an LLM-driven hunt whose output is gated by named oracles rather than a chat log; the MIT licence and the --fail-on verified CI flag make it usable as a gate. Skip it if your target is mostly Active Directory, cloud, or UI-only work, because the README states 64 of 65 probes target HTTP and only one replayable AD class exists.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 3 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem ptai targets: scanner output that nobody wants to triage
A conventional web scanner returns a list, and someone has to decide which rows are real. The README states the project's own framing of this: "A scanner that dumps 500 'possibles' is a second job." The benchmark table in the README compares ptai against ZAP 2.17.0, Nuclei 3.8.0, and HexStrike v6.0 on the same OWASP Juice Shop instance, and reports a 47% false positive rate for ZAP against 0% for ptai. That table carries an explicit caveat in the README itself: n=1, and it is a ptai 0.13.0 sweep. Treat the numbers as a directional claim from the maintainer, not as a reproduced result.
The people this is aimed at are not beginners. The install path assumes you already have a model client (Claude Code, Cursor, or Codex) or an API key, and the responsible-use section requires written authorization for every target. This is tooling for a pentester, an appsec engineer, or a bug bounty hunter who already knows what BOLA and alg:none mean and is tired of hand-checking whether a scanner's row is real. The scope guard is enforced at tool-invocation time, so out-of-scope hosts are refused rather than merely warned about.
How the oracle gate actually works, and why the model cannot rubber-stamp itself
The pipeline in the README runs recon into auth, then branches into web, ad, cloud, and api, writing into a scope-guarded findings DB. From there it goes to verify, then chain, validate, detect, and report. The README is direct about the division of labour: "The LLM drives. The probes detect. The oracles prove." Swapping the model does not change detection.
The mechanism that matters is what happens between a candidate and a VERIFIED badge. Every finding stays a candidate until a named machine oracle re-runs the exploit N/N times against a control that must fail. The README gives three concrete shapes for that. A trusted-header bypass has to leak privileged content with the header present and deny without it. A leaked credential has to authenticate as the real secret and fail as a corrupted twin. An endpoint that returns 200 to everything earns nothing, which is the failure mode that makes naive scanners noisy. Third-party scanner output stays unverified until one of those oracles re-proves it.
Each verified finding ships a portable proof capsule, and the README is explicit that capsules are unsigned and that ptai replay is how you decide whether to believe one. That is a deliberate design choice: the capsule is evidence to inspect, not a signed attestation. If you need cryptographic provenance for a report, this is not it.
Install paths: MCP client, standalone CLI, or the bundled demo
The primary path drops ptai into an existing model client, which then supplies the LLM. Two commands, per the README:
pip install ptai ptai setup --mcp
After a client restart the README states 52 tools are available, and the first action is start_engagement against an authorized target. The standalone CLI path uses your own key:
pip install ptai export ANTHROPIC_API_KEY=sk-... ptai start https://target.example.com
OPENAI_API_KEY is listed as an alternative. Spend is capped at $10 per engagement by default through PTAI_PRICE_LIMIT. A fully local model is selectable with PENTEST_AI_LLM_PROVIDER=ollama. If you want to see the tool work before pointing it at anything, pip install ptai && ptai demo runs a bundled app that prints findings, replays a proof capsule, then runs the same routes hardened and prints zero.
For CI, the README shows a GitHub Actions snippet where ptai start runs with --ci --fail-on verified --sarif pentest.sarif, and the SARIF file is uploaded with github/codeql-action/upload-sarif@v3. The point of --fail-on verified is that the build fails only on an oracle-proved finding. GitLab and Jenkins templates are referenced in docs/ci-cd.md. ptai setup --tier recommended installs the scanner binaries, ptai serve exposes a REST/WebSocket surface, and ptai playbook run prints the plan without executing it.
The guardrails are off by default, which is the first thing to change
The README lists three guardrails and states plainly that they are off by default: intensity=safe skips state-mutating probes, respect_rate_limits honours 429 responses, and strict_scope refuses off-host requests. Off-by-default is a defensible choice for a tool whose users are expected to be professionals running authorized tests, but it means the default configuration will mutate application state and follow redirects off-host if the target does so. If you point ptai at anything shared, set those three before the first run rather than after.
Authorization is handled separately. You need written authorization for every target, and out-of-scope hosts are refused at tool-invocation time. The first run stores AUP consent, and CI bypasses the prompt with PENTEST_AI_AUP_ACCEPTED=1. Installing the package accepts the AUP and Terms linked from the README, so the consent record is not a formality you can skip in a pipeline.
Where the coverage is thin: probes, oracles, and wrappers do not line up
The numbers table is the most useful part of the README because it publishes its own gaps. 65 probes exist in the library, but only 34 can earn a VERIFIED verdict. 64 of the 65 probes target HTTP, and a single probe, ad.asrep_roast, talks Kerberos. 203 tool wrappers are registered, but only 20 turn tool output into findings. There are 28 oracle kinds across 18 vulnerability classes with a working oracle, 52 MCP tools, and 18 specialist agents.
Read together, those figures describe a project whose verification depth is concentrated in web. The README says as much: "web still holds the HTTP library, AD now has one replayable class." The 1.4.0 scoreboard covers 81 of 90 in-scope HTTP challenges on a clean OWASP Juice Shop, and the README notes that OSINT, Web3, and UI-only keys stay off that board. So the headline score is a web score. If your engagement is mostly Active Directory, cloud, or client-side work, the oracle-backed portion of it is small, and the 203 registered wrappers mostly will not convert their output into findings. That is a real limitation, not a roadmap item to wait on.
The benchmark table has a second honesty problem the README acknowledges: the ptai row is from a 0.13.0 sweep while the current release is 1.4.0, and the table scores raw findings while 1.4.0 scores challenges. Those are different units, so the table and the scoreboard should not be read as one continuous trend line.
How it differs from ZAP and Nuclei in approach, not just in numbers
ZAP and Nuclei are the obvious comparison points because the README benchmarks against both. The difference is not the vulnerability list. ZAP 2.17.0 is a proxy-driven scanner: it crawls and fuzzes, applies passive and active rules, and emits everything the rules matched. The README's table records 593 findings with a 47% false positive rate, which is the expected shape of that design. Nuclei 3.8.0 is template-driven: you write or import a YAML template describing a request and a matcher, and it reports what matched. The README records 1 finding and 0% false positives on that box, which is the opposite failure mode. A template either matches or it does not, and a narrow template set finds little.
ptai inserts a model between the target and the probe library to choose what to try next, then requires an oracle to independently reproduce the result before it counts. That is a third position: the detection surface is chosen adaptively, but the verdict is deterministic. If you already have a mature Nuclei template set tuned to your stack, ptai does not replace it, because third-party scanner output can be fed in and stays unverified until an oracle re-proves it. If you want a scanner whose entire rule set is auditable line by line, ZAP or Nuclei give you that and ptai does not, because the probe selection is model-driven.
Maintenance cost, licence, and what to check before you commit
The licence is MIT, and the README states the tool is free forever, with the commercial offering being delivered pentests and a hosted workspace at pentestai.xyz. MIT means you can fork, vendor, or embed it without a copyleft obligation on your own code. That is not legal advice, and the README does point to a separate AUP and Terms that govern use of the tool itself, which is a different question from the source licence.
Maintenance cost is where the numbers table bites. A project with 65 probes, 34 verifiable probes, 203 wrappers, and 20 productive wrappers carries surface area that has to track upstream tool output formats. The wrapper-to-finding ratio of 203 to 20 is the number to watch across releases: if it does not move, the integration layer is mostly registration rather than parsing. Release cadence is visible from the tags, with v1.3.0 and v1.3.1 on the same day and v1.4.0 about two weeks later, so expect frequent small releases rather than long stable plateaus. Pinning a version in CI is the low-effort hedge.
Before committing, verify three things against your own scope. Run ptai demo and inspect the replayed capsule to confirm the oracle output is what you expect evidence to look like. Run ptai start on a staging URL with strict_scope, intensity=safe, and respect_rate_limits enabled, and check whether the findings your team already knows about come back VERIFIED or as candidates. Then read the 34 verifiable probes against your stack, because those are the only ones that can produce a badge your CI gate will act on.
Editorial conclusion
Adopt ptai if you run authorized web app tests and want an LLM-driven hunt whose output is gated by named oracles rather than a chat log; the MIT licence and the --fail-on verified CI flag make it usable as a gate. Skip it if your target is mostly Active Directory, cloud, or UI-only work, because the README states 64 of 65 probes target HTTP and only one replayable AD class exists. Before trusting it on your own scope, run ptai demo, then ptai start against a staging URL with strict_scope and intensity=safe, and read the proof capsule with ptai replay to confirm the oracle actually re-runs the exploit rather than restating the model's claim.
Community notes