Pentest Harness: a plugin-based AI agent workspace for authorized security work

Who Pentest Harness is aimed at, and what it replaces

The README frames the project as an agent harness for authorized penetration tests, bug bounty research, security labs and CTF engagements. That is a narrower audience than the general coding-agent crowd. The workflow it targets is a long engagement: recon, tool invocation, note taking, and resuming a session days later without rebuilding context. Two of the stated features map directly onto that: durable sessions persisted as JSONL or SQLite with replay, and token metering with automatic compaction plus tool-result pruning. The second one matters more than it sounds. An agent that runs shell commands, reads files and does web research accumulates enormous tool output, and most chat front ends either truncate it silently or blow the context window. Pentest Harness claims to manage that explicitly. The other half of the pitch is that nothing about the model is fixed. The README lists OpenAI Chat Completions and Responses, Anthropic Messages, DeepSeek, and any custom OpenAI-compatible gateway, with one-click model auto-discovery. For a consultant who already pays for one provider and wants to swap to another mid-engagement, that is the actual selling point. The dark-only theme is stated as a deliberate choice for long sessions, which is a preference rather than a capability, but it does tell you the project is not trying to be a general-purpose desktop assistant.

The Cordis plugin layer and what being replaceable actually means

The README states the harness is built on Cordis and that model adapters, tools, sessions, settings and credentials are each replaceable from configuration. That is a stronger claim than "has a plugin API." It means the session store is not hard-wired to SQLite, and the credential store is not hard-wired to a file. Whether that holds in practice depends on the interfaces Cordis exposes, and the README does not show them. What it does show is the configuration surface: a settings file at $DSH_HOME/settings.yaml (default ~/.dsh/settings.yaml) and credentials at $DSH_HOME/.credentials.yaml with owner-only permissions. The split is deliberate. API keys are stored as references in the credential store and, according to the README, never in settings files or logs. That is the right shape for a tool that shells out and reads arbitrary files, because settings files get copied, pasted into issues and committed by accident. A credential file with restrictive permissions is a smaller blast radius. The trade-off is operational: $DSH_HOME is now a directory you have to back up, migrate and protect, and the README does not describe what happens when it is missing, unreadable or on a filesystem that does not support the permission bits it relies on. On Windows, owner-only permissions are not the same mechanism, and the README does not address the platform at all.

Getting it running: the four commands and the port flag

The quick start is short. Clone the repository, run pnpm install, run pnpm build, then pnpm dsh web. The Web UI opens at http://127.0.0.1:2323. Requirements are Node.js 20 or newer, with Node 22 recommended, and pnpm installed globally via npm i -g pnpm. If pnpm dsh is not on your PATH, the README gives two fallbacks: pnpm exec dsh web, or node apps/cli/lib/index.js web. That second path is worth noting because it tells you the CLI entry point lives at apps/cli/lib/index.js, which is a compiled artifact, so pnpm build is not optional. Running the CLI from source without building will not work. To change the port, pass --port: pnpm dsh web --port 3000. Binding is to 127.0.0.1 by default, which is the correct default for a tool that holds API keys and can execute shell commands. The README does not document a flag for binding to a non-loopback interface, and it does not discuss authentication on the web UI. If you want to reach it from another machine, the documentation is silent on how, which is probably intentional but leaves you guessing about the intended deployment model.

Provider setup is the part with the most moving pieces

Connecting a model is a six-step flow in the UI: open Settings → Models, add or edit a custom provider, paste the API base URL (provider ID and display name auto-fill), enter the API key, let models auto-discover from the endpoint, then pick one in the composer. The auto-discovery step is the one most likely to fail in a way that is hard to diagnose. A gateway that exposes an OpenAI-compatible chat completions route does not necessarily expose a models listing route, and the README does not describe a fallback beyond adding a model by hand. It also does not say what the discovery request looks like, which headers it sends, or whether the API key is used for the listing call. Provider cards are described as supporting live connection testing and enabled/disabled toggles, with per-model context badges, so there is at least a way to check a connection before starting work. The per-model context badge is a small but useful detail: context window size varies wildly between models on the same gateway, and compaction behaviour depends on knowing that number. If a custom gateway reports an incorrect context length, compaction will either fire too early or too late, and the README does not say whether the value can be overridden manually.

Pentest Mode is named but not specified

The README describes Pentest Mode in one line: "a professional offensive-security operating standard for authorized engagements." That is a positioning statement, not a specification. It does not say whether Pentest Mode changes the system prompt, restricts which tools the agent may call, requires an explicit scope declaration before shell commands run, or logs actions differently. It does not say how the harness distinguishes an authorized engagement from an unauthorized one, or whether it tries to at all. This is the largest gap in the documentation for the audience the project is aimed at. A harness that bundles shell execution, filesystem access and web research into an autonomous loop has a real failure mode: the agent runs a command against a host that was never in scope, and the transcript is the only record. The README's secure-by-default claim is specifically about API key storage, not about engagement boundaries. Anyone evaluating this for client work should treat the authorization boundary as something they enforce outside the tool, at the network and rules-of-engagement level, until the project documents otherwise. The toolset itself (shell, filesystem, web research, skills, goals, subagents, background jobs, workflow control) is broad enough that this is not a minor concern.

How it compares to wiring an existing agent framework to your own tools

The obvious alternative is to take a general agent framework and register pentest tooling as functions yourself. The difference is where the defaults sit. A general framework gives you a blank tool registry and a chat loop; you decide how sessions persist, how context is compacted, and where keys live. Pentest Harness ships those decisions already made: JSONL or SQLite sessions with replay, token metering with compaction and tool-result pruning, a credential store separate from settings, and a UI with provider cards and per-model context badges. You trade control for a working starting point. The reverse trade-off is that a general framework will not fight you if you want a session store backed by Postgres, or a credential backend that is not a YAML file, whereas Pentest Harness's replaceability is only as good as the Cordis interfaces, which the README does not show. The other axis is scope. A general framework makes no claim about offensive-security operating standards; Pentest Harness claims one without documenting it, which is arguably worse than making no claim, because it invites reliance on a mechanism that has not been described.

Maintenance, licence and what an upgrade costs you

The repository is not archived, and the last push was on 2026-09-11. There are no retrieved releases, so there is no tagged version history to read and no changelog to consult before upgrading. In practice that means tracking main, and the state that has to survive an upgrade is $DSH_HOME: settings.yaml, .credentials.yaml, and whatever session data lives alongside them. The README does not document a migration path for the settings schema, and it does not document rollback. A settings file that gains a required key in a later commit will fail on a version that predates it, and you will not find out from release notes. The licence is MIT, which permits commercial use, modification and redistribution with the copyright notice and permission notice retained. MIT gives no patent grant and no warranty, and it places no obligation on anyone to keep publishing. For an internal engagement tool that is usually acceptable. If you intend to redistribute a modified harness, the MIT notice has to travel with it, and you should read the licence text yourself rather than relying on this summary.

Editorial conclusion

Adopt Pentest Harness if you already run authorized engagements, want the model endpoint under your own key, and are willing to read the Cordis plugin layer because the README does not document how Pentest Mode restricts tool use. Do not adopt it if you need a written scope-enforcement mechanism before tools execute, or if you want a maintained release cadence with tagged versions. Verify first that Settings → Models auto-discovery returns the model list you expect from your gateway, and that $DSH_HOME/.credentials.yaml lands with owner-only permissions on your filesystem.

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. S1N6H/pentest-harness on GitHub