AuditPilot (intelligent-audit-system): an evidence-grounded agent workbench for audit delivery

The problem AuditPilot picks: audit evidence, not chat

Most agent repositories demonstrate conversation. AuditPilot targets a narrower and less forgiving workflow: enterprise audit delivery, where a finding has to be traceable to a document, a control has to map to a test procedure, and a reviewer has to be able to re-run the reasoning. The README states the goal directly: the project is not meant to replace auditors, but to organize the repeated acts of evidence gathering, mapping, checking, gap-filling and delivery into a governed agent workflow. The intended user is an audit or internal-controls team, not a general developer looking for an agent SDK. The README's own pain-point table is the clearest statement of scope: scattered materials that force repeated evidence requests, control testing that depends on individual experience, AI conclusions that cannot enter workpapers, and findings that lose context between remediation and re-review. Each row pairs a pain with a verifiable output such as an evidence request form, page-level citations, a missing-evidence list, a quality gate, or an immutable deliverable. That pairing is the design thesis. The project also declines to quote savings figures, saying value should be measured in pilot projects through evidence cycle time, first-pass workpaper review rate, evidence sufficiency and on-time remediation closure.

The runtime is a bounded loop, not an open-ended agent

The architecture section lays out a linear pipeline: an audit request enters a hybrid intent router, passes through working, episodic and profile memory, then reaches a set of named roles (Planner, Evidence, Control, Risk, Compliance, Remediation, Verification, Delivery) inside a bounded, dependency-aware agent loop. Retrieval, an evidence graph and MCP-style tools sit underneath; a safety gate, reflection step and human review sit above; the output is a delivery package plus a governed experience candidate. Two boundaries in that description matter more than the role names. First, the loop is bounded and steps carry dependencies, which means the runtime is not free to spawn arbitrary sub-agents until a token budget runs out. Second, the README separates what the LLM is allowed to do (understand, summarize, explain) from what stays deterministic and auditable (evidence gaps, quality gates, permissions, risk signals, delivery state). That split is the reason the project can run without a model key at all: when no LLM is configured, it falls back to a deterministic mode and the audit workflow, RAG, evaluation and interface still work. For a reviewer, deterministic fallback is a genuine feature, because it makes the evaluation harness reproducible. It is also a constraint, since the quality of the non-fallback path depends entirely on the model you plug in.

Retrieval is hybrid and deliberately not vector-first

The Agentic RAG module is described as TF-IDF plus keyword multi-recall with fusion reranking, metadata filtering, page and section provenance, conflict detection between sources, and missing-evidence prompts. Semantic vectors are optional. The README is explicit that the default installation does not include Torch or Transformers, and that you install requirements-embeddings.txt and set RAG_ENABLE_EMBEDDINGS=1 only if you actually need local semantic vectors. That is a real trade-off, not a marketing line. A lexical-first retriever is cheap to run, easy to inspect and reproducible, which fits an audit context where you may need to explain why a passage was retrieved. It will also miss paraphrases and cross-lingual matches that a dense retriever would catch, and the project does not claim otherwise. Three tuning keys are exposed: RAG_CHUNK_SIZE, RAG_CHUNK_OVERLAP and RAG_TOP_K. Because provenance is tracked at page and section level, chunking decisions directly affect whether a citation points at the right passage, so those values are worth setting deliberately rather than leaving at defaults. The evidence graph is the second half of this story: it connects tasks, steps, tool runs and artifacts, then checks source coverage, broken dependencies and critical orphan nodes.

Getting it running and the keys that matter

The README gives explicit commands. On macOS or Linux: python3 -m venv .venv, source .venv/bin/activate, pip install -r requirements.txt, cp config.env.example config.env, python start.py. On Windows PowerShell the same sequence uses .\.venv\Scripts\Activate.ps1 and Copy-Item config.env.example config.env. Python 3.10 or higher is required, and the README notes the app can run with no model key configured. After startup you open the address printed in the terminal and stop with Ctrl+C. Configuration is optional by default and lives in config.env, which is gitignored. The optional keys are grouped by purpose: DEEPSEEK_API_KEY or a compatible provider variable for LLM enhancement, MYSQL_HOST/MYSQL_USER/MYSQL_PASSWORD for a MySQL standards library, NEO4J_URI/NEO4J_USER/NEO4J_PASSWORD for the graph, and the three RAG parameters. Production authentication is a different category: SECURITY_MODE=enforced, AUDITPILOT_API_TOKENS_JSON and AUDIT_LOG_SIGNING_KEY are listed as required in production. Upload governance uses KNOWLEDGE_UPLOAD_MAX_BYTES, EVIDENCE_UPLOAD_MAX_BYTES and REJECT_PROMPT_INJECTION. One boundary deserves attention: SECURITY_MODE=local is restricted to a loopback bind, so any shared or network deployment must move to enforced mode with secret-managed bearer tokens and a signing key. Validation commands are also published, including a coverage run scoped to six modules with --cov-fail-under=75, plus ruff, bandit and scripts/audit_repro.py.

The evaluation harness is the part worth copying

The Evaluation Harness scores task results, execution traces, tool calls, evidence grounding, safety permissions, context and robustness in separate layers, then emits calibrated scores with a confidence lower bound. The README states that a failed critical assertion blocks release outright. That is a stronger position than most agent projects take, and it is the mechanism that makes the rest of the system governable: a finding cannot ship if its evidence layer fails, regardless of how fluent the generated text is. The same gate appears in Governed Improvement, where failures become experience candidates only, and may be reused only after regression evaluation and human approval. Episode and observability support this with privacy-oriented trace packages containing standard semantic fields, tool evidence, safety gates, failure attribution, intervention records and an integrity digest. The consequence is that AuditPilot is expensive to operate in a specific way: every change to prompts, retrieval parameters or tools implies a re-run of the layered evaluation before the release gate opens. Teams that treat evaluation as a one-time setup step will find the gate blocks them rather than helping them. The README does not publish baseline scores, so you cannot compare your numbers against a reference run; you establish your own baseline on your own corpus.

Storage, maintenance and the missing licence

The Security and Storage row states that audit, evaluation, task and memory data are persisted in SQLite with WAL transactions, behind Bearer identity, RBAC plus project-level ABAC, request rate limiting, HMAC-signed checkpoints and upload governance. SQLite in WAL mode is a reasonable choice for a single-node pilot and a poor one for concurrent multi-team deployment, and the README does not describe a migration path to a server database even though MySQL appears as an optional standards library. Permissions are checked at both the HTTP and Skill boundaries, with repositories additionally enforcing tenant and project membership, and uploads are streamed with a hard size limit, content and extension checks, hashes, credential scanning and prompt-injection quarantine. Maintenance cost is dominated by two things: the release gate described above, and the boundary between the public repository and everything excluded from Git. The README lists what stays out: API keys, database passwords, local run records, session memory, evaluation results, uploads, generated RAG stores, internal design documents, local models and logs. That means a fresh clone starts from public seed knowledge and you rebuild the local RAG store on first run. On licensing, the repository material does not state a licence identifier, so no permission to use, modify or redistribute can be inferred from what is available here. Treat licence verification as a prerequisite step, not a formality, and do not read this article as legal advice.

Where it fits against a general agent framework

The obvious alternative is a general orchestration framework such as LangGraph, which gives you graph-based control flow, checkpointing and human-in-the-loop interrupts as primitives, and leaves evaluation, evidence grounding, tool permissions and release gating to you. AuditPilot makes the opposite bet: it ships those as opinionated, non-optional layers. Its nine-layer component evaluation, evidence graph, RBAC plus ABAC checks at the tool boundary, HMAC-signed checkpoints and governed improvement loop are part of the product, not a pattern you assemble. The cost is flexibility. If your workflow is not audit-shaped, you inherit a control matrix, sampling plans, remediation tasks and a release gate that have no meaning for you, and you will spend time removing rather than adding. The other real difference is delivery: AuditPilot is a FastAPI application with a web workspace, screenshots for an audit workbench, agent runtime, agent chat and layered evaluation views, and a start.py entry point. A framework gives you a library. This gives you an application you have to operate, secure and upgrade. The README also notes that six built-in templates cover four standards, 34 control topics, 36 evidence input types and 25 deliverable types, counted from services/audit_templates.py, and explicitly says those numbers describe out-of-the-box scope rather than full industry coverage. That is an accurate description of a starting point, not a finished control library.

Editorial conclusion

Adopt AuditPilot if you are building internal audit or control-testing workflows where every conclusion must point back to a source document and pass a release gate, and you accept SQLite-backed storage plus a Python 3.10+ service you run yourself. Do not adopt it if you need a general-purpose agent framework, a hosted product, or a licensing answer today: the repository material does not state a licence. Before committing, verify the licence file, run the pytest command with its --cov-fail-under=75 gate, and confirm that SECURITY_MODE=enforced plus AUDITPILOT_API_TOKENS_JSON and AUDIT_LOG_SIGNING_KEY are wired into your deployment, because the README treats those as production requirements.

Official sources

1. Issues
2. README
3. Ricky-7-Yan/intelligent-audit-system on GitHub