iFixAi audits AI agents against business KPIs, not just token counts
Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
At a glance
- What is it?
- iFixAi is an Apache-2.0 Python CLI that runs a guided diagnostic over a provider or your own agent endpoint and returns an A to F grade. It targets the operational question of whether an agent does its job, and it ships as a wizard, a flag-driven CLI, or an agent plugin.
- Who is it for?
- Adopt iFixAi if you already have an agent endpoint and a rough idea of the business outcome it is meant to produce, and you want a repeatable, scriptable audit artifact rather than a one-off manual red-team session. Do not adopt it if you have no defined success criteria for the agent yet, because the five-pillar scorecard will have nothing to grade against, or if you need a hosted dashboard with historical trend lines, which the material does not describe.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The question iFixAi is built to answer
The README draws a line between iFixAi and the eval, red-teaming and observability tools it says came before. Those, in its framing, grade an agent on technical capability: token efficiency, latency, prompt injection resistance. iFixAi's stated target is different. It asks whether the agent is doing the job it is supposed to do, measured against business KPIs and organizational structure. That reframing is the whole pitch, and it is also the source of the project's main constraint, which I come back to later. The intended user is anyone who has to sign off on an agent before it touches real work: an engineer wiring a CI gate, a risk or compliance owner who needs an artifact, or an agent operator who wants a second opinion from inside the tool they already use. The repository topics point at the same audience, listing eu-ai-act, iso-42001, nist-ai-rmf and owasp-llm alongside agent-evaluation and hallucination-detection. The claimed turnaround is under 120 seconds, which is a marketing figure from the README rather than something I measured.
Five pillars, a judge, and an A to F grade
The mechanism visible in the material is a fixed set of inspections run against a target system. The banner image caption describes one ifixai run end to end: guided setup picks the system, the judge and the suite, the run verifies the connection and saves config, then inspections execute across five pillars, and the result lands as an A to F grade with a scored core-pillar scorecard. The badge at the top of the README says 50 inspections, while the same caption describes 32 inspections executing in that particular run. Those two numbers are not reconciled in the material, and the most likely reading is that the library holds 50 inspections and a given suite runs a subset, but the README does not say so explicitly. Treat the count as suite-dependent until you check. The judge is a first-class configuration choice: you can grade with yourself, with one independent vendor, or with a multi-judge ensemble. That is a meaningful design decision, because a self-judged run and an ensemble-judged run are not the same evidence, and the scorecard alone will not tell a reader which one produced it unless the config travels with the report. Output is JSON plus Markdown reports plus a rich terminal scorecard, with the plugin path producing an interactive results artifact and a JSON source of truth. The JSON is the part that matters for automation.
Getting a first run out of the wizard
The README's recommended path is three commands. Install the provider extra you intend to test, run the wizard once, then run the audit with no flags. The extra matters: pip install "ifixai[openai]" is the example, and the README names anthropic and gemini as alternatives. Running ifixai setup opens an arrow-key wizard where you pick provider, model, judge and suite. It writes ifixai.yaml. After that, ifixai run needs no flags and drops reports into ./ifixai-results/. On key handling, the README is specific and the choice is sound: the wizard detects API keys already present in your environment and surfaces them at the top of each prompt, and it stores the env-var name in ifixai.yaml rather than the secret itself. If no key is found, the wizard tells you which variable to export, and if it is still missing at run time you are prompted before the first API call. The Windows note is worth repeating because it is a common false alarm: if PowerShell cannot find ifixai after pip install, add Python's Scripts folder to PATH or invoke it as python -m ifixai. That is the ordinary Python-on-Windows PATH gap, not a defect in the tool.
Scripted runs, suites, and the plugin path
The second CLI mode skips the wizard and passes every option as a flag, which the README positions for CI, automation and audit-ready scripted batches. Suite selection is exposed as --suite with the values smoke, strategic, core, extended and all. Keys come from a --api-key flag or an environment variable. The third mode inverts the operator: the agent itself discovers your setup, builds the fixture, runs the diagnostic and explains the scorecard. For Claude Code the install is a plugin marketplace add followed by a plugin install, and the invocation is either plain English or the /ifixai:ifixai command. Codex uses codex plugin marketplace add and codex plugin add. For agents without a plugin, uvx ifixai install scaffolds a /ifixai-skill file. The README lists Claude Code, Cursor, Codex, VS Code, Windsurf, Cline, Continue, Gemini and Zed as working environments. Two migration details are documented and both are the kind of thing that wastes an afternoon if missed. The marketplace was renamed, so on Claude Code you run /plugin marketplace remove ifixai-ai before adding the new one, and on Codex, codex plugin marketplace upgrade fails against the renamed marketplace, so you remove first and re-add. The README also notes that the plugin path names the cost before anything is billed, which matters because a multi-judge ensemble multiplies API calls.
Where the KPI claim gets thin
The central claim is that iFixAi tells you whether an agent does its job against business KPIs and organizational structure. The README does not show how those KPIs enter the system. There is no example of a KPI file, no described schema for organizational structure, and no worked case where a business objective is translated into an inspection. What the material does describe concretely is a fixed inspection library organized into five pillars, with a scorecard that grades a core pillar subset. That is a real and useful thing, but it is closer to a structured checklist than to a bespoke audit of your particular business logic. The gap matters at adoption time. If your agent's success condition is something the five pillars do not encode, the grade will be a measure of general agent hygiene and not of your KPI, and the README gives no path for adding your own criteria. A second limitation is environmental. The tool audits a target endpoint, so it needs a reachable provider or your agent's real endpoint, and the README's own troubleshooting is about connection verification during setup. If your agent only runs inside a proprietary harness with no callable endpoint, this is the wrong shape of tool. Third, the under-120-seconds figure is a README claim with no stated conditions: model, suite, judge count and network all plausibly move it, and a multi-judge ensemble on the all suite is not the same run as a smoke suite with a self-judge.
How this differs from promptfoo and similar harnesses
The closest comparison in this space is promptfoo, a widely used open source LLM eval harness, and the difference is one of unit of analysis. A harness like promptfoo is built around test cases you author: you write prompts, define assertions, and the tool tells you which assertions pass. The burden of deciding what correct means sits with you, and the output is per-test-case. iFixAi inverts that. You point it at a system, pick a suite, and it supplies the inspection set from its own library, then compresses the result into a single A to F grade with a pillar breakdown. The trade is real in both directions. iFixAi gives you a comparable score across agents and runs without authoring anything, which is exactly what a governance or sign-off workflow wants. It gives you less control over what is being checked, which is exactly what a team with unusual requirements wants. The judge configuration has no clean equivalent in the assertion-based model either: choosing between self-grading, one independent vendor, or an ensemble is a specific answer to the problem that an LLM grading its own output is a weak signal. If your need is regression testing of prompts you already understand, an assertion harness fits better. If your need is an external, repeatable verdict on an agent you did not build, iFixAi's shape is the more natural one.
Licence, upgrade cost, and the release cadence
The project is Apache-2.0, which permits commercial use, modification and redistribution provided the licence and notices are preserved, and it includes an explicit patent grant. That is a permissive licence and it is the reason a compliance team is unlikely to block adoption on licensing grounds alone. This is a description of the licence text, not legal advice; if you are embedding the tool in a product or a regulated process, have counsel read the NOTICE and attribution requirements rather than relying on a summary. On maintenance, the release history shows a fast cadence: v3.3.0, v3.4.0 and v3.4.1 landed within roughly three weeks of each other in August 2026, with titles that read like behavioural changes rather than patches. v3.3.0 is labelled truer scores and runs that finish, and v3.4.1 introduces influence inspections. Two of those three titles imply the scoring or the inspection set changed. For anyone pinning iFixAi as a CI gate, that is the practical cost: a grade produced by v3.3.0 and a grade produced by v3.4.1 may not be comparable, and the material does not describe a scoring-version field in the report. Pin the version in CI, record it alongside the JSON output, and re-baseline when you move. The Python floor is 3.10 or newer, per the badge. The provider extras mean your dependency surface is the provider SDK you install, so an upgrade of ifixai can pull a new SDK version with it.
Editorial conclusion
Adopt iFixAi if you already have an agent endpoint and a rough idea of the business outcome it is meant to produce, and you want a repeatable, scriptable audit artifact rather than a one-off manual red-team session. Do not adopt it if you have no defined success criteria for the agent yet, because the five-pillar scorecard will have nothing to grade against, or if you need a hosted dashboard with historical trend lines, which the material does not describe. Before trusting a grade, verify three things: that the provider extra you install matches the endpoint you actually test, that the judge configuration in ifixai.yaml is the one you intend (self, one vendor, or an ensemble), and that the suite you select covers the pillars you care about, since smoke and all differ in scope.
Community notes