iFixAi: Independent Auditing for AI Agents

What iFixAi Audits and Who It Is For

Most AI evaluation tools measure token efficiency, latency, and resistance to prompt injection. They answer whether an agent is technically capable. iFixAi addresses a different question: is the agent doing the job it is supposed to do based on the business KPIs and organizational structure it was deployed to serve?

The README describes this as striking a balance between AI red-teaming and operational assurance. The intended users are teams that have already deployed an agent and want an auditable record of whether it behaves as specified, not just whether it produces output. That includes developers running CI pipelines, security-conscious operators, and anyone doing team onboarding around a shared agent configuration.

The Five-Pillar Inspection Engine

A single iFixAi run executes 32 inspections organized across five pillars. The README names these pillars as part of the scorecard output but does not enumerate each pillar by name in its public documentation. What is documented is the output structure: a JSON report, a Markdown report, and a rich terminal scorecard that displays an A-F grade with per-pillar scores.

The inspection suites are tiered. The --suite flag accepts smoke, strategic, core, extended, or all. The wizard lets you pick with arrow keys. Smaller suites run faster and cost less in API calls; the all suite runs every available inspection. The release history shows the suite system has grown across versions: v3.4.1 introduced INFLUENCE inspections and v4.0.0 added V-Series Inspections, meaning the inspection library expands with releases.

Results land in ./ifixai-results/ by default. The directory holds both the JSON source of truth and the Markdown report, so audit trails are preserved on disk without any external database. The README does not document the internal scoring formula used to map inspection results to letter grades.

Installing iFixAi and Running the Guided Wizard

iFixAi requires Python 3.10 or later. Installation uses provider-specific extras so you only pull in the SDK for the provider you intend to test.

pip install "ifixai[openai]"   # or anthropic, gemini, etc.: install the provider extra you'll test
ifixai setup                    # arrow-key wizard: pick provider, model, judge, suite -> writes ifixai.yaml
ifixai run                      # no flags needed; reports land in ./ifixai-results/

The setup wizard detects API keys already in your environment and surfaces them at the top of each prompt. If no key is found, the wizard tells you which environment variable to export. The wizard writes an ifixai.yaml file so subsequent runs need no flags at all.

On Windows, if PowerShell does not find the ifixai command after pip install, the README notes you should add Python's Scripts\ folder to your PATH, or run it as python -m ifixai. This is a standard Python-on-Windows PATH issue.

For scripted or CI environments, all options can be passed as explicit flags instead of using the wizard. The README describes the CLI as fully scriptable and documents the --suite flag and --api-key flag for this purpose.

The available providers as optional extras are: openai, anthropic, gemini, azure, bedrock, huggingface, litellm, atlascloud, openrouter, orcarouter, and requesty. The litellm extra is constrained to >=1.60.0,<2.0.0.

Running iFixAi as a Plugin or Skill Inside an Agent

The third mode of operation makes the agent being audited also the operator running the audit. iFixAi ships a Claude Code plugin and a Codex plugin, and a universal skill scaffold for any agent.

For Claude Code:

/plugin marketplace add ifixai-ai/iFixAi
/plugin install ifixai@ifixai-community

After installation, running /ifixai:ifixai or asking in plain English causes the agent to discover your config, build the fixture, show the projected cost before billing, and run the diagnostic. The interactive results artifact is the primary output; JSON is the fallback.

For agents without a native plugin system, the uvx scaffold writes a /ifixai-skill slash command into the agent's config directory:

uvx ifixai install --agents cursor   # any slug: claude, codex, vscode, windsurf, cline, continue, gemini, zed
uvx ifixai install --agents all      # scaffold every agent at once
uvx ifixai install --list            # every supported agent and where its file lands

The skill's --dry-run shows the cost before anything is billed. The skill requires only uv and Python 3.10+, with no provider extra needed at scaffold time.

What iFixAi Does Not Do, and Where It Falls Short

iFixAi is not a latency profiler and does not measure token throughput. If your primary concern is response time or cost per token, you need a different class of tool.

The scoring model depends on a judge, which is itself an AI model. The README offers three judge options: self (the same model being tested), one independent vendor, or a multi-judge ensemble. A self-judged run has an obvious conflict of interest. The ensemble option costs more in API calls and the README does not document how disagreements between judges are resolved.

The inspection library is versioned and expanding, which means a run on v3.4.1 and a run on v4.0.0 may not be directly comparable without checking the suite definition. Teams using iFixAi as a compliance baseline should pin the version.

iFixAi also requires an internet connection and live API calls to execute. There is no offline or mock mode documented in the README. Every inspection fires real calls against the agent under test, which means a full suite run accumulates actual API costs. The --dry-run flag in the skill scaffold shows the projected cost before anything is billed, but the README does not document a comparable flag for the CLI wizard mode.

Comparison with Existing Eval and Observability Tools

The README places iFixAi in explicit contrast with existing eval, red-teaming, and observability tools, which it characterizes as evaluating agents primarily based on technical capability. The distinction iFixAi draws is between technical capability (can the agent do it?) and business alignment (is the agent doing what it was deployed to do?).

A tool like DeepEval or LangSmith measures output quality against a test set. iFixAi adds a layer that frames the test against organizational KPIs and structure. That framing is only as useful as the fixture you build to represent your actual deployment. The wizard and skill scaffold automate fixture creation, but a poorly specified fixture produces a scorecard that reflects the fixture, not the real deployment.

Maintenance Status and Licensing

The last push was on 2026-09-25, and v4.0.0 was released on 2026-09-15. The project is under the Apache-2.0 license, which permits commercial use, modification, and redistribution, subject to the attribution and NOTICE file requirements in that license.

The pyproject.toml lists the project development status as Beta (Development Status :: 4 - Beta). The inspection library grew across v3.4.0 (persistence and identity inspections), v3.4.1 (INFLUENCE inspections), and v4.0.0 (V-Series inspections), showing the suite definitions are still in motion. Teams embedding iFixAi in CI pipelines should treat minor version upgrades as potentially changing the baseline scorecard.

Editorial conclusion

iFixAi is for teams running AI agents in production who need evidence that those agents are doing the right job, not just doing a job quickly. It is the wrong tool for pure performance benchmarking or latency profiling. Before adopting it, verify that your agent's provider is listed in the optional extras, that your API keys are available as environment variables, and that you understand the per-run inference cost, which the --dry-run flag exposes before committing.

Frequently asked questions

Official sources

1. ifixai-ai/iFixAi on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases