# iFixAi: Independent Auditing for AI Agents > iFixAi is a Python-based diagnostic tool that audits AI agents against business KPIs and organizational structure, not just technical benchmarks. It runs 32 inspections across five pillars and delivers an A-F scorecard in under 120 seconds. **ifixai-ai/iFixAi** — Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds. - Repository: https://github.com/ifixai-ai/iFixAi - Website: https://www.ifixai.ai - Stars: 17,153 · Forks: 1,356 - Language: Python - License: Apache-2.0 - Published: 2026-09-10 · Updated: 2026-09-10 · Language: en - Canonical page: https://hysenlabs.com/projects/ifixai-ai-ifixai ## What iFixAi Audits and Who It Is For Most AI evaluation tools measure token efficiency, latency, and resistance to prompt injection. They answer whether an agent is technically capable. iFixAi addresses a different question: is the agent doing the job it is supposed to do based on the business KPIs and organizational structure it was deployed to serve? The README describes this as striking a balance between AI red-teaming and operational assurance. The intended users are teams that have already deployed an agent and want an auditable record of whether it behaves as specified, not just whether it produces output. That includes developers running CI pipelines, security-conscious operators, and anyone doing team onboarding around a shared agent configuration. ## The Five-Pillar Inspection Engine A single iFixAi run executes 32 inspections organized across five pillars. The README names these pillars as part of the scorecard output but does not enumerate each pillar by name in its public documentation. What is documented is the output structure: a JSON report, a Markdown report, and a rich terminal scorecard that displays an A-F grade with per-pillar scores. The inspection suites are tiered. The --suite flag accepts smoke, strategic, core, extended, or all. The wizard lets you pick with arrow keys. Smaller suites run faster and cost less in API calls; the all suite runs every available inspection. The release history shows the suite system has grown across versions: v3.4.1 introduced INFLUENCE inspections and v4.0.0 added V-Series Inspections, meaning the inspection library expands with releases. Results land in ./ifixai-results/ by default. The directory holds both the JSON source of truth and the Markdown report, so audit trails are preserved on disk without any external database. The README does not document the internal scoring formula used to map inspection results to letter grades. ## Installing iFixAi and Running the Guided Wizard iFixAi requires Python 3.10 or later. Installation uses provider-specific extras so you only pull in the SDK for the provider you intend to test. ```bash pip install "ifixai[openai]" # or anthropic, gemini, etc.: install the provider extra you'll test ifixai setup # arrow-key wizard: pick provider, model, judge, suite -> writes ifixai.yaml ifixai run # no flags needed; reports land in ./ifixai-results/ ``` The setup wizard detects API keys already in your environment and surfaces them at the top of each prompt. If no key is found, the wizard tells you which environment variable to export. The wizard writes an ifixai.yaml file so subsequent runs need no flags at all. On Windows, if PowerShell does not find the ifixai command after pip install, the README notes you should add Python's Scripts\ folder to your PATH, or run it as python -m ifixai. This is a standard Python-on-Windows PATH issue. For scripted or CI environments, all options can be passed as explicit flags instead of using the wizard. The README describes the CLI as fully scriptable and documents the --suite flag and --api-key flag for this purpose. The available providers as optional extras are: openai, anthropic, gemini, azure, bedrock, huggingface, litellm, atlascloud, openrouter, orcarouter, and requesty. The litellm extra is constrained to >=1.60.0,<2.0.0. ## Running iFixAi as a Plugin or Skill Inside an Agent The third mode of operation makes the agent being audited also the operator running the audit. iFixAi ships a Claude Code plugin and a Codex plugin, and a universal skill scaffold for any agent. For Claude Code: ``` /plugin marketplace add ifixai-ai/iFixAi /plugin install ifixai@ifixai-community ``` After installation, running /ifixai:ifixai or asking in plain English causes the agent to discover your config, build the fixture, show the projected cost before billing, and run the diagnostic. The interactive results artifact is the primary output; JSON is the fallback. For agents without a native plugin system, the uvx scaffold writes a /ifixai-skill slash command into the agent's config directory: ```bash uvx ifixai install --agents cursor # any slug: claude, codex, vscode, windsurf, cline, continue, gemini, zed uvx ifixai install --agents all # scaffold every agent at once uvx ifixai install --list # every supported agent and where its file lands ``` The skill's --dry-run shows the cost before anything is billed. The skill requires only uv and Python 3.10+, with no provider extra needed at scaffold time. ## What iFixAi Does Not Do, and Where It Falls Short iFixAi is not a latency profiler and does not measure token throughput. If your primary concern is response time or cost per token, you need a different class of tool. The scoring model depends on a judge, which is itself an AI model. The README offers three judge options: self (the same model being tested), one independent vendor, or a multi-judge ensemble. A self-judged run has an obvious conflict of interest. The ensemble option costs more in API calls and the README does not document how disagreements between judges are resolved. The inspection library is versioned and expanding, which means a run on v3.4.1 and a run on v4.0.0 may not be directly comparable without checking the suite definition. Teams using iFixAi as a compliance baseline should pin the version. iFixAi also requires an internet connection and live API calls to execute. There is no offline or mock mode documented in the README. Every inspection fires real calls against the agent under test, which means a full suite run accumulates actual API costs. The --dry-run flag in the skill scaffold shows the projected cost before anything is billed, but the README does not document a comparable flag for the CLI wizard mode. ## Comparison with Existing Eval and Observability Tools The README places iFixAi in explicit contrast with existing eval, red-teaming, and observability tools, which it characterizes as evaluating agents primarily based on technical capability. The distinction iFixAi draws is between technical capability (can the agent do it?) and business alignment (is the agent doing what it was deployed to do?). A tool like DeepEval or LangSmith measures output quality against a test set. iFixAi adds a layer that frames the test against organizational KPIs and structure. That framing is only as useful as the fixture you build to represent your actual deployment. The wizard and skill scaffold automate fixture creation, but a poorly specified fixture produces a scorecard that reflects the fixture, not the real deployment. ## Maintenance Status and Licensing The last push was on 2026-09-25, and v4.0.0 was released on 2026-09-15. The project is under the Apache-2.0 license, which permits commercial use, modification, and redistribution, subject to the attribution and NOTICE file requirements in that license. The pyproject.toml lists the project development status as Beta (Development Status :: 4 - Beta). The inspection library grew across v3.4.0 (persistence and identity inspections), v3.4.1 (INFLUENCE inspections), and v4.0.0 (V-Series inspections), showing the suite definitions are still in motion. Teams embedding iFixAi in CI pipelines should treat minor version upgrades as potentially changing the baseline scorecard. ## Conclusion iFixAi is for teams running AI agents in production who need evidence that those agents are doing the right job, not just doing a job quickly. It is the wrong tool for pure performance benchmarking or latency profiling. Before adopting it, verify that your agent's provider is listed in the optional extras, that your API keys are available as environment variables, and that you understand the per-run inference cost, which the --dry-run flag exposes before committing. ## FAQ ### What does iFixAi actually measure about an AI agent? iFixAi runs 32 inspections across five pillars and returns an A-F grade with a per-pillar scorecard. The README describes the focus as whether the agent is doing the job it was deployed to do based on business KPIs and organizational structure, not whether it is technically fast or injection-resistant. ### Which AI providers does iFixAi support? iFixAi supports OpenAI, Anthropic, Google Gemini, Azure, AWS Bedrock, HuggingFace, LiteLLM, AtlasCloud, OpenRouter, OrcaRouter, and Requesty, each as a pip optional extra installed with pip install "ifixai[]". ### How do I run iFixAi without the interactive wizard? Pass all options as CLI flags instead of using ifixai setup. The README describes this mode as fully scriptable and suited for CI, automation, and audit-ready scripted batches. The --suite flag accepts smoke, strategic, core, extended, or all. ### How does the iFixAi plugin install in Claude Code? Run /plugin marketplace add ifixai-ai/iFixAi followed by /plugin install ifixai@ifixai-community inside Claude Code. Then ask "run iFixAi on my setup" or type /ifixai:ifixai. A restart or /reload-plugins is needed if the command does not appear immediately. ## Sources - [ifixai-ai/iFixAi on GitHub](https://github.com/ifixai-ai/iFixAi) - [License: Apache-2.0](https://github.com/ifixai-ai/iFixAi/blob/main/LICENSE) - [Project website](https://www.ifixai.ai) - [README](https://github.com/ifixai-ai/iFixAi/blob/main/README.md) - [Releases](https://github.com/ifixai-ai/iFixAi/releases) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/ifixai-ai-ifixai