Promptfoo: Declarative LLM evals and red teaming from the CLI

What promptfoo actually solves

Promptfoo addresses the trial-and-error approach to building LLM apps. Instead of manually testing prompts against a single model, you define tests in a declarative config and run them across providers. The README lists four concrete use cases: automated evaluations, red teaming and vulnerability scanning, side-by-side model comparison, and CI/CD integration. The tool also includes code scanning for pull request review, which targets LLM-related security and compliance issues. The intended audience is developers who ship LLM features and need repeatable, metric-based decisions rather than gut feel. The project claims to power apps serving 10M+ users, but that is a self-reported figure; verify against your own workload.

How the eval and red team flow works

The core mechanism is a config file that describes prompts, providers, and assertions. You run promptfoo eval, which sends the prompts to the configured models, collects responses, and evaluates them against your assertions. The results are stored locally and can be viewed with promptfoo view, which opens a web UI. Red teaming is a separate mode that generates adversarial inputs and scans for vulnerabilities. The README mentions that evals run 100% locally, meaning prompts and responses do not leave your machine unless you call a remote API. That privacy boundary is a key design choice. The tool supports a wide provider list, including OpenAI, Anthropic, Azure, Bedrock, and Ollama, so the same config can compare models without rewriting code.

Getting started: commands and config keys

Installation is straightforward. The README shows npm install -g promptfoo, but also brew install promptfoo and pip install promptfoo. You can run any command via npx promptfoo@latest without a global install. Node.js >=22.22.0 is required for npm and npx, with Node 24 LTS recommended. After install, run promptfoo init --example getting-started to create a sample project. Then set an API key, for example export OPENAI_API_KEY=sk-abc123. Inside the getting-started directory, run promptfoo eval to execute the eval and promptfoo view to open the results viewer. The exact config schema is not in the README, but the phrase 'simple declarative configs' suggests YAML or JSON files that define prompts, providers, and assertions. For red teaming, the docs are separate; the README points to /docs/red-team/ for details.

Local execution and privacy trade-offs

A major selling point is privacy: 'LLM evals run 100% locally - your prompts never leave your machine.' That is true for the orchestration layer, but the caveat is that if you evaluate against hosted models like GPT-4 or Claude, your prompts are sent to those providers. The local claim applies to the tool's own processing, not the network calls to LLM APIs. For teams with strict data handling policies, this distinction matters. You can mitigate by using local models via Ollama, which the provider list includes. The trade-off is that local models may have lower quality, so your eval results may not reflect production behavior. The README does not discuss this nuance, but any engineer should spot it.

Limitations and when it is the wrong tool

Promptfoo is not a hosted platform. There is no managed dashboard, no team collaboration features, and no built-in scheduling. If you need a centralized evaluation platform with user management and persistent history, this CLI will not replace it. The tool also requires Node.js >=22.22.0, which may be a problem in older CI environments or for teams standardized on a different runtime. The README mentions a Node.js runtime support guide, but the minimum is explicit. Another limitation is that the red teaming feature is a separate mode with its own docs; the README does not list the specific attack types or plugins. You must consult the red team documentation to know if it covers your threat model. Finally, the project is now part of OpenAI, which introduces a governance question. The README states it remains open source and MIT licensed, but the long-term roadmap is not guaranteed.

Alternatives and how they differ

The obvious alternative is building your own eval harness with a scripting language and direct API calls. That gives you full control over assertions and data collection, but you lose the declarative config, the built-in red teaming, and the result viewer. Another alternative is a hosted evaluation platform like LangSmith or Weights & Biases Prompts, which offer dashboards and collaboration but require sending data to a third party. The key difference is that promptfoo is local-first and CLI-driven, so it fits into existing CI pipelines without a new SaaS dependency. For red teaming specifically, you could use a dedicated security tool like Garak or PyRIT, but those are focused on adversarial testing only, not general eval. Promptfoo combines both eval and red team in one config, which reduces tool sprawl but may not be as deep in either area as a specialized tool.

Maintenance and license considerations

The project is actively maintained, with release 0.122.2 pushed on 2026-08-28 and a separate code-scan-action release on the same day. The license is MIT, which permits commercial use, modification, and redistribution with attribution. The README notes that promptfoo is now part of OpenAI, but the license and open source status are unchanged. For upgrade cost, the frequent releases (three versions in a week) suggest a fast-moving codebase; you should pin versions in CI to avoid breaking changes. The code scanning action is a separate release artifact, so you may need to update that independently. There is no mention of migration guides or deprecation policies in the README, so check the changelog before upgrading. The npm package is the primary distribution, but the pip and brew channels exist, which means you may need to maintain multiple install methods if your team spans environments.

Editorial conclusion

Adopt promptfoo if you need a local, scriptable eval and red team tool that works with any provider and can run in CI. Skip it if you require a hosted dashboard or managed infrastructure, or if your team cannot maintain Node.js >=22.22.0 environments. Before committing, verify your provider's API key handling and rate limits, and check the red team plugin coverage against your specific threat model. The project is now part of OpenAI, which may affect long-term direction, but the MIT license and active release cadence (0.122.2 on 2026-08-28) suggest continued availability.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes