ChainForge: A Visual Flow Editor for Comparing Prompts Across LLM Providers
An open-source visual programming environment for battle-testing prompts to LLMs.
At a glance
- What is it?
- ChainForge is a browser-based dataflow tool that fans a prompt template out across models, providers and parameter values, then scores the responses. It is useful for people who need to see a prompt's behaviour across combinations rather than chat with one model at a time.
- Who is it for?
- Adopt ChainForge if your problem is comparison: you have a prompt template with variables and you need to see how it behaves across several models or parameter values before committing. Skip it if you need a deployed evaluation harness with versioned datasets and CI integration; the repository describes an interactive environment, not a pipeline.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly TypeScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem ChainForge solves: one prompt, many combinations
Chat interfaces answer one prompt at a time. If you want to know whether a template works as well with GPT as it does with Claude, or whether the temperature setting matters, you end up retyping and re-reading by hand. ChainForge replaces that loop with a graph. You define prompt templates with variables, connect them to model nodes, and the environment sends the queries and collects the responses into a table you can inspect and export.
The README frames the audience directly: people doing "rapid-fire, quick-and-dirty comparison of prompts, models, and response quality that goes beyond ad-hoc chatting with individual LLMs." That word choice matters. This is a tool for exploratory work, not for a formal evaluation pipeline with dataset versioning and regression gates. It is aimed at prompt engineers and applied researchers who are still deciding what the prompt should be.
The cross-product is the actual mechanism
The README states that "ChainForge takes the cross product of inputs to prompt templates, meaning you can produce every combination of input values." This is the design decision the rest of the tool hangs on. If a template has a {game} variable with four values and you wire it to three models, you get twelve requests, not four or three. The response inspector then presents the results as a formatted table with plotting, so you can compare response length or a score across model and argument at the same time.
That combinatorial behaviour is also the main cost. Sending every combination multiplies API calls and spend by the number of variables and their value counts. The README describes the approach as "incredibly effective at sending off hundreds of queries at once," which is a fair description of both the benefit and the bill. Nothing in the supplied material describes a built-in cap on how many requests a single run can dispatch, so treat the graph as the throttle: keep the variable sets small while you are still shaping the prompt.
The architecture underneath is ReactFlow for the canvas and Flask for the backend, with the project written primarily in TypeScript. The browser draws and edits the graph; the Python process handles provider calls and, in a local install, the evaluation code.
Evaluation and ground truth via Tabular Data nodes
Comparison alone does not tell you which response is correct. The README describes ground truth evaluations using Tabular Data nodes: import a dataset, connect it to a template variable in a Prompt Node, and run. Each response is then compared against the expected answer, with the math-problem example given as the illustration.
The scoring itself is where the local install differs from the web version. The README states that in a locally installed version you can "write Python code to evaluate LLM responses." That means a scoring function is code you supply, not a menu of built-in metrics. The repository ships example evaluation flows, including what the README describes as 188 example flows generated from benchmarks in OpenAI evals, which is the fastest way to see the shape of a working scorer before writing your own. The docs also describe prompting a model to generate starter code for an eval, which is a convenience rather than a substitute for knowing what you want to measure.
Installing and running: pip, then chainforge serve
The documented path is short. With Python 3.8 or higher installed:
pip install chainforge chainforge serve
Then open localhost:8000 in Chrome, Firefox, Edge or Brave. API keys go in through the Settings icon in the top-right corner, or you can load them from environment variables, which the README recommends if you do not want to re-enter them each session. The supported provider list covers OpenAI, Anthropic, Google Gemini, DeepSeek, HuggingFace (Inference and Endpoints), Together.ai, Ollama, Microsoft Azure OpenAI Endpoints, Aleph Alpha and Amazon Bedrock on-demand inference including Anthropic Claude 3, with custom provider scripts for anything else.
There is a Docker route as well:
docker build -t chainforge . docker run -p 8000:8000 chainforge
and then http://127.0.0.1:8000. For sharing, the hosted version at chainforge.ai/play/ has a Share button that produces a link to a flow. The README notes two limits on that: ten shared flows at a time, and each flow under 5MB after compression. Exceeding ten breaks the oldest link, so the README tells you to export important flows to cforge files and treat Share as ephemeral.
What the hosted version leaves out
The README is explicit that the web version "has a limited feature set." Three capabilities are named as local-only: loading API keys automatically from environment variables, writing Python code to evaluate LLM responses, and querying locally-run models hosted via Ollama. If your evaluation depends on a custom scorer or on a model running on your own machine, the hosted play environment is not a substitute for the install.
This is a sensible split, since running arbitrary Python on a shared host is a different security proposition from running it on your laptop. It does mean the first thing to decide is where the work happens. Trying ChainForge on the web is a reasonable way to see whether the graph model fits how you think, but any serious scoring work moves to a local install and the provider keys move with it.
Where ChainForge is the wrong tool
The clearest boundary is durability. The README's own advice about Share links is telling: the oldest link breaks after ten shares, so flows should be exported to cforge files rather than relied on as hosted artifacts. Nothing in the supplied material describes scheduling, a server-side run history, or integration with a CI system. If your requirement is a nightly evaluation that fails a build when a metric regresses, ChainForge as documented does not provide that.
The second boundary is scale. The cross-product design is a feature when you are exploring a handful of variables and a liability when a template has many. There is no described sampling mode that would let you test a subset of combinations. You either narrow the inputs yourself or you pay for the full grid in requests and latency.
The third is scoring depth. A Python scorer you write is as good as you make it, and the material does not describe statistical treatment of results, confidence intervals, or repeated sampling to separate a real difference from run-to-run variance. For a quick read on which prompt looks better, that is fine. For a claim that one model is measurably better at a task, the graph alone will not support it.
The alternative: code-first evaluation frameworks
The obvious alternative is a Python evaluation library such as the OpenAI evals format that ChainForge itself draws example flows from, or a general evaluation framework driven from scripts and notebooks. The difference is in who holds the graph. In ChainForge the experiment is a document you edit on a canvas and export to a cforge file; in a code-first harness the experiment is source you check into version control, diff and run headless.
That difference cuts both ways. Code-first gives you reproducibility and CI, but building a comparison matrix means writing the loops yourself and re-running the whole script to change one variable. ChainForge gives you the matrix immediately and lets you edit it visually, at the cost of the experiment living outside your repository unless you export it. The README points to the 188 example flows as a starting point, which is effectively a bridge: you can begin in the canvas and move a flow's logic into code once the prompt stabilises.
Licence and maintenance cost
ChainForge is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are preserved. That is a permissive arrangement, and it means embedding the tool in an internal workflow does not create a copyleft obligation on your own code. This is a description of the licence text, not legal advice; if the tool becomes part of a product, have counsel read the actual LICENSE file rather than a summary.
The maintenance cost is the provider surface. The supported list spans OpenAI, Anthropic, Google Gemini, DeepSeek, HuggingFace, Together.ai, Ollama, Azure OpenAI, Aleph Alpha and Amazon Bedrock. Each of those has its own API shape and deprecation schedule, and each is a place where a release can break. You are depending on the project to keep that list current, or on the custom provider script path to carry your own integration. The release cadence visible in the repository is steady rather than rapid, with v0.3.6 in May 2025 adding a Media Node and image inputs, v0.3.5 adding favourites, an Ollama list, encryption and persistent settings, and v0.3.4 improving the table view and string storage. Upgrading is a pip install away, but a flow that depends on a node's behaviour is worth re-running after any version bump.
Editorial conclusion
Adopt ChainForge if your problem is comparison: you have a prompt template with variables and you need to see how it behaves across several models or parameter values before committing. Skip it if you need a deployed evaluation harness with versioned datasets and CI integration; the repository describes an interactive environment, not a pipeline. Before installing, verify three things: that your Python is 3.8 or higher, that the provider you intend to use is in the supported list or has a custom provider script, and whether you need Ollama or Python-based scorers, since the hosted version at chainforge.ai/play/ does not include them.
Community notes