IntellAgent: policy-graph simulation for conversational agent evaluation
A framework for comprehensive diagnosis and optimization of agents using simulated, realistic synthetic interactions
At a glance
- What is it?
- IntellAgent is a Python framework that decomposes an agent prompt into a policy graph, samples policy combinations, and runs simulated user conversations against them. It is a scenario generator and critic, not a test runner, and the repository is still at version 0.0.1.
- Who is it for?
- Adopt IntellAgent if your conversational agent's behaviour is already specified in a long system prompt and you want coverage you did not think to write by hand. Do not adopt it if you need a CI gate with deterministic assertions, or if you cannot route simulation traffic to a provider whose content filters you control.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap IntellAgent targets: untested policy combinations
Conversational agents are usually specified as a long system prompt containing rules. A refund policy, an escalation path, a tone constraint, a data-handling rule. Handwritten test suites cover these rules one at a time. Real users do not respect that separation. They ask for a refund while angry, while referencing an order that does not exist, while switching topics mid-conversation. The interesting failures live in the combinations, and the combinations are what nobody writes tests for.
IntellAgent's stated goal is to generate those combinations. The README describes it as a multi-agent framework that "simulates thousands of realistic, challenging interactions" and "stress-tests agents to uncover hidden failure points." The unit of work is not a test case but a scenario: a synthetic user, a synthetic database state, and a policy subset that the conversation is meant to exercise. The intended user is someone who owns a deployed or near-deployed conversational agent and wants coverage evidence before real traffic finds the gaps.
The framing has a consequence worth stating early. IntellAgent does not tell you whether your agent is correct. It tells you which policies were exercised and what a critic model thought of the responses. Those are different claims, and the README does not blur them, even if the marketing language around "reliable real-world deployment" invites the blur.
The three-stage pipeline: decompose, sample, simulate, critique
The README describes three steps, and the mechanism is specific enough to reason about.
First, decomposition. Given the user prompt, plus optional material such as tool definitions and a database schema, the system "decomposes the prompt into a policy graph." The graph is the framework's internal representation of what your agent is supposed to do. This is the load-bearing step. Everything downstream is generated relative to this graph, so an incomplete decomposition produces scenarios that look thorough while missing the rules you actually care about.
Second, sampling. The system "samples a subset of policies based on their concurrence in real conversation distributions." That phrase matters. The sampling is not uniform over the graph. Policies that co-occur in real traffic are more likely to be combined, which is a reasonable bet: rare pairings are less likely to be hit by users, so testing them has lower expected value per sample. It also means the framework needs some notion of a real distribution to sample from, and the README does not explain where that distribution comes from when you are testing a new agent with no traffic history.
Third, generation and simulation. A scenario is generated to address the selected policy subset, including the system databases the conversation needs. Then a user agent plays the user side of the interaction against your chatbot. Finally, a critic reviews the conversation and gives feedback on the tested policies. The output is not a pass or fail verdict. It is a critique keyed to policies, which is why the visualization step exists.
Getting a run to completion: config keys and commands
The prerequisites are narrow. Python 3.9 or later, a clone of the repository, and dependencies from requirements.txt. The README gives the clone command as an SSH URL, so you need a key registered with GitHub or you swap in HTTPS yourself.
LLM credentials go in config/llm_env.yml. The README shows the OpenAI shape:
openai: OPENAI_API_KEY: "your-api-key-here"
Provider selection is per-role, not global. In a config file such as config/config_education.yml, the llm_intellagent key sets the provider for the framework itself and llm_chat sets it for the chatbot under test. Both accept values like 'azure'. The README lists OpenAI, Azure, Vertex and Anthropic as supported options, so the two roles can run on different providers, which is useful when you want the critic on a stronger model than the agent being tested.
Scenario volume is controlled by dataset.num_samples, shown as 30 in the example. Worker concurrency and timeouts live in the config_default file, and the troubleshooting notes say to decrease num_workers on rate-limit errors and increase timeout values on frequent timeouts.
A run is one command. The fast path, with no database:
python run.py --output_path results/education --config_path ./config/config_education.yml
The slower path, with a database:
python run.py --output_path results/airline --config_path ./config/config_airline.yml
Results are viewed through a Streamlit app: streamlit run simulator/visualization/Simulator_Visualizer.py. One operational detail is easy to miss. The README instructs Azure OpenAI users to disable the default jailbreak content filter before running the simulator. A scenario generator that deliberately constructs adversarial user turns will trip that filter, and leaving it on produces failures that look like simulation bugs.
Cost is a first-class config knob, and the numbers are the vendor's
The README states that with default parameters the expected cost per sample is approximately $0.10, and that a cost_limit parameter in the config file bounds total spend. Treat that figure as a vendor estimate for the default configuration, not a property of your setup. Cost scales with num_samples, with the number of policies in your graph, and with the token prices of whichever providers you assign to llm_intellagent and llm_chat.
The cost_limit parameter is the more useful fact. It means a run has a hard ceiling, which matters because the failure mode of a scenario generator is generating more scenarios. A misconfigured num_samples on a large policy graph can produce a bill before it produces a report. The README also mentions work on using user data to reduce per-sample cost, which is a roadmap item, not a shipped feature.
One thing the material does not settle: whether cost_limit aborts the run, truncates the sample set, or stops generating new scenarios while finishing in-flight ones. That distinction changes what a partial results directory means, and the README does not say.
Where IntellAgent is the wrong tool
The framework evaluates the agent as a whole, through conversation, using a model as the judge. If your requirement is a deterministic assertion (this input must produce this tool call with these arguments), IntellAgent is the wrong layer. Its output is a critique, and critiques vary between runs. You cannot put a critique in a CI gate without a threshold, and the README does not describe one.
There is a second boundary in the integration story. The roadmap shows LangGraph checked and CrewAI and AutoGen unchecked, with API integration for external conversational agents still pending. If your agent is not reachable through the supported path, you are writing the adapter yourself. The README calls integration "simple" but does not show the interface, so the claim is unverified from the repository description alone.
The optimization half of the pitch is also partly gated. The roadmap lists system prompt optimization, tools optimization and graph structure optimization under a premium offering. The open repository diagnoses; the closed product fixes. That split is worth knowing before you plan a workflow around it.
Finally, the project is at version 0.0.1, released 2025-01-22, with the last push in July 2026. A 0.0.x version number is an honest signal about interface stability, and the repository makes no compatibility promise.
Alternatives and the actual difference in approach
DeepEval and Ragas take the opposite stance on where test cases come from. Both are Python evaluation libraries where you supply the dataset: a list of inputs, optionally expected outputs or retrieved contexts, and a set of metrics. The metric layer is the product, and it is deterministic in the sense that the same input and the same metric version produce the same score. IntellAgent inverts this. The dataset is the product, and the metric is a model-generated critique. If your problem is that you have a test set and no way to score it, IntellAgent does not help. If your problem is that you have a scoring method and no test set, that is exactly the gap it claims to fill.
LangSmith and similar tracing platforms sit at a third point: they observe real traffic and let you turn production traces into regression cases. That approach has an advantage IntellAgent's README implicitly concedes with its talk of sampling "based on their concurrence in real conversation distributions." Real traffic is the ground truth for what users actually do. IntellAgent's value is that it runs before you have traffic, or on combinations that traffic has not yet produced. The trade-off is that synthetic scenarios are only as realistic as the policy graph they were derived from. A tracing-based workflow cannot invent a scenario nobody has hit; a simulation-based workflow can invent one that no user would ever hit.
Licence, maintenance and what to verify first
The repository is Apache-2.0, which permits commercial use, modification and redistribution, with the usual requirements around preserving notices and stating changes. That covers the open code. It does not cover the premium optimization features the roadmap points to, and it does not govern the terms of whichever LLM providers you configure in llm_env.yml. Those provider terms, not the Apache grant, determine what you may do with the simulated conversations, especially if the scenarios embed customer data or database schemas. This is not legal advice; read the licence file and your provider agreements.
Maintenance cost is mostly configuration drift. The framework has two provider roles, a sample count, a cost ceiling, a worker count, a timeout, and per-environment config files. Every one of those is a knob that can silently change what a run means. Upgrading is the other cost: with a single 0.0.1 release and no stated compatibility policy, expect config keys to move between versions, and pin the commit you validated rather than tracking main.
What to verify before you rely on it: run the education config first because it skips the database, inspect the policy graph produced from your own prompt, and check that the graph contains the rules you would enforce in code review. The graph is the input to everything else, and it is the one artifact you can audit without trusting a model's judgement.
Editorial conclusion
Adopt IntellAgent if your conversational agent's behaviour is already specified in a long system prompt and you want coverage you did not think to write by hand. Do not adopt it if you need a CI gate with deterministic assertions, or if you cannot route simulation traffic to a provider whose content filters you control. Before committing, run config_education.yml with num_samples left at its default, confirm the cost_limit value that ships with it, and read the policy graph the decomposition step produces for your own prompt. If that graph does not reflect the rules you actually enforce, the scenarios downstream will be plausible but wrong.
Community notes