AppWorld: A Sandboxed Benchmark for Agents That Call APIs and Write Code

The Gap AppWorld Fills: Multi-Step Tasks With a Checkable End State

Most function-calling evaluations hand a model a schema and a single request, then check whether the emitted call matches a reference. That format cannot express a task like booking a trip that requires reading a calendar, checking a balance, and then writing a script to reconcile the two. AppWorld targets exactly that gap. The README describes it as a high-fidelity execution environment of nine day-to-day apps, operable via 457 APIs, populated with digital activities of roughly 100 people living in a simulated world, plus a benchmark of natural, diverse and challenging autonomous agent tasks requiring rich and interactive coding. The intended user is a researcher or agent engineer who needs a repeatable score for a multi-step agent, not a developer looking for a tool-calling library to ship. Two properties matter here. The world is controllable, meaning the same task ID produces the same starting database state, and the evaluation is not a string match against a reference trajectory. The repository also carries an MCP server and an MCP client, so the same environment can be driven from outside the Python API. That combination is why the paper took an ACL'24 Best Resource Paper award, and it is also why the project is best understood as infrastructure for measurement rather than as a component you embed in a product.

Inside the Engine: Nine Apps, 457 APIs and a Per-Task Database

The material available describes the architecture at the level of what the agent touches. An agent receives a task instruction, then works inside a world that exposes an apis object. The README's TLDR snippet shows the shape: a loop over task IDs, a context manager that opens a world for one task, an instruction read from world.task.instruction, and a call to world.execute with a code block. In that code block, the agent writes ordinary Python and reaches the apps through attributes on apis, as in response = apis.spot. This is the interactive coding part of the title. The agent is not restricted to emitting JSON tool calls; it can loop, branch, transform data and call several APIs in one submission. Each task opens its own world, so state changes made during a run do not leak into the next task. The task suite is split into named sets: train, dev, test_normal and test_challenge, and load_task_ids is the function that returns the list for a given split. The environment can be served with or without Docker, per the README's section on serving the environment and APIs, which matters when the code the agent writes needs isolation. The API Explorer and Task Explorer on the project site are the intended way to inspect the 457-API surface and the task text before writing agent code, since neither is enumerated in the README itself.

Getting a First Run: install, download data, then open a world

Installation is three shell commands, quoted directly from the README. First pip install appworld. Then appworld install, which performs the environment setup. Then appworld download data, which fetches the world database and task data. The package requires Python 3.11 or later, per the badge in the README. After that, the shortest path to a running agent is the pattern in the TLDR block: import AppWorld and load_task_ids from the appworld package, iterate over load_task_ids("test_challenge") or another split, and open each task with AppWorld(task_id=task_id, experiment_name="sample") as world. The experiment_name argument is what groups run outputs, and the README's agent experiments section covers running and evaluating agents under that name, along with downloading the authors' own experiment outputs for comparison. For a working reference rather than a skeleton, the README points at notebooks/minimal_agent.ipynb, a minimal ReAct agent. Two further hooks are worth knowing before you write much code. Adding your base LLM is documented as a step, so a model that is not already supported needs an adapter. And customizing agents is documented separately, which implies the default agent configuration is meant to be overridden rather than used as-is.

Evaluation Is State-Based, Not Trajectory-Based

The README lists evaluating the agent as its own walkthrough step, and the surrounding structure tells you why that step is nontrivial. Because the agent writes code and calls APIs inside a world, the natural thing to score is the state of that world when the run ends, not the sequence of calls that produced it. Two agents can reach the same correct end state by different routes, and a benchmark that graded the route would penalize a valid solution. This is the design choice that separates AppWorld from single-turn function-calling suites, where the emitted call is the artifact under test. The practical consequence for anyone building on it: your agent's score depends on side effects inside the nine apps, so debugging a failure means inspecting the world, not just the transcript. The README's mention of downloading the authors' experiment outputs is useful here, because it gives a reference point for what a given score looks like in practice. Note also that the task splits are fixed and named, so a comparison across two agents is only meaningful if both ran the same split under the same experiment configuration.

Two Constraints the README States Plainly: Development Restrictions and Code Execution Safety

The walkthrough contains two sections whose titles are warnings rather than features: Agent Development Restrictions and Code Execution Safety. Both are listed as top-level steps alongside the primer and the minimal agent, which suggests they are conditions of valid use rather than optional reading. The restrictions section implies that some agent designs are not permitted within the benchmark, which is a reasonable rule for a shared leaderboard but an easy thing to miss if you skim the install commands and start coding. The safety section concerns where agent-written code actually runs, and the README separately documents serving the environment and APIs with or without Docker. That is the real trade-off. Running without Docker is lighter and faster to set up, but the code your agent writes executes somewhere you have to think about. Running with Docker costs you container management. Neither option is described in the README as a default recommendation, so this is a decision you make deliberately. For a benchmark that invites arbitrary Python from a language model, the Docker path is the one that matches the threat model, and the absence of a stated default is a gap worth flagging.

Where AppWorld Is the Wrong Tool, and What to Use Instead

AppWorld is a fixed world. The nine apps, their 457 APIs and the roughly 100 simulated people are the benchmark, not a plugin system you extend for your own domain. If you want to test an agent against your company's internal ticketing API, AppWorld will not host it; the README does document developing new apps and new task generators in its guides, but that is a substantial contribution to the benchmark rather than a configuration step. If your actual question is whether a model can emit a well-formed call for a given schema, a single-turn function-calling suite answers it faster and with less setup, because there is no world to download and no code execution to sandbox. If your question is whether an agent can plan and recover over many steps in a live system, AppWorld is closer to the right instrument than a static prompt set, but it is still a simulation. The honest boundary is this: AppWorld measures agent behaviour inside AppWorld. A high score is evidence about multi-step API-and-code competence in a controlled setting, and the README makes no claim beyond that. Treating the score as a proxy for production reliability would be reading more into it than the material supports.

Maintenance, Releases and the Apache-2.0 Terms

The release history is uneven in a way worth planning around. v0.1.3 and v0.1.2 both landed on 2024-12-03, and v0.1.3.post1 arrived on 2026-02-17, a post release rather than a minor bump. The repository's last push is dated 2026-09-04, so there is activity between releases, but the version number has stayed in the 0.1.x line. For a benchmark, that pattern has a specific cost: a post release can change task data or evaluation behaviour without a version signal that reads as breaking. Pin the version you evaluated against and record it alongside your scores, because a leaderboard entry without a version number is not reproducible. The project is Apache-2.0 licensed, which permits commercial and academic use and modification, with the usual conditions around notices and patent grant; the README keeps a separate License section, and the full text governs rather than any summary here. This is not legal advice. The practical licence question for most users is narrow: if you extend the engine or add apps, Apache-2.0 lets you do that, and if you redistribute a modified engine you carry the notice obligations that come with it. The README also mentions a Release Disclaimer section, which is worth reading before you cite any result from the package.

Editorial conclusion

Adopt AppWorld if you are evaluating an agent that must both call APIs and write code across multiple steps, and you want a scored environment rather than a demo. Do not adopt it if you need a production tool runtime or a single-turn function-calling test; the nine apps and their API surface are fixed by the benchmark. Before committing, verify the Python 3.11+ requirement, run appworld install and appworld download data, and read the Agent Development Restrictions and Code Execution Safety sections, because both constrain how you may write your agent and where its code runs.

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. StonyBrookNLP/appworld on GitHub