AndroidWorld: benchmarking phone agents on a live Android emulator

An environment where the phone is real

AndroidWorld is described in the README as an environment for building and benchmarking autonomous computer control agents, and the project page describes it as a reproducible benchmark of 116 hand-crafted tasks across 20 apps, dynamically instantiated with randomly generated parameters to create millions of unique task variations.

The choice of a live emulator rather than a simulation is the defining decision. An agent that drives a real Android stack encounters the real settings application, the real contacts database, the real permission dialogs, and the real consequences of a mis-tap. It also inherits the cost of that choice, because every episode involves a running emulator and a model.

The feature list in the README describes what the project believes those properties buy. There are 116 diverse tasks across 20 real world applications. The dynamic instantiation is what turns a fixed set of 116 tasks into millions of variations, since a task such as adding a contact takes parameters that are generated rather than fixed, so an agent cannot simply memorise a small number of end states. The reward signals are described as durable, which in this context means the same task reports success the same way every time rather than through a heuristic a model might exploit. The footprint is given as 2 GB of memory and 8 GB of disk, which is light for a full Android environment. The design is extensible, so new tasks and new benchmarks can be added. The environment is open, with access to millions of Android apps and websites, and it integrates with MiniWoB++, the web benchmark from Liu et al, so web tasks and phone tasks can be run in the same framework.

The project publishes a website, a paper on arXiv at 2405.14573, a task list page and a leaderboard in a shared spreadsheet, and it is hosted under the google-research organisation, which is where the reproducibility argument starts.

The stack: android_env, dm_env, gRPC and an accessibility bridge

The dependency list shows the architecture. android_env at 1.2.3 is the environment layer that talks to the Android stack, dm_env at 1.6 is the standard environment interface that the agent consumes, grpcio-tools at 1.71.0 and protobuf at 5.29.5 provide the RPC and schema layer, and the rest is ordinary Python: absl-py, numpy, pandas, opencv-python, pydub for audio, thefuzz and python-Levenshtein for string matching, tenacity for retries, jsonschema for validation, fastapi and uvicorn for the server, and IPython for the interactive parts.

The gRPC layer is not incidental. The setup.py file contains a custom setuptools command that generates Python protobuf bindings for two proto definitions in the information retrieval task evaluation, and it runs the protoc compiler from grpc_tools at build time rather than shipping generated files. That tells you the environment is a service, not a library, and that agents talk to it over RPC.

The bridge to the phone is an accessibility forwarding application, and that is why the emulator has to be launched in a particular way. The README is explicit that the emulator must be launched from the command line rather than through the Android Studio interface, with the -grpc 8554 flag, which is needed for communication with the accessibility forwarding app. That app is the piece that turns the operating system's accessibility tree into something the agent can read and act on, and without it the agent would be driving the screen blind.

The server directory in the repository holds the service side, and the app and android_world directories hold the task definitions and the environment implementation. The task definitions are what make a benchmark a benchmark, because they define both the goal and the check.

Setting up the emulator, the environment and the keys

The installation sequence is long but each step is specific.

First, install Android Studio and create an Android Virtual Device with the hardware set to Pixel 6, the system image set to Tiramisu at API level 33, and the AVD named AndroidWorldAvd. The exact hardware and image choices matter because the tasks are written against a particular configuration of the operating system.

Second, launch the emulator from the command line, and the README gives the command:

EMULATOR_NAME=AndroidWorldAvd # From previous step
~/Library/Android/sdk/emulator/emulator -avd $EMULATOR_NAME -no-snapshot -grpc 8554

The -no-snapshot flag matters for reproducibility, because a restored snapshot carries state from the previous episode.

Third, create a Python environment. Conda is recommended rather than required, and the example pins a specific interpreter version:

conda create -n android_world python=3.11.8
conda activate android_world

Fourth, install the project, and Python 3.11 or above is required:

git clone https://github.com/google-research/android_world.git
cd ./android_world
pip install -r requirements.txt
python setup.py install

Fifth, export the model provider keys, and the README gives the two the project expects:

# Add to .bashrc.
export OPENAI_API_KEY=your-key
export GCP_API_KEY=your-key

Sixth, install ffmpeg, which the environment needs for the screen and audio capture:

# Linux (Ubuntu/Debian)
# sudo apt update && sudo apt install ffmpeg

# macOS
brew install ffmpeg

The whole setup is therefore a full Android development environment plus a Python environment plus two API keys, and the last two are what make the benchmark cost money.

Running one task, and running the suite

The quickstart runs a single task to show the mechanics. The script initialises the environment, sets up a task, and runs the default agent, M3A:

python minimal_task_runner.py --task=ContactsAddContact

If you do not name a task, a random one is selected, which is a useful way to get a feel for the range. There is an important caveat for open source apps, that is, apps not included with the operating system: the README says to run --perform_emulator_setup in that case, and the same flag appears in the full benchmark run.

The cost warning is explicit. The quickstart script uses the legacy model gpt-4-turbo-2024-04-09 by default and the README says this model can be expensive, with the recommendation to change the model_name in the script for serious usage. A benchmark where the agent is a paid API call is a benchmark whose numbers move with the price list, and that is a structural property of the design rather than a misconfiguration.

The full suite is run with run.py, and the flags select the suite, the agent and the subset of tasks:

python run.py \
  --suite_family=android_world \
  --agent_name=t3a_gpt4 \
  --perform_emulator_setup \
  --tasks=ContactsAddContact,ClockStopWatchRunning \

The trailing comment marks the task list as optional, and the point of running a subset is cost control, since each task is a full episode.

One more piece of housekeeping matters for anyone comparing numbers. The README notes a task step limit update as of 2024-11-18, where the max_steps and step_budget for each task were changed to approximately twice the human average completion time, both to give agents enough time to finish and to reduce the overhead of running the benchmark. The per-task changes are in a linked spreadsheet. Any score published before that date was measured under a different budget, so the numbers are not comparable, and that is the sort of detail that separates a usable benchmark from a leaderboard that nobody can reason about.

Docker support, and the price of running an emulator in an emulator

The Docker path was added on 2025-06-02 and is labelled experimental, with the note that it has not been extensively tested. The value it offers is a consistent environment without the manual Android Studio steps, and the README says as much: simplify setup and ensure a consistent environment.

The build is a standard image build from the repository root:

docker build -t android_world:latest .

and the run publishes the service port and gives the container the privileges the emulator needs:

docker run --privileged -p 5000:5000 -it android_world:latest

The container starts the Android emulator and a FastAPI server, and the server is reachable on http://localhost:5000. The repository includes scripts/run_suite_on_docker.py as an example client for talking to the environment server in Docker, which is the same separation the gRPC architecture implies, an agent on one side and the environment as a service on the other.

The privileged flag is the part to notice. An Android emulator needs access to the kernel's virtualisation and input devices, and --privileged is the blunt way to grant it. The alternative is a more careful set of device and capability mappings, and the project has not made that the default.

The Apple Silicon note is the most useful piece of practical advice in the README, and it is also a warning. Installing the emulator package on ARM chips has known issues, so images built locally should target the AMD64 instruction set:

docker buildx build --platform linux/amd64 -t android-emulator:latest .

The consequence is stated plainly: running in a container on an Apple Silicon device runs quite slowly compared with running the device and emulator natively, because you end up running an Android emulator inside a Linux emulator. The Dockerfile reflects the same constraint, building on an x86_64 base with an x86_64 system image at API level 33.

Licence, versions and what this benchmark is not

The licence is Apache-2.0, with the LICENSE file at the repository root and the standard header in the source files. The project is hosted under the google-research organisation, and the repository is not archived with the last push on 2026-10-19. There are no GitHub releases, and the project version in pyproject.toml is 0.1.0.

The dependency pinning is where a practical constraint shows up. The requirements file fixes absl-py at 2.1.0, android_env at 1.2.3, dm_env at 1.6, thefuzz at 0.19.0, protobuf at 5.29.5, numpy at 1.26.3, pandas at 2.1.4 and matplotlib at 3.6.1, while google-generativeai is left unpinned. The Docker-only path has its own pyproject.toml with a slightly different set that adds fastapi and uvicorn, and the comment at the top of that file says the non-docker implementation uses the requirements.txt file. Two dependency manifests for one project is a mild smell, and it is the kind of thing that means you should test both paths before depending on either.

What this benchmark is not is worth stating. It is not a general agent framework, and it is not a way to run an agent on a real phone. It is an environment where a phone is a controlled resource: an emulator you can reset, an accessibility tree you can read, a task set you can instantiate with random parameters and a check you can run afterwards. That makes it excellent for comparing agents against each other, and poor for anything where the point is the messiness of a real device.

The cost model is the second limitation. Every episode is a model call, and the leaderboard is therefore a snapshot of models as they were when the numbers were produced, not of agents as an idea. A team adopting this should expect to pay for evaluation runs, and should budget the number of episodes before the number of tasks, because the parameterisation means a statistically meaningful result needs many more episodes than the 116 task names suggest.

Editorial conclusion

Adopt AndroidWorld when you are evaluating an agent that is supposed to operate an Android phone, because the tasks, the reward functions and the parameterisation are written to be comparable across runs in a way that a hand rolled test set is not. It is the wrong tool if you need a fast unit test loop, since every evaluation needs an emulator and a model, and it is the wrong tool if you want to run the suite on Apple Silicon hardware, where the documented path is a slow amd64 emulation. Before adopting, read the step budget change from 2024-11-18, because older published numbers used a different budget and are not directly comparable, and pin the dependency versions, since the requirements file fixes exact pins.

Frequently asked questions

Official sources

1. google-research/android_world on GitHub
2. Issues
3. License: Apache-2.0
4. README