MobileGym: a browser-hosted Android simulator where the judge reads JSON, not pixels

The three walls MobileGym claims to knock down

The README frames the problem as three failures of real-device and emulator pipelines. State is unreadable: adb and accessibility trees show the UI but not balances, orders or chat history, so verification falls back on a VLM judge, which the project measures at 10.2 percent misjudgment. State is unwritable: daily-app state lives in encrypted databases and server backends, so you cannot reset it or clone it, and group-relative RL methods such as GRPO need both. Side effects are irreversible: a transfer moves real money and a deactivation is permanent. MobileGym's answer is to stop driving the real app and reimplement it. Every app is a simulated one whose entire state is a structured JSON snapshot, which makes it readable, writable and disposable. The audience is narrow and identifiable: researchers running mobile GUI agent benchmarks who need identical initial conditions across models, and anyone attempting online RL on mobile agents where each rollout must be reset to a byte-identical starting state. If your evaluation does not require resetting account-bound state, the simulator's central advantage does not apply to you.

Programmable state as the core mechanism

The design decision that everything else follows from is that the environment is a JSON blob. The README states you can capture, configure, diff and restore the entire environment, and that initial state is exactly identical across all models and trials. Judges then read that state directly rather than inferring success from a screenshot. Two consequences are worth separating. First, verification becomes a programmatic check function shipped with each task, which the README describes as sub-millisecond verdicts at million-judgement scale. Second, full-environment state comparison becomes possible, so a run can be flagged for unexpected side effects such as an accidentally followed user or an inadvertently sent message. That second capability is the more interesting one, because it is a class of failure that real-device pipelines cannot see at all: if the state is not exposed, the side effect is invisible to the harness even though it happened on the device. The trade-off is that fidelity now means behavioural fidelity rather than pixel fidelity, a phrase the README uses explicitly. An agent trained here learns the app's state machine, not its rendering.

What a run actually looks like: install, boot, benchmark, train

The README's Quick Start is a five-step path. Install comes first, then boot the simulator, then talk to an agent in plain language, then run the benchmark, then train with RL. The repository requires Node 22 or newer and Python 3.11 or newer, per the badges in the README. The RL code lives in a mobilegym-rl/ directory and ships from v0.1.0, which the release notes call the first stable release and the recommended version for running experiments. The benchmark side is MobileGym-Bench, described as 256 test tasks; the README states a full 256-task evaluation finishes in about 6 minutes. The reported Sim-to-Real case study used 10 GRPO steps on one node with Qwen3-VL-4B, gaining 42.8 points in simulation and retaining 95.1 percent of that gain on a real device, or 40.7 points. Treat those numbers as the authors' reported results from the paper, not as independently reproduced figures. What the README does not give is the exact command lines and config keys for each step; the section headings name the steps but the cleaned text does not carry the invocations, so check the repository's Quick Start section directly before planning a run.

The AnswerSheet protocol and why free-text answers fail as a metric

One design choice stands out because it addresses a specific evaluation exploit rather than a performance concern. The README describes an AnswerSheet protocol in which agents fill structured forms with declared field types instead of returning free-text answers, with the stated reason that chain-of-thought leakage can no longer game the metric. This matters for any benchmark where the agent is asked a question about app state. If the answer is a string, a model that reasons well but reads state poorly can still land near the expected tokens, and a judge that compares strings will accept it. Forcing the answer into declared fields makes the check a type and value comparison against the JSON snapshot. The cost is that task authors must decide the field schema up front, which constrains what a task can ask. That is a real constraint, not a free win: a question whose correct answer is naturally open-ended has to be reshaped into fields or dropped. The README presents this as a solved problem, but the burden simply moves to whoever writes the task template.

Parallelism, resource envelope and where the numbers come from

The scaling claim is the most concrete thing in the README: roughly 400 MB RAM and about 50 MB disk per instance, 256 parallel instances on a single server using under 10 percent CPU, and about 3 seconds of cold start per instance. If those hold in your environment, the practical effect is that a 256-task evaluation is a coffee break rather than an overnight job, and a GRPO group can be rolled out on one machine. The claim is plausible given the architecture, since a JSON-state simulator has no GPU rendering or device virtualization in the loop, but it is a reported figure and the README does not break it down by host specification, which makes it hard to plan capacity against. The number that would change a procurement decision, CPU model and per-instance memory under a realistic agent policy, is not in the supplied material. Verify it on your own host before sizing anything. Also note that 256 instances is the figure the README gives for one server; it does not state whether that is a tested ceiling or a comfortable operating point.

Extending it, and the limits of a reimplemented app

New apps arrive through a manifest contract, and the README states that adding one requires no edits to the OS or benchmark layers. The same contract is described for new tasks, agents, judges and reward functions. The catalog is 28 simulated apps and 416 parameterized task templates, with every screen, transition and action of every app specified as a finite-state machine. That last detail is the honest limit of the approach. A finite-state machine spec is driveable by static analysis, which is a genuine advantage for coverage and for generating tasks, but it also means the environment contains exactly the states someone wrote down. Real apps have states that emerge from backend behaviour, partial failures, and version drift. MobileGym cannot surprise you with a state its spec does not include, and an agent that learns to exploit an underspecified transition will be rewarded for it. The README's own framing concedes the point: behavioural fidelity, not pixel fidelity. If your research question is about rendering, animation timing, network flakiness or any app outside the 28, this is the wrong tool, and the sim-to-real result does not extend to those cases because it was measured on a signal bucket tasks subset.

Alternatives: what you give up by not using a real device

The obvious alternative is the thing MobileGym argues against: driving real devices or a standard Android emulator through adb and the accessibility tree, with a VLM judge scoring the outcome. The difference in approach is not cosmetic. With a real device, the agent sees the actual rendering and the actual app, and any behaviour it learns is behaviour on that app. Verification, though, has to be inferred, which is where the README's 10.2 percent VLM misjudgment figure bites: a benchmark whose judge is itself a model has an error floor that no amount of agent improvement removes. The second alternative is a static offline dataset of trajectories, which is cheaper than either but cannot support online RL at all, since there is no environment to step. MobileGym sits between them: it gives up the real app to gain a readable, resettable, cloneable state. The choice is between an environment that is faithful and unverifiable and one that is verifiable and a reimplementation. Pick based on which failure your project can tolerate.

Licensing, maintenance and what to check before adopting

The code is Apache-2.0 and the data is CC BY-NC 4.0, per the two license badges and the separate LICENSE-DATA file. The split matters if you intend to use the shipped tasks or data commercially: the non-commercial term applies to the data, not the code, and that distinction should be confirmed with whoever owns your compliance decision rather than assumed from the repository badges. Maintenance signals in the supplied material are limited. The last push is dated 2026-08-28, the only releases are v0.1.0 and data-v0.1.0 from 2026-06-26, and the paper was accepted to EMNLP 2026 on 2026-08-21. That is an active project by its own timeline, but a single stable release means the API surface is young and the mobilegym-rl training code has shipped exactly once. The upgrade cost is therefore unknown rather than low. Before you build on it, read the manifest contract for one app end to end, run the 256-task benchmark on your own hardware to check the 6-minute and 400 MB figures, and confirm whether the RL path is tied to the Qwen3-VL-4B setup described in the release notes or accepts another checkpoint.

Editorial conclusion

Adopt MobileGym if your work needs deterministic, repeatable verdicts on account-bound mobile tasks and you want GRPO-style rollouts without a device farm; skip it if your research question is about real rendering, real network latency, or apps outside the 28 simulated ones, because the environment is a reimplementation, not a mirror. Before committing a training run, verify the two things the README does not settle: how a task template's judge function is written and registered in your target app, and whether the mobilegym-rl training path is pinned to the Qwen3-VL-4B configuration described in the release notes or can be pointed at another checkpoint.

Official sources

1. License: Apache-2.0
2. Project website
3. Purewhiter/mobilegym on GitHub
4. README
5. Releases