Meta Agents Research Environments: a runner for the Gaia2 dynamic agent benchmark

The gap ARE is built to fill: static benchmarks versus scenarios that change mid-run

Most agent benchmarks hand the model a fixed prompt and a fixed set of tools, then score the final answer. ARE is aimed at the case where that framing breaks down. The README states the platform introduces evolving environments where agents must adapt their strategies as new information becomes available, and it lists dynamic environments, multi-step reasoning and a real-world focus as the three gaps being addressed. The scale claim attached to that is concrete: tasks requiring 10 or more steps and several minutes to complete, and a Gaia2 benchmark of 800 scenarios across 10 universes. The intended user is a researcher or evaluation engineer who already has an agent and needs a harder test for it. ARE is not trying to be the agent. The default agent referenced in every command is a reference implementation that plugs into the harness, and the repository is organised around scenarios, apps and events rather than around an agent runtime.

What actually runs: scenarios, apps, events and the Gaia2 split

The vocabulary the documentation uses is agents, environments, apps, events and scenarios, and the scenario is the unit of work. A scenario is executed against an environment, the agent acts, and the harness records the trajectory. The GUI section of the README describes a Scenarios Mode with structured task execution and evaluation plus a DAG visualisation, which tells you the harness models a scenario as a dependency graph rather than a linear script. That matters for evaluation design: a DAG can express steps that must complete before others become available, which is how an environment evolves while the agent is still working. Gaia2 is the benchmark distributed on top of this machinery, published as a Hugging Face dataset at meta-agents-research-environments/gaia2 with at least a validation split, and described as a follow-up to Gaia covering a broader range of capabilities. Omnilingual-Gaia2 is a multilingual translation of the same scenarios, hosted as facebook/omnilingual-gaia2, with the translation pipeline living in gaia2-cli/mt. Note that the scenarios are data, not code: they are fetched at run time via the --hf and --hf_split flags rather than bundled into the wheel.

Getting a first scenario to run with uvx

The README recommends uv, and the shortest path avoids a permanent install. The command given for a custom scenario is uvx --from meta-agents-research-environments are-run -s scenario_tutorial -a default, and for the benchmark it is uvx --from meta-agents-research-environments are-benchmark gaia2-run --hf meta-agents-research-environments/gaia2 --hf_split validation -l 1. The -s flag selects a scenario by name, -a selects the agent, and -l caps how many scenarios run. Two pip variants are documented: a minimal install of meta-agents-research-environments for CLI benchmarking, and meta-agents-research-environments[gui] which adds the web interface. The README calls the GUI variant recommended for local development, and are-gui -s scenario_find_image_file starts it, typically on http://localhost:8080. The GUI has two modes, a Playground for direct chat-style interaction and Scenarios for structured execution with the DAG view. Model selection goes through LiteLLM, so the flags are --model, --provider and, for self-hosted endpoints, --endpoint. The README shows LLAMA_API_KEY exported for the llama-api provider and --provider local --endpoint "http://localhost:8000" for a local deployment.

Two stacks in one repository, and the container question you have to answer first

The README opens with a pointer that is easy to skim past: for the container-based Gaia2-CLI benchmark stack, start with gaia2-cli. That means the repository contains more than one way to run the benchmark. The Python package gives you are-run, are-benchmark and are-gui as entry points. The gaia2-cli directory is a separate stack, container-based, and it is also where the Omnilingual-Gaia2 translation pipeline lives under gaia2-cli/mt. Nothing in the supplied material states which of the two produces the results on the leaderboard, or whether they are numerically equivalent. If you intend to publish numbers, that is the first thing to resolve, because a container-based runner and a pip-installed runner can differ in pinned dependencies, environment setup and tool availability. Treat the two as distinct execution paths until the documentation says otherwise, and pick the one that matches whatever the leaderboard expects.

Where ARE is the wrong tool

ARE evaluates agents. It does not give you one. If your goal is to build an agent that books meetings or edits files, the harness is overhead: you would be adopting a scenario runner, a DAG model and a benchmark dataset in order to test something you have not written yet. The default agent exists so that the harness has something to drive, and the README's own framing places it alongside --agent default in every example, which is a reference implementation rather than a component to build on. The second limitation is cost. The README states that tasks require 10 or more steps and several minutes to complete, and that Gaia2 spans 800 scenarios. A full sweep is therefore long and, if you are calling a hosted model, expensive per run; the -l flag exists precisely because you will want to cap it. The third is that the benchmark is not static data you can evaluate offline in a spreadsheet. Scenarios are fetched from Hugging Face at run time, so a network dependency sits between you and a reproducible score. The fourth is that this is research software from a research organisation. There are no releases retrieved for this repository in the material supplied, so there is no version history to reason about, and no information about how scenario definitions change between runs. Pin what you can and record the split you used.

Alternatives and the actual difference in approach

The obvious comparison is the original Gaia benchmark, which the README names directly as the predecessor and links to its paper. The difference is scope rather than mechanism: Gaia2 is described as evaluating a broader range of agent capabilities, and it adds the dynamic, evolving environment layer that a fixed question-and-answer benchmark does not have. If your agent's weakness is tool use on a single well-specified task, the original Gaia framing may be sufficient and cheaper to run; if the weakness is replanning when the situation changes, Gaia2 is the one that exercises it. A second comparison is against building your own harness with a general agent framework. That gives you full control over the environment and no dependency on someone else's scenario format, at the cost of writing the environment, the scoring, the DAG and the visualisation yourself. ARE's value is that all four already exist and share a data model. A third is the container-based gaia2-cli stack inside the same repository, which is not really an alternative so much as a parallel path with different operational assumptions. The honest summary is that ARE competes with your internal evaluation harness, not with agent frameworks, and it wins on the scenarios rather than on the code.

Licence and the ongoing cost of keeping an evaluation current

The repository is MIT licensed, which is permissive and places few constraints on how you use the code. That covers the harness. It does not automatically cover the datasets: Gaia2 and Omnilingual-Gaia2 are hosted as separate Hugging Face artifacts, and the README does not state their terms. Check those independently before redistributing scenarios or publishing derived results, and treat this as a question for whoever handles licensing on your side rather than something the repository answers. On maintenance, the material supports a limited but real observation: the last push is dated 2026-08-26, the repository is not archived, and the README carries a New marker on the Omnilingual-Gaia2 addition, so the project is being extended rather than frozen. What that means in practice is that the benchmark surface can move. A multilingual extension was added after the initial Gaia2 work, and scenario sets fetched by name from Hugging Face can change underneath a pinned package. If you are tracking scores over time, record the dataset revision alongside the package version and the split name, because the --hf and --hf_split flags identify the data but the README does not show a revision pin.

Editorial conclusion

Adopt ARE if you need to score an agent against the Gaia2 scenarios and want the harness, the DAG visualisation and the leaderboard format supplied rather than assembled. Do not adopt it if you want a library to build an agent: ARE evaluates agents, it does not provide one, and the default agent is a reference implementation, not a product. Before committing, verify three things: which scenarios the validation split actually contains, whether your model provider is reachable through LiteLLM with the provider and endpoint flags shown in the README, and whether the container-based gaia2-cli stack is required for the evaluation you intend to publish. The MIT licence settles the code question, but the Gaia2 scenarios and the Omnilingual-Gaia2 translations are hosted separately on Hugging Face, so check the terms attached to those datasets before you redistribute anything.

Official sources

1. facebookresearch/meta-agents-research-environments on GitHub
2. Issues
3. License: MIT
4. Project website
5. README