Factorio Learning Environment: A REPL-Based Eval Harness for LLM Agents
A non-saturating, open-ended environment for evaluating LLMs in Factorio
At a glance
- What is it?
- FLE turns Factorio into an agent evaluation environment where the model acts by writing Python against a live game state. It is a research harness first and a product second, and the README is thinner on failure modes than on setup.
- Who is it for?
- Adopt FLE if you are evaluating code-writing agents on long-horizon planning and you accept that the environment itself is the research artifact, not a finished benchmark. Do not adopt it if you need a fixed task suite with published normalised scores, or if you cannot run Docker.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 4 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The Gap FLE Targets: Evals That Stop Being Hard
Most agent benchmarks saturate. Once frontier models clear the task set, the benchmark stops discriminating between them, and the field moves to a harder one. FLE's stated goal is the opposite: an open-ended, non-saturating environment. Factorio is the substrate because it is a resource management simulation with no fixed win condition in the way a puzzle benchmark has one. The README frames the contribution as building 'open-ended / unsaturateable evals for post-AGI frontier models', which is a claim about the shape of the task space rather than about any particular score.
The audience is narrow and specific. This is for researchers who want to measure whether a language model can plan over long horizons, write code that manipulates a live simulation, and recover from its own errors. It is not for teams looking for a drop-in safety classifier or a general-purpose agent runtime. The paper linked in the README (arXiv 2503.09617) is where the evaluation methodology lives; the repository is the machinery.
The REPL Loop: Code Synthesis as the Action Space
The mechanism is a Read-Eval-Print-Loop, and the README is unusually explicit about the three steps. Observation: the agent sees the world through the output streams (stderr and stdout) of its last program. Action: the agent generates a Python program to perform the desired action. Feedback: the environment executes the program, assigns variables, adds classes and functions to the namespace, and returns an output stream.
That third step is the design decision worth pausing on. The environment does not reset between turns. It accumulates. Variables the agent assigned in one program persist into the next, and the namespace grows as the episode continues. This means the agent's action space is not a fixed list of game verbs but the Python API surface the environment exposes, and the agent's own prior code becomes part of the state it must reason about. An agent that writes a helper function early and forgets its signature later will fail in a way that is invisible in a stateless tool-use benchmark.
It also means errors compound. A program that raises an exception produces an output stream the agent must read and interpret. There is no separate error channel described in the README. The same stderr and stdout that carry game observations also carry tracebacks.
Installation and the Cluster Command
The prerequisites are Docker, Python 3.10 or later, and Factorio version 2.0.73 or later if you want optional rendering. The game itself is not needed for the core loop, which matters because it means the cluster runs headless.
The core package installs from PyPI as factorio-learning-environment. Optional extras are declared per feature: [eval] for running experiments, [mcp] for MCP protocol support, [psql] for PostgreSQL support, and they can be combined as [eval,mcp,psql]. The README also suggests uv sync as the recommended path.
The quickstart is three commands. Activate the virtual environment with source .venv/bin/activate, start the game cluster with fle cluster start, then run evaluation trajectories with fle eval --config configs/gym_run_config.json. The fle CLI is the entry point for both cluster lifecycle and evaluation runs, and the config path is a real file in the repository rather than a placeholder.
What the README does not describe is what fle cluster start actually launches, how many containers it brings up, or how to tear it down. If you are provisioning this on shared infrastructure, that gap is worth resolving by reading the CLI source before you run it.
What the README Does Not Tell You About Running Evals
The eval extra is required for fle eval, but the README never documents the schema of configs/gym_run_config.json. It names the file and stops. That means the first real task for a new user is opening the config and reverse-engineering which fields select a model, which set an episode budget, and which point at the cluster. For a framework whose central claim is reproducibility of evaluation, leaving the config schema to the reader is a genuine weakness.
The same applies to results. There is a leaderboard link in the README, but no description of how a run's output is written, where trajectories land, or what format they take. The v0.4.2 release note mentions 'Lab Observation Test Coverage', which suggests the observation surface is under active test development, and v0.4.1 is labelled a 'Critical Direction System Hotfix'. A hotfix release for a direction system tells you the action semantics were recently wrong in a way that mattered. Pin a version rather than tracking main.
The licence metadata reports NOASSERTION. GitHub uses that string when it cannot match the repository's licence file to a known identifier. That is not the same as having no licence, and it is not the same as having a permissive one. Read the file.
Where FLE Is the Wrong Tool
FLE is a poor fit if you need comparable scores across models today. A saturated benchmark with a fixed task list gives you a number you can put in a table with a confidence interval. FLE deliberately avoids that shape. The task space is open-ended, which is the point, but it also means there is no single scalar that summarises an agent's competence. You get trajectories and whatever the leaderboard aggregates, and you inherit the job of arguing that your run is comparable to someone else's.
It is also a poor fit for teams without Docker on the eval host, or without the appetite to debug a game cluster. The dependency on Factorio 2.0.73 or later for rendering is optional, but the Docker prerequisite is not. If your environment forbids container runtimes, this framework does not have a path for you.
Finally, cost. Long-horizon episodes in a simulation mean many turns, and each turn is a code synthesis call plus an execution. The README gives no token budget guidance, no episode length defaults, and no cost estimate. Anyone planning a sweep should measure a single trajectory's cost before scaling.
The Alternative: Text-Only Tool-Use Benchmarks
The obvious comparison is a text-only agent benchmark where the action space is a fixed set of JSON tool calls and the environment returns structured results. SWE-bench-style harnesses and tool-calling suites work this way. The difference in approach is not difficulty, it is statefulness and error surface.
In a tool-calling benchmark, the environment validates the call before executing it. Passing the wrong argument type produces a schema error the harness can report cleanly. In FLE, the agent writes arbitrary Python. There is no schema between the agent and the simulation. A syntax error, a wrong API name, and a logically valid program that does the wrong thing all arrive as an output stream the agent must parse. That makes FLE a stricter test of code synthesis and self-correction, and a noisier one to score.
The MCP extra is the interesting middle ground. Installing factorio-learning-environment[mcp] exposes the environment over the Model Context Protocol, which means an agent can reach FLE through a standard tool interface instead of raw Python synthesis. That is a different evaluation than the REPL loop the README describes, and it is worth deciding which one you are actually measuring before you run anything.
Maintenance Cost and Version Discipline
The release cadence visible in the metadata is fast. Three releases landed in roughly ten days in March and April 2026, including a hotfix for a direction system and a release adding Inspect AI sandbox evaluation. A project moving that quickly will break config files and API surfaces. The practical move is to pin the package version in your environment and re-read the changelog before bumping.
The optional extras mean your dependency surface is your choice. If you only need the SDK, the core install avoids the eval, MCP, and PostgreSQL stacks. If you are running experiments, you are pulling in the eval dependencies plus whatever the cluster needs on the Docker side.
On licensing: because the metadata says NOASSERTION, do not assume the terms. Check the licence file in the repository and, if you plan to redistribute anything derived from the environment or its assets, get a human to read it. That is a boundary, not legal advice.
Editorial conclusion
Adopt FLE if you are evaluating code-writing agents on long-horizon planning and you accept that the environment itself is the research artifact, not a finished benchmark. Do not adopt it if you need a fixed task suite with published normalised scores, or if you cannot run Docker. Before committing, verify three things: that your Docker host can run the cluster started by fle cluster start, that the config file at configs/gym_run_config.json matches the model endpoints you intend to use, and what the repository's licence file actually grants, since the metadata reports NOASSERTION rather than a named licence.
Community notes