FineEnvs: open source RL environments you can actually run
FineEnvs — RL Environments 101: building and scaling RL environments in the age of LLMs
At a glance
- What is it?
- FineEnvs is a set of self-contained reinforcement learning environment projects for LLM agents, built so the same environment is implemented in several frameworks side by side, the rewards are served live, and every training curve comes from a job you can launch yourself.
- Who is it for?
- FineEnvs fits a team that has to pick an RL environment framework and wants to see what the choice actually costs, because project 00 implements three environments six times over and the diff between two server files is the lesson, and it fits anyone who wants an environment with a live reward server rather than a diagram of one. It does not fit a team that wants one production environment today, since most of the numbered projects are still marked as work in progress.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on October 23, 2026, and from our analysis. They are not legal advice.
Editorial analysis
The argument the project makes
The readme opens with a diagnosis of where the difficulty in RL for language models currently sits. The RL environment ecosystem is moving fast, new frameworks land every few weeks, each with its own vocabulary for the same handful of ideas, and most of what is written about them is either a launch post or a specification. Meanwhile the actual bottleneck has quietly shifted, because the algorithm is not the hard part any more and the environment is. That is the claim the whole repository is built around, and it is a narrower claim than it first appears. The project is not proposing a new training algorithm. It is proposing that the environments themselves deserve to be treated as the artefact worth sharing, and that the way to share them is to publish the environment, the deploy, the training job and the evaluation together so that someone else can reproduce the curve rather than trust a number. The stated promise is that every environment works, every rollout has been executed, and every training curve came from a job you can launch in one command without a GPU of your own.
How the projects are laid out
The repository is organised as numbered folders, and each one is a self-contained end-to-end project with its own environments, notebooks, results and README, plus the Hub repositories it owns. They read in order but stand alone, which is the design choice that makes the repository usable as a course rather than a monolith. Project 00, RL Environments 101, is the entry point and carries three environments, six frameworks and eight deployed Spaces, and it is marked stable. Project 01, LaTeX OCR, takes one environment all the way to a trained model and is also stable. Project 02, Watercolour, is marked as trained, and the rest, from 04 through 07, are marked as work in progress. The table itself is generated from each project's project.yaml manifest by a script in the tools directory, and the readme notes that adding a project means editing that manifest rather than the table, which is the kind of small automation that keeps a long index honest. On the Hub side, the environments, datasets, models and demos live on the FineEnvs organisation page, while the source, recipes and notebooks stay in the repository.
Project 00, which is the actual lesson
RL Environments 101 is described as one environment implemented six ways, and its stated purpose is blunt. Three environments, each implemented six times, same logic, six framework dialects, and the claim is that diffing any two server.py files and looking at the differences is the lesson. The three environments are a Jupyter agent with four tools running on an E2B sandbox with real code execution, a Wordle with one tool running in pure Python with no backend at all, and a desktop environment with nineteen tools running on E2B Desktop with vision. The six frameworks named are openenv, ors, nemo_gym, verifiers, skyrl_gym and gem, giving eighteen implementations in total, eight of them deployed as Spaces you can hit right now. The spread of the three environments is what makes the exercise useful. A Wordle needs nothing but Python, a Jupyter agent needs a real sandbox to execute code, and a desktop agent needs a whole virtual desktop, so the same conceptual environment turns out to need a very different amount of infrastructure depending on the task.
Projects 01 to 03, the three reward shapes
The next three projects each take a single environment through a full loop, and what changes between them is the kind of reward. LaTeX OCR trains Qwen3-VL-2B to read rendered maths into LaTeX with GRPO, scored by a reward served from a live OpenEnv Space, where the scorer compares normalised LaTeX using edit distance and exact match with a guard against padded completions. That is a verifiable reward, and the guard against padded completions is the detail that makes it a real one rather than a toy, since a model can otherwise score well by emitting padding. Watercolour asks what happens when there is no correct answer at all. Qwen3.5-35B-A3B learns to paint by writing p5.brush sketches, headless Chromium renders each sketch, and the render is scored by an aesthetic preference model plus a VLM judge comparing it against a hand-rated reference pool. GeoGuesser drops a VLM at a random street corner on Earth and scores it on kilometres of error. Between them the three cover verifiable string comparison, model-judged aesthetics, and a continuous spatial error.
Projects 04 to 07, the less finished ones
The later projects are where the scope widens and the status becomes provisional. SmolDataEnvs covers data-analysis tasks, verified SFT traces and a single-turn GRPO tutorial on HF Jobs. The multi-harness RL project trains on SmolDataEnvs using SETA, native OpenCode or Harbor, running on HF Jobs or Slurm, and it exists to test the claim that the same environment can be driven by different harnesses. Multilingual is the largest in scope: document OCR over a million pages in 22 languages and speech recognition over 102, each behind one OpenEnv server, with Gemma 4 trained to read and hear Kannada. The two simulation environments are the Port of Barcelona, one re-planning a broken week of container-ship dockings graded against a proven optimum, and one running a quay through a week as it unfolds with live news, pilots and tugs, real wind and shipping lines that answer back. The second set is credited to an outside contributor from Scale AI, and the v1 also credits an agentenv port by another name from the same organisation.
Configuration, and which keys each piece needs
The repository ships an example environment file that is more informative than it looks, because it documents which provider each kind of rollout expects. The file is copied to .env at the repository root and filled in with real values, and every rollout script reads from the RL_Envs_101 location of that file via python-dotenv.
# Hugging Face Inference Providers - used when ROLLOUT_MODEL contains a `:provider`
# suffix (default model is Qwen/Qwen3-Coder-480B-A35B-Instruct:together).
# Get a token at https://huggingface.co/settings/tokens
HF_TOKEN=hf_...
# OpenAI native - used when ROLLOUT_MODEL has no `:` (e.g. gpt-4o-mini, gpt-5).
# Get a key at https://platform.openai.com/api-keys
OPENAI_API_KEY=sk-...
# E2B Code Interpreter - required for:
# - any in-process env (verifiers, skyrl_gym, gem) -- the sandbox is created
# in the rollout process
# - running OpenEnv / ORS / NeMo Gym SERVERS locally (the deployed HF Spaces
# already have this baked in)
# Get a key at https://e2b.dev
E2B_API_KEY=e2b_...The selection rule is a suffix. A model name containing a colon means Hugging Face Inference Providers and the HF token, and a model name with no colon means OpenAI native and the OpenAI key. The E2B key is needed by any in-process environment and by the server implementations when they are run locally rather than as deployed Spaces. Weights and Biases logging is optional and is not used by the rollouts, only by the training scripts.
Where the fine detail lives, and how the guide fits
Two pointers matter for anyone who wants to go past the table. The first is the guide, published as a Space on Hugging Face under the author, described as the ultimate guide to RL environments, and the readme recommends reading it rather than working through the projects blind. The second is the collection pages on the Hub, where each project links to a collection holding the environment, the model, the dashboards and the run data, so the artefacts of a run are browsable alongside the code that produced them. The LaTeX OCR project is the one to read first if you want a complete picture, because it is described as the full loop and its environment source, its notebook and its HF Jobs scripts all live together with reproducible smoke tests. The repository itself has a CONTRIBUTING.md, a content/ directory, a tooling/ directory, a tools/ directory and a tutorials/ directory alongside the numbered projects, and a .claude/ directory at the top for agent configuration.
What this repository is not
It is worth being clear about the category, because the name invites a wrong comparison. FineEnvs is not a training framework and it does not compete with the algorithm libraries; the topics list mentions grpo, trl and rlhf because those are what the projects are trained with, not because the repository provides them. It is also not a paper with a method and a benchmark table. The claim is about environments, and the unit of work is the numbered project with its own environments, notebooks, results and README. The second boundary is the maturity of the content. Projects 00, 01 and 03 are marked stable, project 02 and project 06 are marked as trained, and projects 04, 05 and 07 are explicitly work in progress, so if you are picking something to build on, the status column is the thing to read. And the third boundary is the resource assumption: this is a repository that expects hosted GPUs through HF Jobs and a Hugging Face account with tokens, so it is aimed at people who have some access to that infrastructure rather than someone wanting to run everything locally from a laptop.
Editorial conclusion
FineEnvs fits a team that has to pick an RL environment framework and wants to see what the choice actually costs, because project 00 implements three environments six times over and the diff between two server files is the lesson, and it fits anyone who wants an environment with a live reward server rather than a diagram of one. It does not fit a team that wants one production environment today, since most of the numbered projects are still marked as work in progress. It also does not fit a reader who wants a paper-shaped result, because the pitch is explicitly that the resource is the environment work rather than a new algorithm. Before you start, copy the.env.example file to.env at the repository root and fill in the provider and sandbox keys, since every rollout script reads its configuration from there, and read the guide on the Hugging Face Space rather than only the project table.
Frequently asked questions
What is FineEnvs and what problem does it address?
It is a collection of open source reinforcement learning environments for LLM agents. Its argument is that the bottleneck in RL for language models has shifted from the algorithm to the environment, so it publishes environments, deployments, training jobs and evaluations together, so that every training curve comes from a job you can launch yourself in one command without a GPU of your own.
Why does project 00 implement the same environments several times?
Because the point is to show what a framework choice costs. Three environments are each implemented six times across openenv, ors, nemo_gym, verifiers, skyrl_gym and gem, giving eighteen implementations with eight deployed as Spaces, and the readme says the lesson is in diffing any two server.py files to see the differences between the framework dialects.
Which API keys do I need to configure?
Copy .env.example to .env at the repository root and fill it in. A ROLLOUT_MODEL with a colon suffix uses Hugging Face Inference Providers and needs HF_TOKEN, a model name with no colon uses OpenAI native and needs OPENAI_API_KEY, and any in-process environment or locally run server needs E2B_API_KEY for the E2B sandbox. Weights and Biases logging is optional.
Which projects are finished enough to build on?
The status table marks RL Environments 101, LaTeX OCR and GeoGuesser as stable, Watercolour and Multilingual as trained, and SmolDataEnvs, the multi-harness RL project and both PortSimEnv simulations as work in progress. Each numbered folder is self-contained with its own environments, notebooks, results and README.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/adithya-s-k-fineenvs)Was this useful?