Sample Factory: a PPO library built around parallel environment throughput

A policy gradient library organised around environment throughput

Sample Factory is described as a high throughput reinforcement learning codebase, and the package metadata calls it a "High throughput asynchronous reinforcement learning framework". The focus is policy gradients, specifically PPO, and the axis of optimisation is the number of environment steps the trainer can push through per second. That is a different design target from libraries that optimise for the elegance of a single-agent training loop.

The README lists what it considers the key features: an optimized algorithm architecture for maximum learning throughput, both synchronous and asynchronous training regimes, a serial single-process mode for debugging, and comparable performance on CPU-based and GPU-accelerated environments. It also covers single- and multi-agent training, self-play, training multiple policies at once on one or many GPUs, Population-Based Training, discrete, continuous and hybrid action spaces, and vector, image and dictionary observation spaces.

Two details stand out for a reader deciding whether to invest. Model architectures are created automatically by parsing the action and observation space specification, so a baseline often needs no model code at all. And the library is designed to be imported rather than only run, with custom environments described as first-class citizens.

Synchronous, asynchronous, and the serial debugging mode

The most important architectural distinction in Sample Factory is between the synchronous and asynchronous regimes, and the documentation gives them dedicated pages. In the synchronous setup, policy updates and environment collection are coupled, which makes learning curves easier to reason about. In the asynchronous setup, actors collect experience while the learner updates, which keeps the hardware busy and is where the throughput advantage comes from.

Then there is serial mode, described in the README as running a single process for easy debugging. This is the detail that matters most in practice for anyone adopting the library. A parallel RL stack that only misbehaves across dozens of workers is a serious problem, and a single-process mode where the same code path runs serially means you can attach a debugger, print observations, and reproduce a failure without a cluster.

One thing to keep in mind is that async throughput is a trade-off rather than a free win. The README frames the library as reaching state-of-the-art performance while minimising required training time and hardware, but the debuggability cost of a distributed run is not discussed, so a team should plan on the serial mode as part of the normal workflow rather than as a debugging afterthought.

Installing from PyPI and training your first Mujoco agent

Installation is a single command, and the README says to install from PyPI:

pip install sample-factory

The base install is deliberately thin. Each environment integration lives in an extra, so a Mujoco run needs the mujoco extra:

python -m sf_examples.mujoco.train_mujoco --env=mujoco_ant --experiment=Ant --train_dir=./train_dir

The command line follows a pattern that repeats across the examples. An environment module from sf_examples is launched with python -m, then three flags: which environment, a name for the experiment, and a directory to write checkpoints and tensorboard logs into. The README's workflow is to stop the run with Ctrl+C once the desired performance is reached, then evaluate the trained agent:

python -m sf_examples.mujoco.enjoy_mujoco --env=mujoco_ant --experiment=Ant --train_dir=./train_dir

A third script trades rendering for speed. The comment in the README notes that it is an alternative evaluation path with no rendering but much faster, and it warns to keep sample_env_episodes at or above num_workers times num_envs_per_worker:

python -m sf_examples.mujoco.fast_eval_mujoco --env=mujoco_ant --experiment=Ant --train_dir=./train_dir --sample_env_episodes=128 --num_workers=16 --num_envs_per_worker=2

Pixel-based environments follow the same shape, though the VizDoom example needs the vizdoom extra installed first.

Environment integrations, extras, and what setup.py pins

The repository ships example integrations for ViZDoom, IsaacGym, DMLab-30, Megaverse, Mujoco and Atari, and the README shows clips of agents trained in each. The package is built around a setup.py that declares these as optional dependency groups: atari maps to gymnasium[atari, accept-rom-license], mujoco to gymnasium[mujoco] with mujoco<2.5, nethack to nle ~= 1.0 with gymnasium < 1.0, plus envpool, pettingzoo[classic] and an onnx group pulling in onnx and onnxruntime. The core install requires numpy>=1.18.1,<2.0, torch>=1.9,<3.0 excluding 1.13.0, gymnasium>=0.27,<1.0, pyglet, tensorboard, tensorboardx and psutil.

Those pins are the real specification of what the library supports. The numpy ceiling below 2.0 is a hard constraint that will bite anyone building in a newer scientific stack, and torch is allowed a wide range while excluding one specific broken release, which suggests the upper bound is what was tested rather than a hard compatibility claim.

A build detail worth knowing: pyproject.toml lists Cython in the build requirements alongside setuptools and wheel, so a source install compiles rather than installing a pure-Python wheel. The package version in setup.py is 2.1.3, while the only tagged GitHub release is 1.0.0, dated 2020-06-24 and associated with ICML 2020.

Monitoring, model upload, and the docs build

Experiment tracking is integrated rather than bolted on. The README points at detailed WandB and TensorBoard summaries, and at a metrics reference and a custom metrics page, so you can log your own scalars next to the built-in ones. There is also a HuggingFace integration for uploading trained models and metrics to the Hub, which is a modest but useful addition for teams that already keep checkpoints there.

The documentation is a mkdocs site hosted at samplefactory.dev, and the repository carries mkdocs.yml, a docs/ directory and a Makefile target that prepares and serves it:

make docs-serve

The Makefile shows the rest of the developer workflow as plain make targets: format runs black and isort, check-codestyle runs black in check mode, isort in check mode and flake8, test runs pytest with -s --maxfail=2 -rA, and test-cov and test-cov-core run coverage. Building and publishing is a separate pair, build and upload through twine, with an upload-test target against testpypi. Formatting is consistent across the project, with black and isort configured for a line length of 120 and a py38 target version.

One caveat: the README itself is a short overview and points to the site for the rest, so the details of a specific environment integration or of the custom model API live outside the repository.

Where Sample Factory is the wrong tool

The clearest boundary is the operating system. The README states that Sample Factory is known to work on Linux and macOS and that there is no Windows support at this time. If your training fleet is Windows, this is a non-starter, and no workaround is described.

The second boundary is ecosystem shape. The integration list is Gymnasium, Torch, WandB, TensorBoard, HuggingFace, ONNX. There is no mention of other RL libraries to interoperate with, and the design goal is importing Sample Factory into your own project, which means adopting it is closer to taking ownership of a training stack than to adding a dependency.

Third, the performance framing needs care. The claim that the implementation reaches state-of-the-art performance across a wide range of domains is the authors' own claim, and the evidence in the repository is a set of demonstration clips. The clips show agents that learned, but they carry no throughput measurements, no comparison baselines run on identical hardware, and no seeds. If your decision hinges on how many samples per second you will actually get, the numbers to trust are the ones you measure on your own cluster, and the serial mode is the first place to start.

Maintenance, licensing, and the gap between 1.0.0 and 2.1.3

The repository is not archived, and the last push was on 2026-10-19, so the code itself is being worked on. The release history tells a different story. The only GitHub release is 1.0.0 from 2020-06-24, and the version in setup.py is 2.1.3, with the README announcing that version 2 is out. In other words, the versions that matter were published to PyPI rather than cut as GitHub tags with release notes.

That has a practical consequence. There is no changelog in the repository listing what changed between releases, and no upgrade guide for moving from version 1 to version 2. Anyone maintaining a long-lived training setup is effectively tracking the PyPI package and the master branch, with a Discord server and a Twitter account as the change notification channels rather than a versioned changelog.

Licensing is simple. The project is MIT licensed, the LICENSE file sits at the repository root, and the badge in the README matches. The package is designed to be imported into other codebases, so the permissive license is a reasonable fit for commercial work, though specific dependencies such as mujoco, nle and the atari ROM licence carry their own terms that a team should check separately.

Editorial conclusion

Sample Factory suits engineers who already have a PPO baseline that is bottlenecked on environment throughput and who can work on Linux or macOS with a Python 3.8 era toolchain; it does not suit people who need Windows support, a modern release cadence, or a library with a deep-learning-ecosystem integration story beyond Gymnasium, Torch, WandB and TensorBoard, and before adopting it you should confirm that the 2.1.3 package on PyPI still matches the checkout you read, since the only tagged GitHub release is 1.0.0 from 2020-06-24.

Frequently asked questions

Official sources

1. alex-petrenko/sample-factory on GitHub
2. License: MIT
3. Project website
4. README
5. Releases