NVIDIA-NeMo/labs-molt: a 9K-line PyTorch-native stack for agentic RL
An agentic-first RL framework for research (9k lines).
At a glance
- What is it?
- Molt is an Apache-2.0 research framework that keeps one trainable actor, uses Ray for placement and vLLM for rollout, and treats any Python you write inside an Env or ChatAgent as the reward. It is built for agentic RL experiments, not for RLHF breadth.
- Who is it for?
- Molt fits research groups that want to iterate on agent environments in plain Python while training on vLLM with FSDP2 and NVIDIA AutoModel, and who accept CLI-only configuration and a Python 3.10+ environment pinned to a specific AutoModel commit.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 17, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What Molt is for, and who it is not for
Molt targets a narrow problem: running reinforcement learning on agents where the environment itself is the interesting part. The README states that reward is any Python you write inside an Env or ChatAgent, which covers graders, multi-turn tools, VLM environments and LLM-as-judge calls. That framing matters. In most RLHF stacks the reward function is a configuration detail, often a separately trained model. Here the reward is code you own, and the trainer is meant to stay untouched while you iterate.
The intended user is a researcher who wants to read every line that touches a gradient. The README claims roughly 9.2K lines of RL code across three layers, with a footnote defining that count as every Python file the RL path uses, traced from the entry point molt.cli.train_rl_ray, excluding SFT, DPO/KTO/IPO trainers, reward-model training, distillation, vendored code, tests, examples, scripts and docs. That is a defensible definition, and it is also a marketing one: the same footnote measures OpenRLHF at about 7.2K lines on the same basis, so Molt is not the smallest by that metric. The real distinction is the design center, which the README states as agentic-first research rather than RLHF coverage.
The framework is not aimed at teams that want a broad menu of post-training algorithms. The package table lists SFT and RL entry points only (molt.cli.train_sft and molt.cli.train_rl_ray), with vLLM as the runtime. If your work is preference optimization on static datasets, this is the wrong shape of tool.
The three-box architecture and the token-first contract
The README describes three components plus one loop. Ray owns placement and the async queue that sits between the boxes. vLLM performs rollout. NVIDIA AutoModel with FSDP2 performs training in pure PyTorch. The architecture diagram in assets/molt.jpg shows the agent, the vLLM rollout stage, the Ray async queue and a single-actor AutoModel/FSDP2 trainer in a fully async arrangement.
What ties the boxes together is what the README calls a token-first contract: token ids, logprobs, action ranges, rewards and multimodal tensors stay aligned from rollout to training. This is the part worth understanding before adopting. Multi-turn tool calls and VLM inputs produce ragged sequences with different action boundaries, and a contract that keeps those aligned end to end is what lets the trainer consume them without a separate conversion layer. The README states that LLM-as-judge calls go back through the same vLLM engines that drive rollout, so judge traffic competes with generation for the same serving capacity. That is a scheduling decision with consequences: a heavy judge workload will show up as rollout latency, not as a separate line item.
On the training side, the stack is FSDP2 plus AutoModel with TP, EP and CP parallelism, described as MoE-native. The README gives DeepSeek-V3 with --fsdp.ep_size 256 as an example, alongside Adam CPU offload for the largest actors. It also states that the same script scales from 8B to 1T-class MoE without a rewrite. That claim is about the configuration surface, not about hardware: nothing in the README says an 8B recipe will fit on the same node count at 1T.
Installing Molt and running the FlashREINFORCE example
Molt is a Python package with a setuptools build backend. The README does not print a pip install line, so the practical entry point is the repository itself. requirements.txt pins several dependencies directly to git, including nemo-automodel at a specific commit (5c4616c679709ca97b35dd685c87b26bd82cf712) and dion from the microsoft/dion repository, plus ray[default]==2.55.0 and vllm. Python 3.10 or newer is required according to the README badge. A dockerfile/ directory exists at the top level, which is the other place to look if you would rather not resolve those pins yourself.
Because setup.py swaps git pins for a release floor when MOLT_PYPI_BUILD=1, a source or editable install keeps the exact AutoModel commit while a PyPI build does not. For research work where you need the pinned AutoModel revision, install from source:
pip install -r requirements.txt
pip install -e .The README points at a quick-start script for the FlashREINFORCE path, described as critic-free single-rollout RL with stable training beyond 6,000 steps. The script lives in the examples/scripts directory:
examples/scripts/quick_start/rl_flash_reinforce_r1d_1p5b.shThe README does not document the flags inside that script, so read it before running. What it should give you is a Ray-driven RL job that starts a vLLM rollout stage and a single-actor trainer. If you want the SFT path instead, the package table names molt.cli.train_sft as the entry point, and molt.cli.train_rl_ray for RL.
If you are working on the framework rather than with it, the test configuration in pyproject.toml sets testpaths to ./tests and defines unit, integration, system and acceptance markers, so pytest -m unit is the narrow first check.
Where Molt gets in the way
Configuration is the first constraint. The comparison table lists Molt's config surface as CLI flags only, against Hydra plus YAML for verl and CLI plus YAML for slime. That is a deliberate trade: flags are easy to read in a shell history and hard to version. A long training run becomes a long command line, and there is no config file to diff between experiments unless you write one yourself.
The second constraint is the dependency pin. nemo-automodel is referenced by git commit in requirements.txt, and setup.py carries a comment that source and editable installs keep the exact git pins because R3 needs that AutoModel commit. That means the framework is coupled to a specific revision of a fast-moving NVIDIA package. Upgrading AutoModel is not a version bump you can do casually; it is a change to the training backend.
The third is reward cost. Any Python you write is a valid reward, and the README notes that LLM-as-judge calls route through the same vLLM engines as rollout. An expensive grader therefore reduces rollout throughput. Nothing in the README describes a separate judge pool or a rate limit, so budget for that contention when designing an environment.
Finally, the topology. The comparison table lists Molt's RL topology as actor plus optional PPO critic, while OpenRLHF, verl and slime are listed with actor plus critic plus reward model. If your workflow assumes a trained reward model in the loop as a first-class component, you are building that yourself.
How it differs from OpenRLHF, verl and slime
The README's comparison table is the clearest statement of positioning, and it is worth reading as a set of trade-offs rather than a scoreboard. OpenRLHF uses DeepSpeed ZeRO-3 for training and is described as optimizing for RLHF coverage. verl uses FSDP, FSDP2 or Megatron, supports vLLM, SGLang and TRT-LLM as rollout engines, and is described as production breadth. slime is SGLang-only for rollout and Megatron-based for training, with the design center listed as Megatron throughput.
Molt's differences follow from that. Training is FSDP2 plus AutoModel rather than DeepSpeed or Megatron. Rollout is vLLM through Ray, with no SGLang or TRT-LLM option. Parallelism is TP, EP and CP, described as MoE-native. The config surface is flags rather than YAML. And the reward source is agent Python, where the others list combinations of agent code, reward models and endpoints.
The line-count footnote is the part to read carefully. It states that slime loads its Megatron and SGLang backends lazily, so its core package plus its model-zoo plugins (about 4.7K additional lines) are counted on the same basis as Molt's models directory. It also gives measurement dates and commit hashes for each project. Those numbers are a snapshot, not a property of the software, and the README says so. Treat the table as an argument about design center that happens to include code counts, not as a benchmark.
Maintenance, licence and upgrade cost
The repository is not archived, and the most recent push recorded is 2026-09-17, the same day as the v0.1.9 release. The two preceding releases are v0.1.8 on 2026-09-14 and v0.1.7 on 2026-08-28. That is a release cadence measured in days and weeks, which is consistent with a research project under active work. It is also a 0.x version line, so the absence of a stability promise is implied by the version number rather than stated anywhere in the README.
The licence is Apache-2.0, stated in the README badge and in the SPDX header of setup.py. Apache-2.0 permits commercial use and modification and includes a patent grant. The repository also ships a NOTICE file and a THIRD_PARTY_NOTICES.md, which is where you would look for attribution obligations attached to bundled or vendored components. Note that requirements.txt pulls in third-party code under its own terms, including dion from microsoft/dion and nemo-automodel from NVIDIA-NeMo/Automodel; those licences are separate from Molt's. This is a description of what the files say, not legal advice.
Upgrade cost concentrates in two places. The AutoModel commit pin means the training backend moves when you move it, deliberately. And the CLI-flags-only config surface means an upgrade that renames a flag is a change to every launch script you keep, with no YAML to migrate. The examples/scripts directory is the reference for what the current flags look like.
Editorial conclusion
Molt fits research groups that want to iterate on agent environments in plain Python while training on vLLM with FSDP2 and NVIDIA AutoModel, and who accept CLI-only configuration and a Python 3.10+ environment pinned to a specific AutoModel commit. It does not fit teams that need a PPO critic plus a separate reward model by default, or that want Hydra and YAML config files, since the README describes the topology as actor plus optional PPO critic and the config surface as CLI flags only. Before adopting, verify the AutoModel commit pin in requirements.txt resolves in your environment, check whether the dockerfile/ directory covers your CUDA and driver combination, and confirm that the FlashREINFORCE example script matches the model and parallelism you intend to run.
Frequently asked questions
What does molt mean in science?
In biology, molting is the process of shedding an outer layer such as skin, feathers or an exoskeleton. The name of this repository is a reference to that idea rather than a description of the software, which the README describes as an agentic-first RL framework for research.
What is the molt meaning?
The general meaning of the word is the shedding of an outer covering by an animal. NVIDIA-NeMo/labs-molt borrows the word as a project name; the README does not explain the naming choice.
What is NVIDIA NeMo labs-molt?
It is an Apache-2.0 Python framework for agentic reinforcement learning research, built on Ray for placement and async queues, vLLM for rollout, and NVIDIA AutoModel with FSDP2 for training. The README states that reward is any Python you write inside an Env or ChatAgent.
How do I install NVIDIA NeMo labs-molt?
The README does not give a pip install command. The repository ships requirements.txt, which pins nemo-automodel to a specific git commit, and setup.py, which keeps those pins for source installs. A top-level dockerfile/ directory is also present.
Does NVIDIA NeMo labs-molt need a separate reward model?
Not by default. The README's comparison table lists Molt's RL topology as actor plus optional PPO critic, with agent Python as the reward source, while it lists OpenRLHF, verl and slime as actor plus critic plus reward model.
Community notes