BioNeMo Recipes: TransformerEngine checkpoints and FSDP training recipes for biological models

The gap BioNeMo Recipes fills between a prototype and a multi-node run

A researcher who has a working protein language model on one GPU usually faces the same next problem: making it train on many GPUs without rewriting the model. The README frames the project as an alternative to a batteries-included framework, offering model checkpoints with TransformerEngine layers and training recipes that show how to reach high throughput with open source frameworks and fully sharded data parallel scale-out. The stated audience is two groups. Foundation model developers who are scaling prototypes, and domain scientists who want to fine-tune existing models on proprietary data for drug discovery. Both groups are served by the same artifact: a directory you can read, copy and modify. The project does not ask you to import a training library. It asks you to take a recipe and adapt it. That distinction matters, because it means the code is the deliverable, and the quality of your result depends on how carefully you read it.

What a recipe directory actually contains

The repository is organised around two kinds of directories. Under models/ there are TE-accelerated model definitions: amplify, esm2, llama3, mixtral and qwen. Under recipes/ there are training scripts that pair a model with a training stack. The recipe names encode the stack. esm2_native_te is the ESM-2 and AMPLIFY recipe on native PyTorch. llama3_native_te is Llama 3 on native PyTorch. esm2_accelerate_te uses HuggingFace Accelerate. codonfm_ptl_te uses PyTorch Lightning. The remaining recipes cover OpenGenome2 data for Llama 3, CodonFM on native PyTorch, Geneformer with mFSDP and FP8, and a vision transformer. The support table is the most informative part of the README, because it is not uniform. esm2, llama3 and mixtral support FSDP, BF16, FP8, THD, FP8 combined with THD, MXFP8, NVFP4 and context parallelism. amplify supports FSDP, BF16 and FP8, with THD and FP8+THD marked under development, and no MXFP8, NVFP4 or CP. qwen marks FSDP as under development. The vit recipe supports mFSDP and BF16 only, with THD, FP8 and the lower-precision formats listed as not supported. If you assume every recipe has the same feature set, you will pick the wrong one.

Reading the support matrix before you clone

The table uses three states: supported, under development, and not supported. The difference between the last two is practical. Under development means the maintainers intend to add it, so the directory may contain partial code or none at all. Not supported means the combination is not planned in this table, so a recipe that lists THD as not supported will not give you a sequence-packing path today. This is where the project is honest and where it is thin. The table tells you what exists, but it does not explain why a given recipe lacks a feature. It also does not state which GPU generation each precision format requires, and the README does not define the abbreviations in the excerpt available here. Before committing to a recipe, match three columns against your setup: the parallelism column (FSDP, FSDP2 or mFSDP), the precision column (BF16, FP8, MXFP8, NVFP4), and the THD and CP columns if your sequences are long. A recipe that supports NVFP4 is only useful if your hardware can run it, and the repository does not answer that question for you.

Getting a recipe running

The README does not include installation commands, a requirements file listing, or a run command in the material available here, so the exact invocation has to come from the recipe directory itself. What the README does establish is the shape of the workflow: you enter a recipe directory such as recipes/esm2_native_te or recipes/llama3_native_te, and the training script there drives the corresponding model under models/. The framework varies by recipe, so the entry point does too. A native PyTorch recipe and a PyTorch Lightning recipe will not share a launcher. The checkpoints are the other half of the setup. The README points to nvidia/AMPLIFY_350M and nvidia/esm2_t48_15B_UR50D on Hugging Face, which means the intended path is to start from a published TE checkpoint rather than convert one yourself. Treat the recipe directory as the source of truth for commands and configuration keys, and treat this article as a map of which directory to open. Anyone who needs a single documented command line for the whole repository will not find one in the README.

Where the recipe approach stops working

The trade-off is explicit in the framing: this is not a batteries-included framework. That buys extensibility and costs you a supported interface. There is no API stability promise across the recipes, and the version history supports that reading. The releases jump from v2.6.3 in July 2025 to v2.7 in October 2025 to v3.0.0 in June 2026, with the v3.0.0 tag described as BioNeMo Recipes rather than as a framework version. If you write production training code against a recipe directory, you own the upgrade. A second limitation is coverage. The table lists a fixed set of models, and a model that is not there has no recipe. A third is the licence. The repository metadata supplied here does not state a licence, so you cannot determine your redistribution or commercial-use rights from this material. For a project that ships model checkpoints and training code intended for drug discovery work, that is a gap you should close before anything else, and it is not a question this article can answer for you.

How this differs from Hugging Face Accelerate and PyTorch Lightning

The interesting detail is that BioNeMo Recipes does not compete with those frameworks. Two of its recipes use them. esm2_accelerate_te runs the ESM-2 and AMPLIFY models through HuggingFace Accelerate, and codonfm_ptl_te runs CodonFM through PyTorch Lightning. The difference in approach is what each layer owns. Accelerate and Lightning own the training loop, the launcher and the distributed orchestration. BioNeMo Recipes owns the model definition with TransformerEngine layers, the checkpoint, and the parallelism and precision configuration that the recipe applies on top. That is why the support columns differ between the Accelerate recipe and the native recipe for the same model family: esm2_accelerate_te lists FSDP and FSDP2 with BF16 and FP8, while esm2_native_te lists mFSDP and FSDP2 with the full precision and THD set. Choosing between them is a choice about how much of the training loop you want to control. If you already have a Lightning or Accelerate pipeline, the matching recipe is the smaller change. If you want the widest set of precision and sequence-packing options for ESM-2, the native recipe is the one with the fuller row.

Upgrade cost and the licence question

Three releases in roughly eleven months, with a major version bump from v2.7 to v3.0.0, suggests the recipes are still moving. Because the code is meant to be copied and adapted, a version bump does not silently change your training run the way a library upgrade would. It changes the reference you are diverging from. The cost shows up later, when you want a fix or a new precision format and have to diff your modified recipe against the new one. The licence is the harder issue. The metadata supplied here lists the licence as unknown, and the README excerpt does not name one. Model checkpoints and training code can carry different terms, and the checkpoints referenced on Hugging Face have their own pages. This is not legal advice, and the only reliable step is to read the licence file in the repository and the licence listed on each checkpoint page before you plan around either.

Editorial conclusion

Adopt BioNeMo Recipes if you already train transformer-based biological models in PyTorch and want a working FSDP2 or mFSDP starting point for ESM-2, AMPLIFY, Llama 3, Mixtral, Qwen, CodonFM or Geneformer, with FP8 and THD support already wired into the recipe you pick. Do not adopt it if you need a supported framework with an API contract, if you need a licence you can read before cloning, or if your model is not in the table. Verify three things first: the licence file at the repository root, the exact recipe directory whose support row matches your hardware and precision target, and whether the features you need are marked supported or still under development in that row.

Official sources

1. Issues
2. NVIDIA-BioNeMo/bionemo-recipes on GitHub
3. Project website
4. README
5. Releases