# NeMo Skills, a harness for training and evaluating models on Slurm or a laptop

> NeMo Skills is a set of pipelines for improving language model skills, covering synthetic data generation, training and evaluation, with a configuration model that lets a job written for one workstation run on a Slurm cluster with a one-line change. Its own README now points new benchmark work at a separate repository, NeMo-Gym.

**NVIDIA-NeMo/Skills** — A project to improve skills of large language models

- Repository: https://github.com/NVIDIA-NeMo/Skills
- Website: https://nvidia-nemo.github.io/Skills/
- Stars: 1,057 · Forks: 212
- Language: Python
- License: Apache-2.0
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/nvidia-nemo-skills

## One harness from synthetic data to evaluation

NeMo Skills is described as a collection of pipelines to improve the skills of large language models, and the scope is stated in one sentence: it supports everything needed for LLM development, from synthetic data generation, to model training, to evaluation on a wide range of benchmarks. The design goal is also stated in one sentence, and it is the most useful thing in the README. Start developing on a local workstation and move to a large-scale Slurm cluster with just a one-line change.

That one-line change is the whole architectural idea. The pipeline is described declaratively and the execution backend is a configuration value, so the same job description runs as a local process or as a distributed Slurm job. The feature list for flexible LLM inference makes the same point in three parts. You can switch between API providers, a local server and large-scale Slurm jobs for inference. You can host models, on one node or many, with TensorRT-LLM, vLLM, sglang or Megatron. And you can scale synthetic data generation jobs from one GPU on a local machine all the way to tens of thousands of GPUs on a Slurm cluster.

The evaluation side is organised by capability rather than by benchmark, and the organisation is the useful part. There is natural language math with aime24, aime25 and hmmt_feb25. There is formal language math with minif2f, proofnet and putnam-bench. There is code with swe-bench, livecodebench and bird. There is scientific knowledge with hle, scicode and gpqa. There is instruction following with ifbench and ifeval. There is long-context with ruler, mrcr, aalcr and longbench-v2. There is tool-calling with bfcl_v3. There is multilingual with mmlu-prox, flores-200 and wmt24pp. There is speech and audio with asr-leaderboard and mmau-pro. And there is vision-language models with mmmu-pro.

Training goes through NeMo-RL or verl, and the evaluation pipelines can be parallelized across many Slurm jobs, self-host LLM judges, and bring your own prompts or change benchmark configuration in any other way. The self-hosted judge is worth noting, because a judge that runs on a hosted API turns a large evaluation run into a large API bill.

The project is licensed Apache-2.0, requires Python 3.10 or newer, and the last push was on 2026-10-15 with the default branch main.

## The partition that has already happened

The most important thing in this README is a note at the very top, and it is a migration notice rather than a feature list.

Benchmarks and rollouts are moving to NeMo-Gym. The project encourages you to move your rollout workflows over to NeMo-Gym, and to implement new benchmarks there rather than in Nemo-Skills. But Nemo-Skills will continue to add support for running NeMo-Gym benchmarks, so you can run them with the same cluster configs, server configurations and Slurm setup you already use. The benchmark implementation itself should live in NeMo-Gym.

That is a careful split, and the reason to understand it is that the two projects are not redundant. NeMo-Gym becomes the place where a benchmark is defined, and NeMo-Skills remains the place where a benchmark is run, because the running is where the cluster configuration, the model server configuration and the Slurm plumbing live. An early version of the NeMo-Gym bridge exists today as the ns nemo_gym_rollouts command, and the README is explicit that the interface is experimental and will likely change.

The practical consequence for an adopter is a decision about where your new work goes. If you are writing a new benchmark, it belongs in NeMo-Gym. If you are running an existing benchmark on a Slurm cluster, or evaluating a model you just trained, NeMo Skills is the tool and the cluster configs you already have still apply. Anyone picking this up today should expect the split to continue, and should expect the experimental rollout command to move.

The repository layout is consistent with a project of this size. There is a nemo_skills/ package directory, core/ and tools/ directories, a cluster_configs/ directory for the cluster settings that the one-line change switches between, a recipes/ directory holding the reproduction recipes, a requirements/ directory, dockerfiles/, docs/ and a root-level mkdocs.yml, tests/, and a dataset_explorer_demo/ directory. There is also a.pre-commit-config.yaml, a.coderabbit.yaml and a greptile.json, so the repository uses automated code review in addition to pre-commit.

## Recipes, and what the news section is actually for

The News section of the README is a changelog of reproduction recipes, and reading it tells you what kind of artefact this project produces.

On 12/15/2025 it released the recipe for reproducing the Nemotron-Math-v2 and Nemotron-Math-Proofs-v1 datasets, which were used as part of the training data for NVIDIA-Nemotron-3-Nano-30B-A3B-BF16. On 11/25/2025 it added the recipe for reproducing the main experimental results for Scaling Generative Verifiers For Natural Language Mathematical Proof Verification And Selection, which is an arXiv paper. On 08/22/2025 it added details for reproducing evals for the NVIDIA-Nemotron-Nano-9B-v2 model, and on 08/15/2025 the same for Llama-3_3-Nemotron-Super-49B-v1_5.

On 07/30/2025 the datasets used to train OpenReasoning models were released, with math and code available as part of Nemotron-Post-Training-Dataset-v1 and science in OpenScienceReasoning-2. And on 07/18/2025 the OpenReasoning models were released themselves.

The pattern across all of these is that a recipe is a runnable description of how to reproduce a dataset or a number, rather than a description of how to do it in prose. That is a stronger artefact, because it can be re-run by someone else and checked. It also explains why the repository has a recipes/ directory as a first-class citizen rather than a documentation folder, and why the project treats a benchmark implementation and a benchmark run as separate things.

The models and datasets are on Hugging Face, and the recipes in this repository point at them. The evaluation results are published as a graph in the README, and the documentation site carries per-benchmark guides with the configuration for each.

One more piece of the tooling is worth naming, because it is unusual to see in a research harness. The dataset_explorer_demo/ directory suggests a tool for inspecting a dataset interactively, which is the sort of thing that saves a lot of time when a training run is behaving oddly and the question is whether the data is what you think it is.

## The command line, and what the packaging tells you

The package installs as nemo_skills, and it exposes a single console script, ns, mapped to nemo_skills.pipeline.cli:app. So the entry point is the pipeline CLI and the commands hang off it, with the one documented command being ns nemo_gym_rollouts for the experimental NeMo-Gym rollout bridge.

The pyproject.toml shows a packaging arrangement worth understanding, because it explains how the project handles a large optional dependency set. The version, dependencies and optional-dependencies are all marked dynamic. The version comes from an attribute, nemo_skills.version.__version__, and the dependencies are read from files, with core/requirements.txt and requirements/pipeline.txt named as the sources. Optional dependencies are handled the same way, with a comment explaining that the tools extra is for the MCP and tool runtime only.

Reading dependencies out of files rather than declaring them inline is a deliberate choice for a project with several install profiles, and it is also what lets a container image for one pipeline avoid pulling the whole world. The classifiers list Python 3 and Python 3.10, and requires-python is 3.10 or newer.

The documentation is MkDocs, with mkdocs.yml at the root and the site published at nvidia-nemo.github.io/Skills. The per-pipeline documentation links in the README are structured as a documentation site with a section per capability area, which is the right shape for something this wide.

For a reader evaluating the project, the honest summary is that this is a research infrastructure repository rather than a library. It expects a Slurm cluster, it expects GPUs, and its value is that the same configuration covers both the small experiment and the large run. The Apache-2.0 licence and the public recipes mean you can read exactly how the published numbers were produced, which is rarer than it should be in this area.

## Conclusion

NeMo Skills suits a research team that wants one harness for synthetic data generation, training and evaluation, and that runs the same pipeline on a workstation during development and on a Slurm cluster for the real runs, since the configuration carries across. It is the wrong choice if you are building new benchmarks, because the README states that benchmarks and rollouts are moving to NeMo-Gym and asks that new benchmark implementations live there. Before starting a pipeline, read the configuration model rather than the pipeline code, since the cluster, server and Slurm settings are what change between a laptop and a cluster, and note that the current `ns` command is the only documented entry point, with the NeMo-Gym rollout interface marked as experimental and likely to change.

## FAQ

### What does NeMo Skills cover?

It is a collection of pipelines for improving LLM skills, supporting synthetic data generation, model training with NeMo-RL or verl, and evaluation on a wide range of benchmarks. The design goal is that you start on a local workstation and move to a large-scale Slurm cluster with a one-line configuration change.

### Where should I implement a new benchmark?

In NeMo-Gym, not here. The README states that benchmarks and rollouts are moving to NeMo-Gym, encourages you to implement new benchmarks there, and says the benchmark implementation itself should live in NeMo-Gym. NeMo Skills will keep supporting running those benchmarks with the cluster and Slurm setups you already use.

### Which inference servers can NeMo Skills host models with?

TensorRT-LLM, vLLM, sglang and Megatron, on one node or many. Inference can also come from API providers or a local server, and synthetic data generation jobs scale from one GPU locally to tens of thousands of GPUs on a Slurm cluster.

### What are the recipes in the NeMo Skills repository?

They are runnable descriptions for reproducing datasets and published results. The README news records recipes for the Nemotron-Math-v2 and Nemotron-Math-Proofs-v1 datasets, for the experimental results in the paper on scaling generative verifiers for mathematical proof verification, and for reproducing evals of the NVIDIA-Nemotron-Nano-9B-v2 and Llama-3_3-Nemotron-Super-49B-v1_5 models.

### What command does NeMo Skills provide?

The package exposes a single console script named ns, mapped to nemo_skills.pipeline.cli:app. The documented command is ns nemo_gym_rollouts for the early NeMo-Gym rollout bridge, which the README describes as experimental and likely to change.

## Sources

- [Issues](https://github.com/NVIDIA-NeMo/Skills/issues)
- [License: Apache-2.0](https://github.com/NVIDIA-NeMo/Skills/blob/main/LICENSE)
- [NVIDIA-NeMo/Skills on GitHub](https://github.com/NVIDIA-NeMo/Skills)
- [Project website](https://nvidia-nemo.github.io/Skills/)
- [README](https://github.com/NVIDIA-NeMo/Skills/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/nvidia-nemo-skills