NVIDIA NeMo Curator: GPU Pipelines for Text, Image, Video and Audio Curation
Scalable data pre processing and curation toolkit for LLMs
At a glance
- What is it?
- NeMo Curator packages the filtering, deduplication and classification stages of LLM dataset preparation into a Ray-executable pipeline with RAPIDS and vLLM underneath. It is a fit for teams already on NVIDIA hardware and a poor fit for anyone who wants a plain pip install and a single-machine job.
- Who is it for?
- Adopt NeMo Curator if you are preparing text, image, video or audio corpora on NVIDIA GPUs and you want the same pipeline definition to run on a laptop and on a multi-node Ray cluster, because that portability is the actual product here. Do not adopt it for a one-off CSV cleanup on CPU, for a non-NVIDIA accelerator fleet, or if you cannot accept a pinned vLLM and RAPIDS dependency set.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The dataset preparation problem NeMo Curator is aimed at
Training a language model starts with a corpus that is too large, too noisy and too duplicated. The work between raw web data and a trainable dataset is mostly mechanical: strip boilerplate, detect language, score quality, remove near-duplicate documents, classify by domain. The README describes the scale it was built for: the Nemotron-4 pre-training dataset was curated with NeMo Curator's text pipeline across more than 8 trillion tokens of multilingual web data. That number sets the design target. A script that reads a JSONL file and writes a filtered one is fine at a few gigabytes and painful at a few terabytes. NeMo Curator is for ML engineers and data teams who treat curation as a pipeline they will run again next month, not a notebook they ran once. The README states the intended audience directly: teams that need repeatable curation pipelines rather than ad-hoc scripts, and that need GPU and distributed execution for the stages that dominate runtime, which are deduplication, classification, embedding and inference.
Ray as the execution layer, RAPIDS and vLLM as the engines
The architecture visible in the README is a Ray-based pipeline that spans all four modalities, a change the update notes date to the 26.02 release. A pipeline is a sequence of processing stages, and the same definition is meant to execute on a laptop or across a multi-node Ray cluster, which is the claim the project leads with. Underneath, the heavy stages lean on NVIDIA libraries: the README states that NeMo Curator leverages RAPIDS (cuDF, cuML, cuGraph) and the truncated text cuts off mid-sentence there. Deduplication, embedding generation and quality classification are the stages that benefit, since they are either graph problems or GPU inference problems. Text, image, video and audio each get their own set of operations. For text the README lists deduplication, classification, quality filtering and language detection. For image: aesthetic filtering, NSFW detection, embedding generation, deduplication. For video: scene detection, clip extraction, motion filtering, deduplication. For audio: ASR transcription, quality assessment and WER filtering. The practical consequence is that the pipeline graph is uniform but the operators are modality-specific, so a text pipeline and a video pipeline share orchestration and little else.
Installing it: three paths and one non-negotiable override file
NeMo Curator installs through uv, not plain pip. The README instructs you to install uv first with the shell script from astral.sh. Path A is a CPU smoke test: create a virtual environment, then uv pip install "nemo-curator[text_cpu]", then import nemo_curator and print __version__. Path B is the GPU text pipeline. It requires CUDA 12, a driver that supports it, Linux x86_64, roughly 16 GB of GPU memory and network access to Hugging Face. The command downloads requirements/text_cuda12-overrides.txt from the repository, then runs uv pip install with --override pointing at that file, --torch-backend cu129, and an extra index URL at wheels.vllm.ai/0.22.0/cu129, installing the nemo-curator[text_cuda12] extra. The README is explicit that standard pip install is not supported for text_cuda12 because vLLM and RAPIDS declare incompatible Numba requirements, and that any uv pip install including text_cuda12, including nemo-curator[all], needs the override file. From a source checkout, uv sync --extra text_cuda12 and uv sync --extra all apply the project override automatically. Path C is the NGC container, which the README recommends for video and audio because those pipelines depend on system codec libraries that the published container ships preconfigured. The tutorial entry point for Path B is tutorials/quickstart.py, which starts Ray, downloads a Hugging Face model and runs a sentiment classification pipeline on GPU.
The dependency override is a real constraint, not an installation footnote
Two libraries that both matter here want different versions of Numba, and the project's answer is an override file rather than a resolved dependency graph. That is a legitimate engineering choice when you control the tested combination, and it is also a constraint you inherit. Every environment you build has to carry that override, and the README says so twice, once for the direct extra and once to warn that nemo-curator[all] is affected too. The failure mode is quiet: an install that skips the override can resolve to a combination the project has not tested, and the symptom will show up at runtime in a GPU stage rather than at install time. The second constraint is hardware. The GPU path is documented for CUDA 12 on Linux x86_64 with a driver that supports it. Nothing in the supplied material describes a non-NVIDIA accelerator path, so a team running on another vendor's silicon should treat this as out of scope rather than as an untested possibility. The third is scope of the CPU path: text_cpu is described as a smoke test for verifying your environment, not as the way to run a large curation job.
What the project does not tell you in the README
The README is a routing document. It sends you to the documentation site for the Slurm deployment guide, the audio guide, the inference server page and the per-modality get-started pages, and it points at tutorials/ for the synthetic Nemotron-CC stage. The benchmark section is truncated at the point where it would state the numbers, so the supplied material gives no figures for throughput or speedup and this review will not invent any. The same applies to memory behaviour at scale, to how the pipeline behaves when a stage fails mid-run, and to checkpointing. Those are the questions to answer from the documentation and from a pilot on your own data, not from a README. What the README does establish is provenance: the Nemotron-CC curation pipeline is described as using NeMo Curator end to end, from Common Crawl extraction through language ID, exact, fuzzy and substring deduplication, ensemble quality classification and LLM-based synthetic data generation, and the SDG stage is available as an in-repo tutorial under tutorials/synthetic/nemotron_cc/. If your workload resembles that one, the recipes are a starting point you can read rather than a black box.
Where a general-purpose data framework fits better
The obvious alternative for teams that do not want a GPU dependency chain is a general-purpose dataframe or distributed compute framework, such as Spark or Dask, with the curation logic written by hand. The difference in approach is where the work happens. Spark and Dask scale across CPU cores and machines and leave deduplication and classification as algorithms you implement or import. NeMo Curator moves those stages onto the GPU and ships the operators: cuGraph for the deduplication graph work, cuML and GPU inference for classification and embedding. That is the trade. You give up the broad connector ecosystem and the operational familiarity of a CPU cluster, and in exchange you get pre-built GPU implementations of the stages that dominate wall-clock time on large corpora. For a small corpus, the CPU framework wins on setup cost alone, since NeMo Curator's GPU path requires a CUDA 12 environment, a specific override file and roughly 16 GB of GPU memory before it runs anything. Within the NVIDIA stack there is also a boundary worth noting: the inference server feature, which the README describes as an OpenAI-compatible LLM endpoint you can spin up inside your pipeline for synthetic data generation and classification, overlaps with standing up a separate serving stack. The README does not compare the two, so that decision has to be made on your own operational constraints.
Licence and the cost of keeping up
The repository is Apache-2.0, which permits commercial use and modification and requires that you preserve the licence and notices. That is the extent of what can be said here; the terms themselves govern, and this is not legal advice. On maintenance, the release cadence visible in the supplied material is roughly every two to three months, with v1.1.0 in February 2026, v1.2.0 in May, v1.3.0 in July, and a 26.04 update noted for April covering a Cosmos-Xenna 0.2.0 upgrade, a simplified Resources API and a Ray runtime upgrade. Two of those items are breaking-adjacent: a runtime upgrade and an API change. The 26.02 release moved every modality onto the Ray-based pipeline architecture, which means pipelines written before that release do not carry forward unchanged. Budget for reading release notes before each upgrade, and pin your version in the override file and container tag so an upgrade is a deliberate act. The override file is the piece most likely to need attention, since it encodes a tested combination of vLLM, RAPIDS and Numba that will shift as those projects release.
Editorial conclusion
Adopt NeMo Curator if you are preparing text, image, video or audio corpora on NVIDIA GPUs and you want the same pipeline definition to run on a laptop and on a multi-node Ray cluster, because that portability is the actual product here. Do not adopt it for a one-off CSV cleanup on CPU, for a non-NVIDIA accelerator fleet, or if you cannot accept a pinned vLLM and RAPIDS dependency set. Before committing, verify three things: that your driver and CUDA 12 toolkit match the text_cuda12 path, that the text_cuda12-overrides.txt file resolves cleanly in your environment, and that the container image covers any codec-dependent video or audio stage you plan to run.
Community notes