Cosmos-Transfer1 and the mechanics of world-to-world video generation

The perceptual gap this model is built to close

Cosmos-Transfer1 is one of three branches of NVIDIA's Cosmos World Foundation Models, sitting beside cosmos-predict and cosmos-reason. Its job is narrow: given a signal that already describes a scene, render the world that signal implies. The listed conditioning signals are segmentation video, depth video, edge video, blur video, LiDAR video and HDMap video. Each of these comes from a pipeline most robotics and simulation teams already run, and each describes structure rather than appearance. Depth says where surfaces sit, edges say where objects end, HDMap says how the road bends.

That framing matters for who the model serves. A team with a LiDAR track and no camera footage can produce video anyway, and a team with recorded RGB can use the last generated frame as a prompt for the next segment. The output is aimed at building training scenes for embodied systems, not at producing polished video for audiences, which is why the conditioning interface is so technical and the visual polish is not the headline.

ControlNet conditioning for a single signal

The simpler mode of operation attaches one ControlNet to the base model and feeds it a single modality. The project describes these checkpoints by naming the modality, so Cosmos-Transfer1-7B Depth is the depth ControlNet model and the edge variant works the same way. Alongside the control signal, the generation call takes a text prompt and, optionally, an input RGB video frame.

That optional RGB frame is the interesting mechanism. When the last generated frame is fed back in as the frame prompt, the system runs autoregressively, extending a clip one segment at a time while keeping visual continuity. The same input video frame could equally be a real recorded frame, which means a simulation can be started from a genuine capture and then continued in generated space.

The repository ships a separate inference guide for each modality, covering depth, segmentation, edge and vis variants, and the general 7B guide includes an edge example. A distilled edge checkpoint also exists, and the announcement for it is dated 2025/08.

The spatiotemporal control map is why MultiControlNet is the preferred mode

The more expressive setup runs on MultiControlNet, where any combination of segmentation, depth, edge and blur video can be combined, with LiDAR and HDMap joining in the AV sample. The project calls this the preferred mode of Cosmos Transfer. The distinguishing ingredient is the spatiotemporal control map, which controls the strength of each modality across space and time rather than applying a single global weight.

Spatiotemporal control matters because control signals fail in different places. A depth map is informative in the near field and noisy in fog or at the horizon. An edge map is dense in a cluttered street and almost empty on an open highway. A map that can vary weight by region and by frame lets a user trust the depth signal where depth is reliable and lean on the edge signal at a distance, and lets that weighting shift as the camera moves through a scene.

The cost of that flexibility is complexity, since a multi-control run takes several aligned videos, a control map, a text prompt and possibly a frame prompt, and getting the inputs out of alignment produces a plausible-looking but wrong video.

Distilled edge weights trade the 36 step schedule for one

The August 2025 release notes describe Cosmos-Transfer1-7B Edge Distilled, a checkpoint that generates video in a single diffusion step instead of 36. That is the headline latency number in this repository and it applies to one modality of one size. The standard schedule stays at 36 steps for the non-distilled path, so anyone comparing a distilled run against a baseline run should be comparing a different amount of computation, not just a different build.

The project publishes the distillation recipe and the training code rather than only the weights, and a separate guide covers distilling your own models. A distilled edge file sits in the Hugging Face model repository, and the same period added Cosmos AV Single2MultiView, which turns a single video into dynamic multi-view clips, with its own build or PyTorch post-training path.

Practically, the distilled checkpoint is the one to reach for when an inference loop has to run repeatedly, for example generating many segments, while the full schedule remains reasonable for offline asset creation.

The environment build carries a CUDA compilation tax

Installation is delegated to INSTALL.md, and the packaging metadata sets the floor at Python 3.10 with an Apache-2.0 license and version 1.0.0. The package name and Python requirement are declared in pyproject.toml:

[project]
name = "cosmos-transfer1"
version = "1.0.0"
requires-python = ">=3.10"
license = {text = "Apache-2.0"}

The dependency list is heavy and pinned. It includes megatron_core 0.10.0, transformers 4.49.0, sam2 1.1.0, decord 0.6.0 and an apex 0.9.10dev pin, alongside decord-style video decoding and hydra-core 1.3.2 for configuration.

sam2==1.1.0
decord==0.6.0
transformers==4.49.0

The pyproject note warns that the build takes time because transformer_engine does a lot of CUDA compilation, and that after the build an `import cosmos_transfer1` in Python should work. The Dockerfile avoids that local build by starting from a container and installing a prebuilt apex wheel:

FROM nvcr.io/nvidia/tritonserver:25.04-vllm-python-py3

RUN pip install https://github.com/nvidia-cosmos/cosmos-dependencies/releases/download/v1.1.0/apex-0.1+cu128.torch271-cp312-cp312-linux_x86_64.whl

That path needs an NVIDIA container runtime and a matching CUDA stack, which is the practical reason most first attempts happen on Linux with a discrete GPU rather than on a laptop.

Post-training and from-scratch training ship as scripts, not as a framework

Two training routes are offered. Post-training scripts let Physical AI builders adapt pre-trained Cosmos-Transfer1 models to their own application, and the April 2025 news entry marks that as released. Pre-training scripts let a team train a Cosmos-Transfer1 model from scratch. Both sit under examples/ alongside per-modality inference guides, and the AV sample has its own training path for building on it.

The top-level layout makes the shape of the project plain. There is a `cosmos_transfer1/` package, a `scripts/` directory, a `server/` directory, a `checkpoints/` directory, a `Dockerfile`, and a `cosmos-transfer1.yaml` configuration file at the root that the container copies in. The `examples/` folder holds thirteen markdown guides covering inference for each modality, the 4K upscaler, distillation, region-wise prompting, AV single to multi-view, and training for both the general and AV cases.

A separate guide covers processing control input data for training, which is the step that turns raw sensor output into the aligned videos and control maps the model expects. There is no claim here about cluster scheduling, checkpoint conversion utilities or evaluation harnesses, so a team planning a large training run should check the example guides directly for what is and is not scripted.

The 4K upscaler and the AV sample answer narrower questions

Two components extend the main transfer path. The 4KUpscaler takes a 720p video and produces a 4K video, with its own inference guide listed alongside the 7B and Sample-AV entries, each marked as supporting multi-GPU. It upscales existing frames rather than generating new worlds, which makes it the right tool when the simulated scene is already correct and only the output resolution is short.

The Sample-AV model targets autonomous driving material and supports LiDAR and HDMap inputs inside the multimodal path, plus the Single2MultiView behaviour that derives extra camera views from one video. For a team working on driving data, that combination is the distinctive offering: a single recorded camera video can become a multi-view training set, and LiDAR plus HDMap can drive the same world generation used in other domains.

The trade-off is weight and hardware. These are 7B class checkpoints with a multi-GPU path, so the practical ceiling on throughput is the accelerator count available, not the model code. A single-GPU team will find the distilled edge path far more attainable.

Development has already moved to Cosmos 3

The repository carries a notice that Cosmos 3 is NVIDIA's next-generation foundation model platform for Physical AI, with stronger transfer across domains, sensors, environments and embodiments, and a single model that can reason, predict future world states, transfer across domains and modalities, and generate actions and policies for embodied agents. The notice states that this repository is no longer under active development and will receive only limited maintenance updates, and it directs users to the Cosmos 3 home for future releases, documentation and community support. It also records a migration path to Cosmos-Transfer 2.5 as a general availability release dated 2025-10-29, with a Cosmos at CES 2026 announcement dated 2026-01-06 covering Transfer 2.5 and more.

The last push to this repository was on 2026-10-15, and it is not archived. The code is licensed under Apache-2.0, with a LICENSE file and an ATTRIBUTIONS.md at the root, and the project ships a CONTRIBUTING.md and a pre-commit configuration, so it is usable in commercial settings under that license subject to its terms, which every adopting team should read for itself.

For a new project in late 2026, the practical reading is that this is a well-documented, mature code base in maintenance mode, and that the model family you ultimately want to build on lives one repository over.

Editorial conclusion

Cosmos-Transfer1 fits teams that already own sensor data and want photorealistic video conditioned on it, and it stops being the right choice once a project needs the newest transfer fidelity, since the project README states development has moved to Cosmos 3. Anyone adopting it should first check GPU capacity, confirm the 36-step versus 1-step inference path matches their latency budget, and read the Apache-2.0 terms for their own use case.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. nvidia-cosmos/cosmos-transfer1 on GitHub
3. Project website
4. README
5. Releases