Latte: a video diffusion model that replaces the U-Net with Transformers

What the model actually does

The paper proposes a latent diffusion transformer, named Latte, for video generation, and the README's own summary of the method is precise enough to be useful.

Latte first extracts spatio-temporal tokens from input videos, and then adopts a series of Transformer blocks to model the video distribution in the latent space. So the diffusion process still happens in a compressed latent space, and the change is the denoiser: instead of a convolutional backbone that works on a grid, a stack of attention blocks that work on a sequence of tokens.

The reason to do that is stated in the next sentence. In order to model a substantial number of tokens extracted from videos, four efficient variants are introduced, and they come from the perspective of decomposing the spatial and temporal dimensions of input videos. A video has far more tokens than an image, so the decomposition is the part that makes the approach work at all, and the four variants are the experimental space in which that decomposition was explored.

The rest of the contribution is the practice rather than the architecture. The authors say they determine the best practices of Latte through rigorous experimental analysis, and they list what that analysis covered: video clip patch embedding, model variants, timestep-class information injection, temporal positional embedding and learning strategies. That is a list of the places where a video transformer has to make a decision that an image transformer does not, and each of them is a design choice somebody would otherwise have to guess.

The evaluation is on four standard video generation datasets, FaceForensics, SkyTimelapse, UCF101 and Taichi-HD, and the claim is state-of-the-art performance across them. The authors also extend Latte to the text-to-video generation task, where they report results comparable to recent T2V models, and they frame the contribution as insight for future research on incorporating Transformers into diffusion models for video.

The repository is a research codebase, and it says so

The repository contains the PyTorch model definitions, the pre-trained weights and the training, sampling and evaluation code for the paper, and the layout tells you what kind of project it is.

There is a `models/` directory with the model definition, a `diffusion/` directory, a `train.py` that uses PyTorch distributed data parallel, and a `train_pl.py`. There is a `train_with_img.py` and a `train_with_img_pl.py`, `configs/`, `datasets/`, `utils.py`, and a `train_scripts/` directory. There is a `sample/` directory with the sampling entry points, a `tools/` directory, an `environment.yml` for the environment, a `slurm_scripts/` directory for cluster training, `docs/` for the tutorials, and `visuals/` for the figures.

The pre-trained models are the other half of the deliverable. They are trained on FaceForensics, SkyTimelapse, Taichi-HD and UCF101 at 256 by 256, and there is a text-to-video checkpoint at 512 by 512, and they are published on Hugging Face under the `maxin-cn` account.

The timeline in the news section explains the shape of the project. In January 2024 an updated LatteT2V model initialized with PixArt-alpha was released as Latte-0. In May 2024 Latte-1 was released, supporting both text-to-video and text-to-image, with the instruction to run `bash sample/t2v.sh` and `bash sample/t2i.sh`. In June 2024 support was added in VideoSys, described as an infrastructure for video generation. And in March 2025 the work was accepted by Transactions on Machine Learning Research for TMLR 2025.

So the publication is recent and the model line is old enough to have a predecessor that was initialised from a different project, which is a good sign about how the work accumulates rather than restarting.

The practical route is diffusers, not the training script

For most people who want to run Latte, the repository is not the interface. In July 2024 Latte-1 was integrated into diffusers, and the README shows the resulting pipeline.

from diffusers import LattePipeline
from diffusers.models import AutoencoderKLTemporalDecoder
from torchvision.utils import save_image
import torch
import imageio

torch.manual_seed(0)

device = "cuda" if torch.cuda.is_available() else "cpu"
video_length = 16 # 1 (text-to-image) or 16 (text-to-video)
pipe = LattePipeline.from_pretrained("maxin-cn/Latte-1", torch_dtype=torch.float16).to(device)

The pipeline is constructed from the Hugging Face identifier with a half precision dtype and moved to the device, and the length parameter distinguishes the two tasks, 1 for text-to-image and 16 for text-to-video. The optional step is the temporal decoder of the VAE, loaded as a subfolder of the same repository and assigned to the pipeline, which is what the commented explanation refers to when it says it is using the temporal decoder of the VAE.

The requirement is stated in a comment on the import: the version of diffusers has to be updated to at least 0.30.0.

The memory numbers are the part to plan around. The README states that inference with 4 or 8 bit quantization can reduce GPU memory from 17 GB to 9 GB, and points at a tutorial in the `docs/` directory for the details. For a video model at 512 by 512 and 16 frames, that is the difference between a large workstation and a single well chosen card, so it is not a footnote.

The integration credits the people who did it, and there are four of them, so the work was done in the open rather than landed as a single pull request.

One sample script per dataset, and a cluster story

The `sample/` directory is organised by dataset, and that organisation says what the model was trained on and how the authors ran it.

There is a shell script per dataset, `ffs.sh` for FaceForensics, `sky.sh` for SkyTimelapse, `taichi.sh` for Taichi-HD and `ucf101.sh` for UCF101, and each of those has a `_ddp` variant, `ffs_ddp.sh`, `sky_ddp.sh`, `taichi_ddp.sh` and `ucf101_ddp.sh`, for sampling with distributed data parallel. There are the two task scripts, `t2i.sh` and `t2v.sh`, and the Python entry points `sample.py`, `sample_ddp.py`, `sample_t2x.py` and `pipeline_latte.py`.

The duplication of every script into a single and a distributed variant is a small design decision with a large practical meaning. Sampling a video model at 512 by 512 over 16 frames is heavy enough that one GPU is a normal request and several is a normal plan, so the authors made the distributed path a first-class citizen rather than an afterthought, and the naming makes it obvious which script to reach for.

The `slurm_scripts/` directory points the other way, towards a cluster. The training script uses PyTorch DDP, the environment is described by an `environment.yml`, and the datasets live in a `datasets/` directory with the configurations in `configs/`. That is the standard shape of a video generation research codebase: the interesting artefact is the model, and everything else exists to get data to it and results out of it.

For a reader who wants to understand the method rather than run it, that arrangement is a feature, because each piece is in its own directory and the sizes are legible.

The lineage and the ecosystem around the checkpoint

Two details in the news section tell you how this kind of work ends up in a general purpose library, and they are worth following.

The first is the PixArt-alpha initialization. The January 2024 release of the updated LatteT2V model was initialized with PixArt-alpha, and the repository still carries a separate training script, `train_pl.py`, alongside the standard `train.py`, plus the image-assisted variants `train_with_img.py` and `train_with_img_pl.py`. Initialising a transformer from a pretrained image model is the standard way to start a video model without training from scratch, and keeping the variant in the tree means the experiment is reproducible rather than a footnote in a paper.

The second is the diffusers integration, which happened in July 2024 and was credited to four contributors. Once a research model lands in a general purpose library with a pipeline class, the number of people who can run it stops being the number of people who can install the right versions of the training stack, and the README's own example is written against the library rather than against the repository, which is the correct choice for the majority of readers.

The other integration, in VideoSys in June 2024, is of the same kind and points at the same conclusion. A model that lives in three places, the repository, the library and a video generation framework, is a model that other people have found useful, and the Discord channel created for the project is where the discussion happens.

The weights are hosted on Hugging Face in two places, the Latte-1 repository with the transformer and the VAE temporal decoder as a subfolder, and the Latte-0 repository for the earlier checkpoint. The project page, the arXiv paper and a Hugging Face Space are all linked from the README, so the paper, the code and a running demo are one click apart.

Where this repository is the wrong tool

The limits are the ones a paper implementation has, and the README is quietly clear about them.

The first is that there is no release. The repository has no GitHub releases and the default branch is `main`. A project that wants to pin a version of the training code has nothing to pin, and a project that wants to run the model is pointed at the diffusers integration instead. So the repository is best understood as the research artefact behind a model that is consumed elsewhere.

The second is the training surface. Training a video diffusion transformer is a cluster activity: distributed data parallel, a Slurm directory, a dataset directory and a configuration set. The scripts are there and they are the honest version of the work, but the cost of a training run is a measure of how much this is a research codebase and not a product.

The third is the resolution and the checkpoint sizes. The pre-trained models are 256 by 256 with a text-to-video checkpoint at 512 by 512, and the sample scripts are per dataset, which means the released weights are tied to the datasets they were trained on. A team that wants a different resolution or a different subject has to train, not to sample.

The alternative, for the large majority of users, is the diffusers pipeline with a version constraint, and the practical difference in approach is that the library gives you a class and a checkpoint identifier while the repository gives you a research codebase with a training script, a dataset layout and a set of sample scripts per dataset. The first is what a team ships, the second is what a lab reads. A team that wants to study the method will spend its time in the training code, and a team that wants video will spend it in the pipeline, and both are legitimate reasons to look at this repository for different reasons.

Editorial conclusion

Latte fits a researcher who wants to study or extend a latent diffusion transformer for video rather than only consume one, and who is prepared to work from a research repository with a training script, per dataset sample scripts and a Slurm directory. It is the wrong choice for a production application, because there are no releases, the main branch is the only interface, and the practical path runs through the diffusers pipeline rather than through this code. Before building on it, read the diffusers tutorial and check the memory numbers, since the README states that 4 or 8 bit quantization takes the GPU requirement from 17 GB down to 9 GB, and decide whether the training code in `train.py` with DDP is something you need to run or only something you need to read.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Vchitect/Latte on GitHub