# Tora: driving video generation with explicit trajectories

> Tora is a CVPR'25 research framework that lets a user hand a video model the path an object should follow. It ships as a CogVideoX-5B codebase for academic use and a separate diffusers build with a much smaller memory footprint.

**alibaba/Tora** — [CVPR'25]Tora: Trajectory-oriented Diffusion Transformer for Video Generation

- Repository: https://github.com/alibaba/Tora
- Website: https://ali-videoai.github.io/tora_video
- Stars: 1,243 · Forks: 61
- Language: Python
- License: Apache-2.0
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/alibaba-tora

## Tora puts a drawn path where a text prompt used to be

Text-to-video systems tell a model what a scene should contain. Tora, a CVPR'25 paper from the Alibaba team behind the project page, adds a second and more literal control channel: the path a specific object should travel across the frame. The project describes itself as the first trajectory-oriented diffusion transformer to combine textual, visual and trajectory conditions at the same time.

That combination matters for a specific set of problems. If a prompt can only say that a bird flies, the model has to decide the geometry of that flight on its own, and two runs of the same prompt will disagree about where the bird starts and how fast it moves. Supplying a trajectory turns an open question into a constraint. The README frames the goal as precise control of the dynamics of generated content across different durations, aspect ratios and resolutions, and the paper is published on arXiv as 2407.21705.

The audience is therefore narrow and research shaped. This is not a general purpose video editor or a consumer product. The scope is a controlled generation experiment where the motion of an object is part of the input rather than a lucky property of the output.

## Three pieces: Trajectory Extractor, Spatial-Temporal DiT, Motion-guidance Fuser

The architecture in the paper is described as three components. The Trajectory Extractor, abbreviated TE, encodes arbitrary trajectories into hierarchical spacetime motion patches using a 3D video compression network. The Spatial-Temporal DiT is the diffusion transformer backbone that does the generation. The Motion-guidance Fuser, or MGF, is the component that injects those motion patches into the DiT blocks so the produced frames actually follow the requested paths.

The data flow is therefore: a user supplies a trajectory, the TE compresses it into motion patches that live in spacetime rather than in a single frame, the MGF folds those patches into the transformer blocks, and the DiT produces video whose content and motion both respect the prompt and the path. The paper calls the design compatible with the scalability of DiT, which is the usual argument for a transformer backbone: more compute and data scale into better samples rather than requiring a new architecture.

One consequence is worth spelling out. Because the trajectory is compressed before it reaches the transformer, the condition is not a literal list of pixel coordinates at every frame. It is a learned representation of motion over time. That is what lets one trajectory description work across different durations and resolutions, and it is also why the model can only follow paths that resemble the kinds of motion seen during training.

## Installation means Python 3.10 to 3.12 and a PyTorch you install separately

The installation section of the README is short, and the top level page deliberately leaves the heavy lifting to PyTorch's own instructions. Clone the repository first, then install PyTorch and torchvision on your own.

```bash
# Clone this repository.
git clone https://github.com/alibaba/Tora.git
cd Tora

# Install Pytorch (we use Pytorch 2.4.0) and torchvision following the official instructions: https://pytorch.org/get-started/
```

The version constraint is stated plainly: Python must be between 3.10 and 3.12, inclusive of both ends. PyTorch 2.4.0 is the version the project says it uses, and the install is left open on purpose, which means you are expected to follow the official selection flow rather than pip install a pinned torch build from this page.

For a first real use, the practical next stop is the diffusers path, not the base one. The README points to a diffusers-version directory with its own README, and the 2024/12/09 update note says that path is where the memory reduction lives.

## Two code paths with very different memory costs

This repository contains two separate implementations and confusing them wastes an afternoon. The first, announced on 2024/10/15, is a CogVideoX version of Tora built on the CogVideoX-5B model, with inference code and model weights released together. The second, announced on 2024/12/09, is a diffusers version, and it comes with its own weights.

The difference that matters day to day is VRAM. The update note for the diffusers release states that inference VRAM requirements are reduced to around 5 GiB. No comparable figure is given for the CogVideoX-5B build in the top level README, so if you are on a single consumer card the diffusers build is the one to evaluate. If you are reproducing paper numbers, the CogVideoX-5B path is likely the one that matches the reported setup.

Weights are published in two families on each of HuggingFace and ModelScope. There are SAT format T2V and I2V weights, and separate diffusers format T2V weights. The image to video release of 2025/01/06 added inference code and weights for that mode. A ModelScope studio demo and an English demo are also linked, so you can sanity check outputs before committing to a local install, and third party ComfyUI support was announced on 2024/10/21.

## The open weights carry an academic research restriction

The 2024/10/15 announcement is unusually direct about this. The released CogVideoX version of Tora is meant for academic research purposes only, and the note explains why: due to commercial plans, the complete version of Tora will not be open sourced at this time. The TODO list later marks release of the complete version of Tora as done, but the licence and usage terms attached to the checkpoints are a separate matter from the status of the checklist.

The repository itself is licensed Apache-2.0, and the tree also carries a CogVideoX_LICENSE file at the top level, which signals that the weights build on a model with its own separate terms. Treat those as distinct: the code licence covers the code, the CogVideoX licence file covers the base model lineage, and the research-only statement covers how the released Tora checkpoint may be used. This is not legal advice, and nobody should read the Apache-2.0 file as permission to ship the released weights in a commercial service.

If your plan is a paid product that generates video on user trajectories, this checkpoint is the wrong starting point, and the wording in the README is clear enough that you do not need to guess.

## SageAttention2 and model compilation speed up each step by about 52 percent

The 2024/12/13 update is the most concrete performance statement in the repository and it is worth reading carefully. SageAttention2 and model compilation are supported in the diffusers version, and on an A10 these approaches speed up every inference step by approximately 52 percent, except for the first step. The first step is excluded because it includes the setup cost, which is a familiar pattern for compiled inference paths.

The hardware reference matters. The figure comes from an A10, and the README does not give a number for other cards, so the 52 percent is a relative speedup rather than a claim about absolute latency or output quality. Nothing in the note claims that the acceleration changes the generated video.

This is also the only quantified performance statement the top level page makes, which tells you something about the documentation style of the project. The authors report measured numbers with the specific setup attached and leave the rest of the claims, such as motion fidelity, to the paper and the arXiv entry.

## Tora2 and Tora3 have moved past this codebase

The repository has been the home of a sequence of projects, and Tora is the first of them. The README timeline records Tora2, accepted by ACM MM25 and released on 2025/07/08, which builds on Tora with design improvements for appearance and motion customization across multiple entities. Tora3 was released on arXiv on 2026/04/10 and accepted by ACM MM26, announced on 2026/07/27, and it extends the idea to trajectory guided audio and video generation, using object trajectories to steer visual motion and sound together for better physical coherence and motion sound synchronization.

That sequencing matters for anyone starting a project in 2026. The techniques described in this README describe the first framework in the line, and the newest work in the repository is two generations ahead of it, with a separate project page. The code in this tree is the code for the original trajectory oriented DiT, not for the audio video system.

On maintenance, the repository is not archived, and the last push was on 2026-07-27, which was the day the Tora3 acceptance was announced. The project has no GitHub releases, so there is no tagged version to pin your environment to and no upgrade channel to subscribe to. The updates list, not the tag list, is the project's release history. Training code for text to video was released on 2024/11/25. Anyone planning to fine tune should read the training section of the README directly.

## Conclusion

Adopt Tora if your research question is motion control, because a trajectory condition is a different and finer control surface than a text prompt. Do not adopt it as a production video service: the released CogVideoX build is limited to academic research, the diffusers build is the only documented path to a lower memory footprint, and the repository has no releases to pin against. Before starting, verify the memory headroom on your own GPU, check which of the two code paths matches your checkpoint, and read the diffusers README yourself, because the top level page does not document per flag behaviour.

## FAQ

### How much GPU memory does Tora need for inference?

The diffusers version of Tora is documented as reducing inference VRAM requirements to around 5 GiB. No equivalent figure is given for the CogVideoX-5B version in the top level README.

### Can I use the released Tora weights commercially?

The release note says the released CogVideoX version of Tora is meant for academic research purposes only, and that the complete version of Tora was not open sourced due to commercial plans. The repository code itself carries an Apache-2.0 license, and a separate CogVideoX_LICENSE file sits at the top level.

### How do I control the motion of an object in Tora?

You supply a trajectory as a condition. A Trajectory Extractor encodes it into hierarchical spacetime motion patches with a 3D video compression network, and a Motion-guidance Fuser integrates those patches into the DiT blocks so the generated video follows the path.

### Which versions of Tora are available and what is the difference?

Tora is the original trajectory-oriented diffusion transformer, released as a CogVideoX-5B codebase and later as a diffusers version. Tora2, accepted by ACM MM25, adds appearance and motion customization for multiple entities, and Tora3, accepted by ACM MM26, guides audio and video generation together with object trajectories.

### What Python and PyTorch versions does Tora require?

The README asks for Python between 3.10 and 3.12, inclusive of both ends. It says PyTorch 2.4.0 is used, and leaves the install itself to the official PyTorch instructions.

## Sources

- [alibaba/Tora on GitHub](https://github.com/alibaba/Tora)
- [Issues](https://github.com/alibaba/Tora/issues)
- [License: Apache-2.0](https://github.com/alibaba/Tora/blob/main/LICENSE)
- [Project website](https://ali-videoai.github.io/tora_video)
- [README](https://github.com/alibaba/Tora/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/alibaba-tora