# UniVideo folds understanding, generation and editing into one video model

> UniVideo is the ICLR 2026 research release from the Kling Team at Kuaishou and the University of Waterloo that handles nine multimodal video tasks through one pipeline with a single task id per configuration. It suits researchers studying unified multimodal models rather than teams shipping a video product.

**KlingAIResearch/UniVideo** — [ICLR 2026] UniVideo: Unified Understanding, Generation, and Editing for Videos

- Repository: https://github.com/KlingAIResearch/UniVideo
- Website: https://congwei1230.github.io/UniVideo/
- Stars: 571 · Forks: 32
- Language: Python
- License: MIT
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/klingairesearch-univideo

## Nine tasks, one checkpoint, one task id

The central claim of UniVideo is unification, and the repository makes that concrete through a table of supported tasks with a consistent shape. Each row has an input type, an output, a task id and a short description, and the ids are the interface you actually program against. Image and video understanding takes an image or a video plus text and returns text, with the task id understanding. Text to image is t2i. Text to video is t2v. Image to video, which animates a still image, is i2v. Instruction based image editing is i2i_edit. Editing an image based on a reference image is i+i2i_edit. Multi subject generation, taking several reference images plus text and producing an image or a video, is multiid. Instruction based video manipulation and stylisation is v2v_edit. And in context video editing, where a reference image drives addition, deletion, swapping or stylisation of a video, is i+v2v_edit.

That last group is where the design gets interesting. Reference based editing of a video is a harder problem than prompting a video into existence, because the model has to hold a subject identity across frames while applying an operation to it, and it has to decide what happens to the parts of the frame you did not mention. By giving that case its own task id rather than folding it into a prompt convention, the model treats the reference image as a first class input. The project describes itself as flexible in its input and output configurations, supporting a wide range of multimodal tasks, and the demo directory mirrors the table exactly, with separate folders for understanding, t2i, t2v, i2v, image_edit, in_context_image_edit, in_context_video_gen, video_edit and in_context_video_edit.

## The repository layout and what each piece does

The top level tells you how the code is organised. There is an __init__.py at the root, pipeline_univideo.py which is the pipeline entry point, univideo_inference.py for running inference, mllm_encoder.py for the multimodal large language model encoder, transformer_univideo_hunyuan_video.py for the transformer itself, utils.py for shared helpers, and download_ckpt.py for fetching the checkpoint. The name of the transformer module is worth pausing on, because it says where the architecture comes from: the video transformer is built on the HunyuanVideo lineage rather than written from scratch. A configs/ directory holds configuration, scripts/ holds run scripts, train/ holds the training code, eval/ the evaluation code, and examples/ the example scripts.

Two documentation files sit at the root and they mark the two halves of the project. TRAINING.md carries the training script and instructions, and the news entry dated 2026-06-03 says the training script and instructions are now available there. EVAL.md sits alongside it for evaluation. The news list gives the sequence of the release: code released on 2026-01-07, acceptance at ICLR 2026 on 2026-01-30, and the training script on 2026-06-03. The order matters. The code came first and the training script came four months later, so the early months of the project were inference only, and the released artefact at the start was a checkpoint you could use but not reproduce.

## Setting up the environment and the pinned dependency wall

The repository provides environment.yml and requirements.txt, and the requirements file is where the practical friction sits. The dependency list is short and specific:

```bash
numpy
pillow
packaging
typing_extensions
tqdm
einops==0.8.1
diffusers==0.34.0
transformers==4.51.3
tokenizers==0.21.1
huggingface_hub==0.36.0
accelerate==1.5.1
safetensors==0.5.3
imageio
imageio-ffmpeg
opencv-python-headless
decord>=0.6.0
```

Most of these are unpinned or loosely pinned, and the pinning is concentrated on the deep learning stack: diffusers at 0.34.0, transformers at 4.51.3, tokenizers at 0.21.1, huggingface_hub at 0.36.0, accelerate at 1.5.1, safetensors at 0.5.3 and einops at 0.8.1. Decord is held at 0.6.0 or newer and is there for video decoding, while imageio and imageio-ffmpeg cover frame handling and the ffmpeg binary, and opencv-python-headless handles image operations without a display server, which is the correct choice for a container. The instruction is to install these into an environment created from environment.yml, and the combination of the two files is deliberate: the conda file pins the non Python side, the requirements file pins the Python side. If you already run a different diffusers or transformers version in the same environment, the conflict is not something you can negotiate, because the model code is written against those exact releases. In practice that means a dedicated environment for this repository rather than adding it to an existing one.

## Weights, pages and where the model lives

The checkpoint is not in the repository. download_ckpt.py is the script that fetches it, and the model is published on Hugging Face under the KlingTeam organisation, alongside a link to the arXiv paper at 2510.08377 and a project page at congwei1230.github.io/UniVideo. That hosting arrangement is a deliberate one for a research release: it is the standard place to put a multi gigabyte diffusion checkpoint, and it means the git repository stays reviewable while the weights live elsewhere.

The provenance is stated clearly and it matters for how you read the results. The work comes from the University of Waterloo and the Kling Team at Kuaishou Technology, with the first author marked as work done during an internship at the Kling Team, and two corresponding authors. This is a paper release with code, not a product, and the audience is researchers. There is also a subtle point about the organisation of the authors across the two institutions that is typical of the current industrial research pattern: the model that powers a commercial video product is released as a research artefact with a permissive MIT licence on the code, which says something about how the two are separated.

## Where this stops being the right tool

The first limit is the absence of a release process. The repository has no GitHub releases at all, the last push was on 2026-07-03, and the published artefact is a checkpoint on Hugging Face rather than a versioned release. For a research codebase that is normal and mostly harmless. For a team planning to depend on this in a product, it is not: there is no upgrade path, no compatibility statement between checkpoint versions, and no support commitment. The second limit is hardware. A diffusion transformer that handles video generation and in context video editing is not a model you run on a laptop, and the repository layout does not mention quantisation, distillation or an efficient variant, so the inference path is the full model.

The third limit is the coverage of the evaluation. The project publishes EVAL.md and a demonstration page with example inputs and outputs, and for a paper that is the expected package. But a demonstration page shows the cases the authors chose, and without an independent evaluation the quality claim for a specific task, in context video editing for instance, is not something you can rely on for your own content. If your decision depends on quality rather than on capability, treat the demos as a starting point and build your own evaluation set.

And the fourth is the overlap with existing tools. If you only need text to video, a hosted service or a well supported open model with an active community gives you a better experience than a research checkpoint, because the surrounding tooling, the prompt handling and the failure modes have all been worked out by other people.

## What the unified approach changes compared with separate models

The difference in approach is worth being concrete about, because unification is a claim with more than one meaning. A separate model per task means each task gets its own architecture, its own training data and its own failure modes, and nothing is shared, so improving editing tells you nothing about improving generation. A unified model trained across all of these configurations on one codebase with one task id per mode makes a different bet. The same weights and the same code path serve understanding, generation and editing, so the model's internal representation of a subject, an object or a scene is shared between reading a video and writing one.

That bet is what makes in context editing interesting. If the model understands a reference image strongly enough to caption it and to generate from it, then asking it to swap that subject into a video is closer to a compositional operation on a shared representation than to a new task trained from scratch. The nine task ids are the observable surface of that design, and the multiid and i+v2v_edit rows, with multiple reference images and reference based manipulation, are where the unified claim does the most work.

The honest counterpoint is that unification costs something. A single model carrying nine tasks has to spend its capacity across all of them, and per task quality in a unified model is often behind a specialist that does one thing. A team that needs excellent text to video and nothing else would be wrong to pick this over a dedicated model, and a team that needs reference based video editing, which few open options cover well, is exactly the audience this release is for. The value of the release is also as a research baseline: the training script, the configs and the evaluation code are there so that others can extend a unified video model rather than rebuild one.

## Conclusion

Adopt UniVideo if you are researching unified multimodal video models and want the training script, the evaluation guide and one checkpoint that covers nine task configurations rather than a separate model per task. Do not adopt it as a production video generation service, since there are no GitHub releases, the last push was on 2026-07-03, and the artefact is a research checkpoint hosted on Hugging Face rather than a versioned release with support. Before you commit, read TRAINING.md and EVAL.md in the repository, confirm the pinned versions in requirements.txt match the diffusers and transformers releases you already use, and expect the diffusers and transformers pins to be the constraint that decides whether the environment builds at all.

## FAQ

### Which tasks does UniVideo support?

Nine configurations are listed: understanding for image and video analysis with a text output, t2i for text to image, t2v for text to video, i2v for animating a still image, i2i_edit for instruction based image editing, i+i2i_edit for editing an image against a reference image, multiid for multi subject generation, v2v_edit for instruction based video editing and stylisation, and i+v2v_edit for reference based video manipulation covering addition, deletion, swapping and stylisation.

### How do I set up the UniVideo environment and get the weights?

Create an environment from environment.yml and install the dependencies in requirements.txt, which pins diffusers at 0.34.0, transformers at 4.51.3, tokenizers at 0.21.1, accelerate at 1.5.1 and einops at 0.8.1 among others. The checkpoint is not stored in the repository, so you fetch it with download_ckpt.py or from the KlingTeam organisation on Hugging Face.

### Can I train UniVideo myself?

Yes. The training script and its instructions are in TRAINING.md, with the training code under the train/ directory and configurations under configs/. The news entry dated 2026-06-03 marks the point at which the training script and instructions became available, after the code release on 2026-01-07 and the ICLR 2026 acceptance on 2026-01-30.

### Is UniVideo available as a released version and is the repository maintained?

The repository has no GitHub releases, and the last push was on 2026-07-03. The model itself is published on Hugging Face under the KlingTeam name, and the project page is at congwei1230.github.io/UniVideo with the paper on arXiv at 2510.08377. The code is MIT licensed, with the terms in LICENSE.txt.

### What does the in context video editing task in UniVideo do?

The i+v2v_edit task takes an image, a video and text, and performs reference based manipulation of the video, with the documentation describing addition, deletion, swapping and stylisation. It is the mode that treats a reference image as a first class input rather than relying on text description alone.

## Sources

- [Issues](https://github.com/KlingAIResearch/UniVideo/issues)
- [KlingAIResearch/UniVideo on GitHub](https://github.com/KlingAIResearch/UniVideo)
- [License: MIT](https://github.com/KlingAIResearch/UniVideo/blob/main/LICENSE)
- [Project website](https://congwei1230.github.io/UniVideo/)
- [README](https://github.com/KlingAIResearch/UniVideo/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/klingairesearch-univideo