OmniWeaving: a unified video generator that reasons over interleaved inputs

One prompt mixing text, stills and clips

OmniWeaving targets a specific gap rather than general video synthesis. Most systems take either a text caption or a set of reference images, and force the user to pick. OmniWeaving, from the Tencent Hunyuan team, is described in the repository as an official implementation of a model for "Unified Video Generation with Free-form Composition and Reasoning", and the abstract says it learns to "temporally bind interleaved text, multi-image, and video inputs" while acting as an agent to infer complex user intentions. The practical target is a request that mixes a written instruction, several stills and an existing clip in one sequence, and expects one composed result. That makes it a research and prototyping tool for engineers building controllable generation pipelines, and it inherits the weight and serving cost that implies. The project states that it is developed by the HunyuanVideo team and built on the latest HunyuanVideo-1.5 as the backbone, so it is an extension of an existing video family rather than a standalone system.

MLLM, VAE and MMDiT share a single generation path

The documented architecture is an integrated MLLM plus MMDiT plus VAE framework, and the split of labour matters if you plan to swap parts of it. The MLLM works as the semantic parser: it takes the interleaved text, image and video inputs, maps them into a high-level semantic space, and forwards its hidden states through an MLP connector. The VAE is the visual tokenizer, compressing visual inputs into low-level latents. The MMDiT then consumes both the semantic conditions and latent noise to produce the output frames. In practice this means an input clip is not fed to the diffusion transformer directly as pixels. It is encoded twice, once into a language-conditioned semantic representation and once into a compressed latent, and the transformer works on the latent side while drawing alignment from the semantic side. The project page describes the result as semantically aligned, high-fidelity video, and the architecture section explains the goal of bridging the gap between abstract user intent and pixel-level generation.

Thinking mode turns the parser into a reasoner

The first of two stated improvements addresses a failure mode that the documentation names plainly: direct encoding of interleaved visual and text input often yields semantic ambiguity, because correlations inside the sequence are weak and the creation intent is unclear. OmniWeaving responds by activating the thinking mode of the MLLM, so the model generates intermediate reasoning steps and, from those, deduces an enhanced prompt on its own. The hidden states of that enhanced prompt are then forwarded alongside the original MLLM features to condition the MMDiT. The practical reading is that one forward pass produces two parallel conditioning streams, the raw parse and the reasoned parse, and the diffusion branch sees both. For an engineer this is the part to watch when debugging: if a composed request comes out wrong, the fault may sit in the reasoning stage rather than in the denoiser, and the project does not publish a separate knob for isolating that stage.

DeepStacking injects multi-level features into three layers

The second improvement targets scenes with several subjects, which the documentation says depend on low-level and high-level semantic representations at once. OmniWeaving takes the DeepStacking mechanism from Qwen3-VL and applies it: hidden states are extracted from a broader range of intermediate MLLM layers to capture a spectrum from fine-grained detail to high-level abstraction. A separate MLP connector projects those multi-level features into the MMDiT embedding space, and the projected features are added directly to the corresponding hidden states inside the first three layers of the MMDiT conditioning branch. The consequence for anyone adapting the model is that conditioning is not a single vector but a per-layer contribution spread over the opening of the conditioning branch. Replacing the connector or changing which MLLM layers are read means retraining or at least fine-tuning that branch, and the repository does not describe a way to disable DeepStacking at inference time.

Installing the pinned dependency set and finding the checkpoint

The project page points readers to the official site, the paper on Arxiv and the model weights on Hugging Face, and the repository root carries a download-checkpoint.md file plus a requirements.txt. The dependency set is fully pinned for most entries, which helps reproducibility but also makes the environment brittle against a newer stack. A representative slice of the file looks like this:

torch>=2.6.0
diffusers==0.32.2
transformers[accelerate,tiktoken]==4.57.1
omegaconf>=2.3.0

The remaining entries pin tooling such as einops, loguru, numpy, imageio, omegaconf and huggingface-hub, and add qwen-vl-utils, decord and modelscope. Two of the pins deserve attention. transformers is held at 4.57.1 while torch is a floor of 2.6.0 rather than an exact version, and torch 2.6.0 is the first release with a binary build for every GPU architecture the project is likely to be used on. Neither the page nor the requirements file states a GPU, VRAM or platform requirement, so the hardware budget has to be established separately before anything is downloaded.

generate.py, train.py and process_data in the root layout

The repository layout is flat and easy to read: generate.py, train.py, a requirements.txt, a download-checkpoint.md, a Notice.txt, a License.txt, and four directories named hyvideo, intelligentVBench, process_data and assets. The open-source plan lists all four OmniWeaving components as released, namely inference code, model checkpoints, training data construction code and training example code. That splits into three practical workflows. Inference runs from generate.py and the hyvideo package. Data preparation lives in process_data, which matches the data construction code entry. Training has its own entry point in train.py, so the repository is a training-capable release rather than an inference-only one, and peft is pinned in the requirements, which suggests parameter efficient fine tuning is part of the intended workflow. The release dates in the news section are specific: the paper appeared on Arxiv on 2026-03-26, the project page on 2026-03-25, and both the code and the model weights were released on 2026-04-03.

A custom licence and a last push on 2026-04-11

Two practical constraints come from the repository's own records rather than the documentation. First, the licence is not a standard one. GitHub classifies it as Other, a custom licence it cannot place in a known category, and the file in the repository is named License.txt with a companion Notice.txt. Anyone planning commercial use needs to read both files directly, since the classification alone tells you nothing about redistribution of weights, acceptable use or attribution duties. Second, the maintenance picture is thin. The repository is not archived, but the last push was on 2026-04-11 and the project has no GitHub releases at all, so there is no tagged version history to anchor an upgrade path. The documentation also leaves some questions open: it does not document a rollback procedure, a supported CUDA version, or a minimum GPU memory, and it does not publish the evaluation numbers behind the claim of state of the art performance among open source unified models.

Editorial conclusion

OmniWeaving fits research teams and pipeline engineers who already work with PyTorch video stacks and can host large checkpoints, and it is the wrong choice for a lightweight product demo or for anyone who needs a single clear licence grant. Before adopting it, verify what your own hardware can hold, read License.txt and Notice.txt rather than assuming a standard open source licence, and check whether the committed code covers the checkpoint you intend to run.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. Tencent-Hunyuan/OmniWeaving on GitHub