Ditto builds a million video editing examples, and Editto trains on them

The dataset and the model are one release, not two

The paper title says it plainly: Ditto scales instruction-based video editing with a high-quality synthetic dataset. The TL;DR states the structure of the contribution, and it is two halves of one thing. A scalable pipeline called Ditto generates high-quality video editing data, and that data is used to train a model called Editto, described as a new state-of-the-art instruction-based video editing model. The repository description marks the work as a CVPR 2026 Highlight, and the author list is long, with thirteen names including Qingyan Bai, Qiuyu Wang, Hao Ouyang and Qifeng Chen in the last position. The published artefacts follow the same pairing, with model weights, the dataset, a Hugging Face demo, a project page and an arXiv paper all linked from the top of the README. For an adopter, the practical question is which half you want, because the data pipeline and the trained checkpoint are separable.

An agent writes the instructions and filters the results

The mechanism that makes the dataset scale is described in the summary, and the interesting part is the last sentence rather than the first. The pipeline fuses the creative diversity of a leading image editor with an in-context video generator, so the editing operation comes from one model and the video synthesis from another. But the part that makes it a dataset rather than a pile of clips is that the entire pipeline is driven by an intelligent agent that crafts diverse instructions and rigorously filters the output, ensuring quality control at scale. That is a differentiator against the previous generation of editing datasets, which were mostly scraped or manually curated, and it explains the cost. The dataset is named Ditto-1M and contains one million high-fidelity video editing examples, and the README says more than 12,000 GPU-days were invested in building it. The number is a fair proxy for what reproducing this yourself would cost.

Distillation and a temporal enhancer are the cost-quality answer

The summary names the technical compromise directly. To make the generation process viable, the framework resolves the prohibitive cost-quality trade-off by employing an efficient, distilled model architecture augmented by a temporal enhancer, which simultaneously reduces computational overhead and improves temporal coherence. That is a lot of engineering compressed into one sentence, and it is the part a reader implementing this needs to understand. Distillation is what makes generation affordable at a million-example scale, and the temporal enhancer is what keeps the generated video coherent across frames rather than flickering, which is the classic failure of frame-independent diffusion. The model is then trained on the resulting data with a curriculum learning strategy, and the results are claimed to demonstrate superior instruction-following ability and to establish a new state of the art in instruction-based video editing. The repository also has a denoising_enhancing/ directory and a README changelog entry from 2025-10-27 adding codes for it, so the enhancement step is shipped as a working component rather than a description.

Running inference needs a 14B video model and a lot of memory

The usage instructions are honest about the hardware, and the ComfyUI section is where the numbers appear. A 832 by 480 video of 73 frames needs 11 GB of GPU memory and runs in about four minutes on an A6000, and the README adds a warning that because the models used are quantized and distilled, there may be some quality degradation. The environment setup is a conda environment on Python 3.10 with the package installed in editable mode, and the models come from a base model downloaded from Hugging Face alongside the project's own weights:

conda create -n ditto python=3.10
conda activate ditto
pip install -e .

The model weights are fetched with the hf command:

hf download Wan-AI/Wan2.1-VACE-14B --local-dir models/Wan-AI/

# Download Ditto models
hf download QingyanBai/Ditto_models --include="models/*" --local-dir ./

The one-command path downloads the weights and test data automatically and runs the tests in parallel, and the general script is a thin wrapper over a Python module that takes an input video, an output path, a prompt, a LoRA path, a frame count and a device index. The default frame count in the example is 73, which is a short clip, and that number is the real constraint on the whole thing.

ComfyUI gets a custom node set and a workflow file

The second integration path is ComfyUI, and the README is upfront that it runs faster with lower computational requirements than the DiffSynth route. It recommends installing the ComfyUI-Manager for custom node handling, and it names the three custom nodes required, the WanVideoWrapper and KJNodes from the kijai organisation and the VideoHelperSuite from Kosinkadink. The weights for this path come from a different repository on Hugging Face, and the file names are given individually, including the fp8 text to video diffusion model, a CausVid LoRA for inference acceleration, a bf16 VAE and the umt5 text encoder. The Ditto models themselves go into the diffusion_models directory and are loaded with the VACE Module Select node. A workflow file named ditto_comfyui_workflow.json ships in the repository and is the starting point, with reference prompts included in the workflow note. The examples/ directory lists integrations with other generation stacks, including ControlNet, ExVideo, HunyuanVideo, InfiniteYou, IP-Adapter, TeaCache, CogVideoX and Flux, which is a sign of how much of the surrounding ecosystem this work touches.

A research layout with a DiffSynth fork inside it

The repository structure is worth reading because it explains the relationship to the surrounding ecosystem. The top level holds a setup.py that installs a package called diffsynth, at version 1.1.8, with the description Enjoy the magic of Diffusion models, which identifies this as a fork or vendored copy of the DiffSynth-Studio framework rather than a project from scratch. The requirements.txt gives the dependency floor: torch at 2.0.0 or newer, torchvision, transformers, imageio with the ffmpeg extra, safetensors, einops, sentencepiece, protobuf, modelscope, ftfy, pynvml, pandas, accelerate and peft, and the package declares Python 3.6 or newer while the README's own setup uses 3.10. There is a diffsynth/ directory, a models/ directory, an inference/ directory containing the infer_ditto.py script, an examples/ directory with the many named integrations, a train.sh and a denoising_enhancing/ directory. The README also mentions uploaded csvs that can be used directly for model training with DiffSynth-Studio, and a metadata json for a sim2real setting, which is the kind of detail that tells you the dataset is intended for reuse by people outside the paper.

The licence is the one thing you have to go and read

The repository records its licence as a custom one that GitHub cannot classify, with the terms in a LICENSE file at the root, and the README says nothing about licensing. For a research release that ships a dataset of a million generated videos, a set of trained weights and a model built on a 14 billion parameter base checkpoint, that is the detail to settle before anything else, because a custom licence can be anything from permissive to restrictive and the metadata will not tell you which. There are no GitHub releases either, so there is no tagged version to pin, and the last push to the repository was on 2026-06-01. The release history is instead tracked in an updating list at the top of the README, with dated entries for the Hugging Face online demo on 2025-10-29, the denoising enhancing codes on 2025-10-27, the uploaded training csvs and the sim2real metadata json on 2025-10-22, and the completion of the video upload on the same day. That list tells you what shipped and when, but it is not a substitute for a licence file.

Editorial conclusion

Ditto suits a research team studying instruction-based video editing who wants both the dataset and a trained checkpoint, and who can budget for the hardware the model implies. It does not suit a team that needs a small local model, since the base model it builds on is a 14 billion parameter video model, and it does not suit a commercial deployment until you have read the LICENSE file, because the repository records a custom licence rather than a standard identifier. Before adopting, check the GPU memory figure the README gives for a 73 frame clip, and note that the last push to the repository was on 2026-06-01 with the project publishing no GitHub releases.

Frequently asked questions

Official sources

1. EzioBy/Ditto on GitHub
2. Issues
3. Project website
4. README