SparkDiffusion: accelerating Wan video diffusion with sparse attention and distillation

Three techniques stacked: sparse attention, few-step distillation, custom kernels

The README describes SparkDiffusion as a video-generation acceleration framework for Diffusion Transformer models, and the mechanism is a stack of three techniques rather than one. Sparse low-rank attention, called RoLa, replaces dense attention with a low-rank approximation. Few-step distillation, called CrossDistill, cuts the number of sampling steps a diffusion model needs. And custom high-performance operators under `sparkdiffusion/ops/`, which the README calls self-developed and dependency-free at inference time. The claimed result is 200x-plus end-to-end inference speedups over the dense multi-step baseline while preserving generation quality. The topics list gives the same picture with more vocabulary, including joint sparsity, distillation, quantization, Triton and w8a8, and the repository description puts a specific number on it, 265x. What makes this different from a single-technique paper is that the speedup comes from the combination, so each piece depends on the others being in place.

The target is Wan 2.1 and Wan 2.2, and the pipeline has three stages

The framework targets the Wan 2.1 and Wan 2.2 video diffusion models and provides an end-to-end pipeline, which the README lists as sparse-attention finetuning, few-step distillation, and optimized single-case T2V/I2V inference. That ordering matters. You cannot distill a model into few steps before you have made it sparse, because the distillation objective has to be computed against the model you will actually run. The repository layout table shows the three stages as directories. `scripts/sparse_finetune/` holds the Wan 2.1 and 2.2 sparse finetuning launchers, `scripts/distill/` holds distillation launchers for supported configurations, and `scripts/inference/` holds single-case shell wrappers. The core code sits in `sparkdiffusion/`, holding model, sampler, dataset, checkpoint, inference and operator code, and `imaginaire/` provides the training framework and configuration utilities.

The checkpoint table is the practical part of the README

Released weights live in a Hugging Face collection, and the README gives a table of what is available. Six checkpoints are listed, all Wan models, four of them text-to-video and one image-to-video for Wan 2.1, plus one Wan 2.2 text-to-video model. The columns are resolution, sparsity and steps, and those last two columns are where the acceleration lives. The 1.3B and 14B 480p text-to-video models sit at 0.90 sparsity with 4 steps. The 14B 720p model reaches 0.97 sparsity with 4 steps, and there is a separate 0.95 sparsity variant with 3 steps, which is the most aggressive step reduction in the table. The image-to-video model is 14B at 720P and 0.97 sparsity. The README notes that dense, sparse and distilled checkpoints are comparable through the same inference wrappers, which is the practical point for anyone switching between them.

Installing means PyTorch first, then requirements, then an env script

The requirements are a Linux box with a CUDA-capable GPU, Python 3.10 or newer, a CUDA-compatible PyTorch installation, and Triton, `flash-attn` plus the packages in `requirements.txt`. The order matters, because the requirements file assumes torch, torchvision and flash_attn are already installed. The README's own comment at the top of requirements.txt says it assumes already installed torch, torchvision, flash_attn. The quick start spells the order out:

git clone [email protected]:AlibabaResearch/SparkDiffusion.git
cd SparkDiffusion
pip install -r requirements.txt
source scripts/env.sh

So you install a CUDA-matched PyTorch and a compatible `flash-attn` build yourself first, then the rest of the file, then the environment script. `scripts/env.sh` is the piece that ties it together. It adds the repository to `PYTHONPATH`, enables offline defaults for Hugging Face and W&B, and defines repository-relative data and output roots.

The environment script is where your paths get set

The plain installation is two commands:

pip install -r requirements.txt
source scripts/env.sh

And the override form sets five roots explicitly:

PRETRAIN_ROOT=/path/to/pretrain_weights \
DISTILL_DATA_ROOT=/path/to/distill_data \
ROLA_DATA_ROOT=/path/to/rola_data \
DISTILL_OUTPUT_ROOT=/path/to/distill_outputs \
ROLA_OUTPUT_ROOT=/path/to/rola_outputs \
source scripts/env.sh

Those five variables, `PRETRAIN_ROOT`, `DISTILL_DATA_ROOT`, `ROLA_DATA_ROOT`, `DISTILL_OUTPUT_ROOT` and `ROLA_OUTPUT_ROOT`, are the contract between the scripts and your storage. The layout table gives the matching directories: `datasets/distill/` and `datasets/rola/` are local mount points for the distillation and sparse-finetuning datasets, `pretrain_weights/` is the local mount point for pretrained models, and `outputs/distill/` and `outputs/rola/` are where the two training stages write their results. The README is explicit that weights, datasets, checkpoints and generated videos are intentionally not included in the repository, so those paths are not optional in practice. You are expected to mount the native Wan assets yourself under `pretrain_weights/`, including the VAE, the T5 encoder, the tokenizer and the base DiT assets.

The dependency list shows a Megatron-based training stack

The requirements file is worth reading as a description of the machinery. The training framework is Hydra and OmegaConf for configuration, and Triton at 3.4 or newer, with a comment that Triton 3.4 fixes native FP8 MMA lowering for RTX 50-series cards on the SM120 architecture. There is a warning above the list that megatron-core must be uninstalled first, with the command to do it given in a comment, which tells you that the training path collides with a Megatron install rather than sitting alongside one. The rest of the list is a video generation stack: loguru, wandb and psutil for logging and monitoring, webdataset, pandas and imageio with ffmpeg for data, einops, safetensors, transformers, diffusers and accelerate for model handling, open_clip_torch for the ViT-H-14 vision tower used by native Wan 2.1 image-to-video CLIP checkpoints, and scipy, attrs, fvcore, nvidia-ml-py, ftfy and regex as supporting libraries. The headline dependencies:

hydra-core>=1.3.2
omegaconf>=2.3.0
# Triton 3.4 fixes native FP8 MMA lowering for RTX 50-series (SM120).
triton>=3.4.0

# Logging & monitoring
loguru>=0.7.0
wandb>=0.15.0
psutil>=5.9.0

The Triton version floor and the FP8 note are the two details to read twice if you are on recent hardware.

Where SparkDiffusion is the wrong tool

Three limits are concrete. The first is the hardware requirement. Linux, a CUDA-capable GPU, and a matching PyTorch and `flash-attn` build are prerequisites, and the README's own note about using `fp8` on GPUs with FP8 Tensor Core support and `bf16` otherwise shows that the speedup assumes modern accelerators. A team without that hardware, or without the appetite to install a specific CUDA-matched PyTorch, is outside the target. The second is that the acceleration is not free of quality trade-offs, and the README's phrasing is careful about it. The distilled models run in 3 or 4 steps against a baseline that runs many more, and the sparse checkpoints are trained at 0.90 to 0.97 sparsity, which means the model itself is different, not just the kernels. Whether that is acceptable is a judgement about output, and it is one you have to make on your own footage rather than on a benchmark number. The third is scope. The framework targets Wan 2.1 and Wan 2.2 specifically, so the sparsity finetuning and distillation work is tied to those checkpoints rather than being a general DiT acceleration layer.

The alternative is plain distillation, and the difference is the sparse structure

The realistic alternative is to take a Wan checkpoint and distill it to a few steps without touching attention structure, which is the CrossDistill piece on its own, and then run the dense model with a fast sampler. The difference in approach is where the speedup comes from. Step reduction attacks the number of denoising iterations, so it multiplies whatever the per-step cost is. Sparsity attacks the per-step cost by making attention itself cheaper, and low-rank factorization attacks it again from a different angle. Doing both means the distilled model has to be trained against the sparse model, which is why the README orders the pipeline as finetune first and distill second, and why the release is a set of checkpoints rather than a sampler you point at an existing model. The other alternative is serving the model on a faster machine or a better inference stack, where the difference is that you get the same model at the same quality and you spend money rather than engineering. SparkDiffusion is the option for a team that has decided latency matters more than the cost of retraining the model.

Editorial conclusion

SparkDiffusion fits a team running Wan 2.1 or Wan 2.2 video generation on Linux with a recent CUDA GPU that needs lower latency and is willing to retrain the model at high sparsity and few steps. It is the wrong tool for a hardware budget that does not include a CUDA GPU, and the wrong tool if you need a different base model, since the sparsity finetuning and distillation stages are tied to the Wan checkpoints. Before you adopt it, check that the checkpoint matching your resolution and task is in the Hugging Face collection, and confirm your Triton version clears the FP8 lowering requirement the requirements file calls out.

Frequently asked questions

Official sources

1. AlibabaResearch/SparkDiffusion on GitHub
2. Issues
3. License: Apache-2.0
4. README