# Prism: dynamic sparse attention for 2K joint video-audio training

> Prism is a sparse attention framework from Tencent Hunyuan and Fudan University for training joint video-audio generation models at 2K resolution. It groups tokens into spatiotemporal macro-zones, scores each zone by visual variance and audio-to-video cross-attention, and gives every zone its own attention block shape.

**Tencent-Hunyuan/Prism** — We propose Prism, a dynamic sparse attention framework for natively training joint video-audio generation models at 2K.

- Repository: https://github.com/Tencent-Hunyuan/Prism
- Website: https://francis-rings.github.io/Prism
- Stars: 835 · Forks: 10
- Language: Python
- License: NOASSERTION
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/tencent-hunyuan-prism

## The problem with full attention at high resolution

Native training of joint video-audio generation models at higher resolutions is supposed to give models richer visual details and sharper motion dynamics. The catch, as the README puts it, is that full attention incurs quadratic cost and, as the resolution increases, spreads attention over increasingly redundant tokens, which dilutes the learning signals for informative content and disrupts pretrained priors. Two distinct problems sit in that sentence. The first is cost, which is the familiar one. The second is more interesting, because a quadratic attention over a long token sequence does not just cost more, it also makes the model attend to a larger number of irrelevant positions per query, and that dilutes the gradient signal for the content that matters. The proposal is a dynamic sparse attention framework for natively training joint video-audio generation models at 2K, and the project topics name the ingredients: diffusion-models, high-resolution, sparse-attention and video-generation.

## What existing sparse attention misses

The README is explicit about the gap it is addressing. Existing sparse attention methods either target training-free acceleration or overlook the unique structure of joint video-audio data, where cross-modal interactions are inherently concentrated around sound-producing regions. That second point is the whole argument for a new framework rather than a new sparsity pattern. A method that was designed for text, or for video alone, assumes that redundancy is roughly uniform, and a sparsity pattern chosen on that assumption will keep the wrong tokens in a video-audio sequence. Here the useful tokens are predictable in a specific way, because the region where a sound comes from is the region that matters visually, so the attention budget should follow the audio rather than being spread evenly. The authors are from Fudan University, Tencent Hunyuan and Zhejiang University, and the paper is on arXiv with a project page and video walkthroughs linked from the README.

## Macro-zones and the two signals that score them

Prism organizes the token sequence into spatiotemporal macro-zones, which is what allows the attention structure to adapt to local content rather than applying one global pattern. Each zone is then scored on two signals, and the README is careful to describe both. The first is local information structure estimated via video feature variance along the channel, which captures how visual content varies directionally. The second is feature norms from the audio-to-video cross-attention, which capture how strongly audio influences each visual region. Together they describe a zone that is either visually busy or quiet, and that is either acoustically driven or not, and a zone that is both is where the interesting tokens are. The framework then assigns a tailored block shape to each zone, applying finer partitioning along the axes of rapid visual content variation and strong audio-visual coupling. The stated effect is that tokens within each block remain semantically coherent, which is what allows block-level features to capture both the visual content and the joint interaction patterns.

## Per-query sparsity on top of the per-zone shapes

Prism further adopts a hybrid block selection strategy to dynamically determine per-query sparsity, and this is the second layer of the design. The zone-level block shape decides how finely a region is partitioned, and the hybrid selection decides, for each individual query, how many blocks to attend to. The distinction matters because a zone is a spatial and temporal region, and the queries inside it are not equally informative. A query on a still background does not need the same number of blocks as a query on a fast moving object. The reported result is that Prism achieves a 2.5 times training speedup compared to full attention while surpassing it in generation quality. That last clause is the one to hold on to, because a sparsity method that speeds up training and quietly degrades the output is a common outcome, and the README states the opposite result for this framework. Whether that holds on your own data and your own resolution is a separate question, and the code and technical report are there to let you check.

## The environment you need before anything runs

The prerequisites are Python 3.10 or newer, CUDA 12.4 or newer, and a system-level `ffmpeg` installation. The install sequence in the README is five steps, and the first one pins the PyTorch stack to a specific CUDA build:

```bash
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu124
```

The second step installs the core dependencies from the requirements file, the third installs FlashAttention with `--no-build-isolation` because it needs CUDA at build time, the fourth installs ffmpeg at the system level, and the fifth is a verification step that imports torch, diffusers, transformers, flash_attn, triton, the AudioSignal class from audiotools and the VideoReader class from decord, and prints the version of each. That last step is worth copying verbatim, because it tells you immediately which part of the stack is broken rather than leaving you to guess from a training crash later.

## The requirements list tells you what the project depends on

The requirements.txt file is grouped by purpose, and the groups read like an architecture summary. The diffusion and transformers group pins diffusers 0.33.0 and transformers 4.57.1, with accelerate, safetensors and peft. The config and CLI group brings in omegaconf, shyaml, einops, loguru and ftfy. The audio codec is DAC VAE through descript-audiotools 0.7.4 or newer. Video I/O comes from decord 0.6.0 or newer plus imageio with the ffmpeg and pyav extras. The image and vision group has Pillow 11 or newer, torchvision 0.22 and torchaudio 2.7. Attention acceleration is flash-attn 2.8 or newer plus triton 3.3 or newer, both marked as requiring CUDA. Sequence parallelism is yunchang, which is the mechanism for splitting a long sequence across devices. Then there is the data and metrics group with numpy, pandas, tqdm, tensorboard and wandb, distributed training through deepspeed 0.18 or newer, and a misc bucket with ipdb, mmengine and scipy. The top level of the repository matches that structure, with a `hymm/` directory for the model code, `configs/`, `scripts/`, `tools/`, `assets/` and two inference shell scripts, `prism_infer.sh` and `prism_infer_fsdp.sh`, the second named for the sequence parallel data parallel strategy.

## Multi-node installs and the state of the release

If you are running on a multi-node cluster, the README gives a one-liner approach using `pdsh` to install the key packages across all nodes in parallel, and the pattern is worth reading even if you run on one machine, because it shows which package versions have to match across the cluster:

```bash
pdsh -f 256 -w $node_ip "pip install ipdb yunchang mmengine"
pdsh -f 256 -w $node_ip "pip uninstall diffusers -y && pip install diffusers==0.33.0"
```

The diffusers line uninstalls before installing, which is the right instinct on a cluster where a stale copy in one node's site-packages causes a mismatch that is very hard to see. The release checklist in the README is also a useful status document. Marked done are the Prism-preview-alpha stable release, the Prism-preview-beta motion release, the data pre-processing code for latent extraction and latent decode, the training code, the full finetuning code and the inference code. Still unchecked is Prism-pro, so the complete model is not released, and what is available is a preview checkpoint on Hugging Face. The news entry records that the project page, the training and inference code, the technical report and a preview model checkpoint were released on 2026-10-06.

## Conclusion

Prism fits a research team training joint video-audio generation models where full attention at 2K is the bottleneck, since the framework's claim is a training speedup rather than an inference trick, and since it is built around the specific structure of audio-visual data. It is the wrong choice if you are doing inference only, because the reported gain is in training, and the wrong choice if you lack multi-GPU CUDA hardware, since the environment requires CUDA 12.4, PyTorch 2.7.1 and FlashAttention. Before you start, read the project page and the arXiv paper, check the release checklist, since the full Prism-pro checkpoint is still an open to-do item, and plan for a multi-node install. The last push was on 2026-10-15, the project page, code, technical report and a preview checkpoint were released on 2026-10-06, and the licence is one GitHub cannot classify.

## FAQ

### What is the Prism framework for joint video-audio generation?

Prism is a dynamic sparse attention framework for natively training joint video-audio generation models at 2K resolution. It organizes the token sequence into spatiotemporal macro-zones, estimates each zone's local structure from video feature variance and audio-to-video cross-attention norms, and assigns a tailored attention block shape per zone, with a hybrid selection strategy for per-query sparsity.

### What does Prism require to run?

Python 3.10 or newer, CUDA 12.4 or newer, a system-level ffmpeg installation, PyTorch 2.7.1 with torchvision 0.22.1 and torchaudio 2.7.1 from the cu124 wheel index, and FlashAttention installed with `--no-build-isolation` because it needs CUDA at build time.

### Is the full Prism model released?

Not yet. The README checklist marks the preview-alpha and preview-beta releases, the data pre-processing, training, finetuning and inference code as done, while Prism-pro remains an open item. The project page, code, technical report and a preview model checkpoint were released on 2026-10-06.

### How does Prism compare to full attention in cost and quality?

The README reports a 2.5 times training speedup compared to full attention, while surpassing it in generation quality, with the reasoning that sparse attention over a high resolution token sequence otherwise dilutes learning signals for informative content and disrupts pretrained priors.

## Sources

- [Issues](https://github.com/Tencent-Hunyuan/Prism/issues)
- [Project website](https://francis-rings.github.io/Prism)
- [README](https://github.com/Tencent-Hunyuan/Prism/blob/main/README.md)
- [Tencent-Hunyuan/Prism on GitHub](https://github.com/Tencent-Hunyuan/Prism)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/tencent-hunyuan-prism