# VideoMaMa: mask-guided video matting built on a video generative prior > VideoMaMa is the official implementation of the CVPR 2026 paper VideoMaMa: Mask-Guided Video Matting via Generative Prior, which produces per-frame alpha mattes for video using a Stable Video Diffusion prior plus mask guidance. Training code and weights are released, evaluation code and the MA-V dataset are not, and the checkpoints carry a separate licence. **cvlab-kaist/VideoMaMa** — Official implementation of "VideoMaMa: Mask-Guided Video Matting via Generative Prior", CVPR 2026 - Repository: https://github.com/cvlab-kaist/VideoMaMa - Stars: 534 · Forks: 33 - Language: Python - License: NOASSERTION - Published: 2026-10-22 · Updated: 2026-10-22 · Language: en - Canonical page: https://hysenlabs.com/projects/cvlab-kaist-videomama ## What the paper contributes VideoMaMa is described as a mask-guided video matting framework that leverages a video generative prior. By utilizing this prior, it supports stable performance across diverse video domains with fine-grained matting quality. Video matting is the problem of producing an alpha matte for a foreground subject in every frame of a video, so that the subject can be composited over a different background. Image matting is hard enough; doing it consistently across hundreds of frames is harder, because a per-frame algorithm drifts. A subject's edges change as they move, hair and motion blur are semi-transparent, and a method that is clean on frame one can smear by frame two hundred. The architecture choice here is to use a generative video model as the prior rather than treating each frame independently. The acknowledgements name Stable Video Diffusion from Stability AI as the base, and the inference command takes a base_model_path pointing at a stable-video-diffusion-img2vid-xt checkpoint. So the temporal consistency comes from a model that was built to generate temporally coherent video, and the matting is obtained by conditioning that model on a mask. The mask is what makes it guided rather than a general generator, and that matters for a different reason. A mask says where the subject is, which is the one piece of information a matting model otherwise has to infer from the image alone and often gets wrong. Supplying it removes an entire class of failure. The authors are from Korea University, Adobe Research and KAIST AI, the paper is a CVPR 2026 paper with an arXiv preprint at 2601.14255, and the project page is at cvlab-kaist.github.io/VideoMaMa. ## Two components, one from Meta and one from Stability The dependencies reveal the architecture more precisely than the abstract does. The first is SAM2, Meta AI's Segment Anything 2, which the setup script downloads and which the README says is needed for training the sam2-matte component. The naming suggests a two-part model: a segmentation model producing masks and a matting model refining them to soft edges. The demo directory makes this concrete, with sam2_wrapper.py and sam2_wrapper_hf.py alongside videomama_wrapper.py and videomama_wrapper_hf.py, and a sam2_hiera_l.yaml configuration. The paired _hf variants suggest the same models are available through Hugging Face integrations, which is what lets the demo run on the Hugging Face Spaces deployment. The second is the diffusion side. The setup script downloads Stable Video Diffusion weights, and the inference script takes a unet_checkpoint_path for the VideoMaMa weights and a base_model_path for the SVD checkpoint. The pipeline file is named pipeline_svd_mask.py, which is a strong hint about the mechanism: a Stable Video Diffusion pipeline extended to accept a mask. So the model is a modified diffusion UNet that has been adapted to take a mask alongside the image, with a projection head mapping it into the UNet's conditioning space. The file dino_projection_mlp.pth, named explicitly in the licence section, is that projection, and DINO is the self-supervised visual representation used to build it. The setup.py dependency list adds diffusers, transformers, accelerate and peft, which is the standard stack for fine-tuning a diffusion model with adapters. The practical consequence is that VideoMaMa is not a lightweight matting network. It inherits the memory and compute requirements of a video diffusion model, and the deepspeed and flash_attn dependencies confirm that the training path is a large-model training path. ## Setting up the environment Setup is a single script: ```bash bash scripts/setup.sh ``` The README explains what it does: it downloads the stable video diffusion weights and sets up the virtual environment needed to run the whole code. The environment is a conda one, activated with conda activate videomama. The script also downloads SAM2, which is needed for training the sam2-matte component, so a training user gets both models in one step. The Python package requirements are listed in setup.py, and they are worth reading before you commit disk and GPU time. The core stack is torch at 2.4.0 or newer with matching torchvision, diffusers at 0.31.0 or newer, transformers at 4.49.0 or newer and accelerate at 1.9.0 or newer for the training loop. There is opencv-python, numpy, scipy, matplotlib, scikit-image, ftfy, moviepy, imageio and pycocotools for data and output handling. There is boto3 at 1.37.22 or newer, which points at S3-backed dataset handling. omegaconf and easydict are configuration handling, tabulate and pandas for reporting, wandb for experiment tracking, datasets for the Hugging Face dataset tooling, and peft above 0.15 for parameter-efficient fine-tuning. Two dependencies stand out. flash_attn is listed, which is the FlashAttention package and needs a GPU architecture and a compiler toolchain to build. deepspeed is pinned to exactly 0.17.1, which is a hard pin rather than a range, so it will conflict with anything else in your environment that wants a different DeepSpeed version. Python 3.9 or newer is required, and the utils3d package is pulled from a git URL rather than PyPI, through the dependency_links mechanism. For a demo-only user, the demo directory has its own requirements.txt and a download_checkpoints.sh script, which is a lighter path than the full training stack. ## Running inference on a folder of frames Inference is driven by a single script, inference_onestep_folder.py, with the argument shape shown in the README: ```bash python inference_onestep_folder.py \ --base_model_path "checkpoints/stable-video-diffusion-img2vid-xt" \ --unet_checkpoint_path "checkpoints/VideoMaMa" \ --image_root_path "/assets/example/image" \ --mask_root_path "assets/example/mask" \ --output_dir "assets/example" \ --keep_aspect_ratio ``` The arguments divide cleanly into three groups. The base_model_path points at the Stable Video Diffusion checkpoint, the unet_checkpoint_path points at the VideoMaMa weights, and the input paths are given as roots of directories: one for images and one for masks, with the output written to a third directory. The script processes a folder rather than a single file, which is the natural unit for video matting since the output is a sequence. The --keep_aspect_ratio flag appears in the worked example but not in the schematic argument list, which shows an optional arguments placeholder after --output_dir. Further inference settings are documented in inference.md. The model checkpoint itself is on the Hugging Face Hub as SammyLim/VideoMaMa, with a model card, and there is a demo on Hugging Face Spaces under the same name. For interactive use rather than a batch run, the demo directory has app.py built on Gradio, with the acknowledgements crediting Gradio for the UI framework. ## What is released and what is not The TODO list in the README is short and it is a status board for the project. Released: the demo and model checkpoint on 2026-01-19, the arXiv paper on the same date, and the training code on 2026-03-14. Not released: evaluation code, and the MA-V dataset. The evaluation section of the README is even commented out, with the line pointing at evaluation/README.md left inside an HTML comment. That absence matters for anyone trying to reproduce the paper's numbers. Without the evaluation code and the MA-V dataset, you cannot compute the reported metrics, so the results in the paper are not independently checkable from this repository. For a project that also does not publish GitHub releases, the repository plus the arXiv paper is the whole artefact set. The training path is documented in two places: the data pipeline README under data_pipeline/ for generating the training dataset, and training.md for model training. The top-level entries show the corresponding code, with train.py, training_scripts/ and a dataloader/ directory. The news list shows the ecosystem forming around the work: a ComfyUI-VideoMaMa node by okdalto in February 2026, an implementation on CorridorKey in March, and one on Sammie-Roto-2 in April, which credits the authors using their model. The most recent entry, on 2026-10-20, is the release of VideoMaMa++, an extension of the same work. ## Two licences, and the one that constrains you The licence section has three parts and they matter in different ways. The code in the repository is released under CC BY-NC 4.0 unless otherwise specified. That is a Creative Commons licence with a non-commercial restriction, which is unusual for research code and immediately rules out using this code inside a commercial product without a separate arrangement. The model checkpoints, specifically VideoMama/unet/* and dino_projection_mlp.pth, are subject to the Stability AI Community License, which is the licence that comes with the Stable Video Diffusion weights the model is built on. That licence has its own terms and is updated from time to time, which means the terms you accept today may not be the terms that apply to a later download. The README links directly to the licence text. So there are two layers of permission, and the narrower one is on the weights rather than the code. A commercial deployment has to satisfy both. The project also builds on implementations and ideas from the Hugging Face ecosystem and from the diffusion-e2e-ft project from the Visual Computing Institute, with thanks to the original authors and contributors. The README classifies the repository licence as a custom licence GitHub cannot map to a known identifier, which is consistent with CC BY-NC 4.0 sitting in a License.md file rather than a LICENSE file. That file is the authoritative text. For the industrial case the README makes an explicit offer: if you are interested in using this model for industrial applications, or have specific questions about the architecture and training, the authors invite you to reach out. That is the route for anyone whose use the non-commercial term blocks. ## Where VideoMaMa is the wrong tool Three limits are structural. The first is that you cannot reproduce the paper's evaluation. The evaluation code and the MA-V dataset are both listed as unreleased, and the evaluation section is commented out in the README, so there is no way from this repository to compute the reported numbers or to compare against a competing method on the same data. For a project whose value is a CVPR result, that is a substantial gap. The second is the compute profile. The stack is a video diffusion UNet fine-tuned from Stable Video Diffusion, with flash_attn, deepspeed and peft in the dependency list. Training is a multi-GPU, large-model job, and inference is not cheap either since the base model generates video frames. A team that needs per-frame alpha mattes at scale for a video pipeline will find this considerably more expensive than a dedicated matting network, and it is not the tool for a real-time or edge deployment. The third is the licence structure, which we have already described but which is worth restating as a practical constraint rather than a legal opinion. CC BY-NC 4.0 on the code and the Stability AI Community License on the weights mean that a commercial use case is not simply a matter of complying with an attribution clause; it needs either a separate arrangement with the authors or a different model. The alternative approach is a dedicated video matting model, one of the architectures built specifically for the task, which are far smaller, trained on matting datasets with matting metrics as the objective, and therefore both cheaper to run and easier to evaluate against published numbers. The difference in approach is that VideoMaMa treats matting as a controlled generation problem and inherits a general video model's temporal priors, whereas a purpose-built matting network treats it as a per-frame regression refined for consistency across frames. If you need a published comparison, a benchmark number, or a model that fits on a single GPU, the purpose-built route is the one that will get you there. ## Conclusion VideoMaMa suits a research team working on video matting or segmentation who wants the paper's model with released training code and a checkpoint, since the mask-guided diffusion approach is the contribution and the code supports training on your own data pipeline. It is the wrong tool if you need to evaluate against published benchmarks, because the README still lists evaluation code and the MA-V dataset as unreleased, and it is the wrong tool for commercial deployment, because the code is CC BY-NC 4.0 and the checkpoints fall under the Stability AI Community License. Verify first which licence your use falls under, that the model checkpoint terms allow your intended distribution, and note that VideoMaMa++ was released on 2026-10-20 with the last push on the same day. ## FAQ ### What is VideoMaMa and what problem does it solve? It is a mask-guided video matting framework that leverages a video generative prior, so it produces fine-grained alpha mattes for video with stable performance across diverse domains. It is the official implementation of the CVPR 2026 paper VideoMaMa: Mask-Guided Video Matting via Generative Prior. ### How do I set up the VideoMaMa environment? Run bash scripts/setup.sh, which downloads the stable video diffusion weights and sets up the virtual environment. The environment is a conda one, activated with conda activate videomama, and the script also downloads SAM2 for training the sam2-matte component. ### How do I run inference with VideoMaMa? Run inference_onestep_folder.py with a base_model_path pointing at the stable-video-diffusion-img2vid-xt checkpoint, an unet_checkpoint_path pointing at the VideoMaMa weights, and the image_root_path, mask_root_path and output_dir directories. The model checkpoint is on the Hugging Face Hub as SammyLim/VideoMaMa. ### Is the evaluation code and the MA-V dataset available? Not yet. The README TODO list marks evaluation code and the MA-V dataset as not released, and the evaluation section of the README is commented out, so the paper's reported metrics cannot be reproduced from this repository. ### What licence applies to the VideoMaMa code and model weights? The code is released under CC BY-NC 4.0 unless otherwise specified, and the checkpoints VideoMama/unet/* and dino_projection_mlp.pth are subject to the Stability AI Community License, which is updated from time to time. The README invites contact about industrial applications. ### Which upstream models does VideoMaMa depend on? SAM2, Meta AI's Segment Anything 2, for the mask side, and Stable Video Diffusion from Stability AI as the generative prior the UNet is adapted from. The acknowledgements also credit Gradio for the demo UI framework. ## Sources - [cvlab-kaist/VideoMaMa on GitHub](https://github.com/cvlab-kaist/VideoMaMa) - [Issues](https://github.com/cvlab-kaist/VideoMaMa/issues) - [README](https://github.com/cvlab-kaist/VideoMaMa/blob/main/README.md) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/cvlab-kaist-videomama