MatAnyone2, video matting driven by a learned quality evaluator

Matting without the segmentation edge

MatAnyone2 is described as a practical human video matting framework that preserves fine details by avoiding segmentation-like boundaries, and it also shows enhanced robustness under challenging real-world conditions. The paper is Scaling Video Matting via a Learned Quality Evaluator, and the repository carries a CVPR 2026 Highlight designation.

The problem it addresses is the difference between cutting an object out and matting it. A segmentation mask is a hard boundary: every pixel is either subject or background, which is fine for recognition and wrong for compositing, because hair, fur, motion blur and semi-transparent edges need soft alpha values. A matting model produces those alpha values, and the quality of that alpha is what determines whether a composite looks like footage or like a sticker.

The contribution named in the title is the learned quality evaluator, which is the component that scales the approach. The idea is that the system can judge the quality of its own output and therefore decide where to spend effort across a video, rather than treating every frame as equally difficult.

The work comes from S-Lab at Nanyang Technological University together with SenseTime Research in Singapore, with Peiqing Yang, Shangchen Zhou as project lead, Kai Hao and Qingyi Tao as the authors. The paper is on arXiv at 2512.11782, and there is a project page, a demo video and a Hugging Face space with a Gradio interface.

The repository is Python, built with hatchling, requires Python 3.10 or newer, and the licence field on the repository points to a custom licence in the LICENSE.txt file rather than to a standard identifier.

The input contract is a video plus a first-frame mask

The interface is simple and the constraint is worth understanding before anything else. For each run, you take a video and its first-frame segmentation mask as input. The mask can come from interactive segmentation models such as the SAM2 image predictor demo.

The layout the README asks for separates the two:

inputs
   |- video
      |- test-sample1          # folder containing all frames
      |- test-sample2.mp4      # .mp4, .mov, .avi
   |- mask
      |- test-sample1.png      # mask for targer person(s)
      |- test-sample2.png

So you draw a mask once, on the first frame, and the model propagates the subject through the rest of the video. That is the practical appeal of the design compared with a per-frame matting pipeline, where the mask would have to be produced or corrected for every frame. It also means the quality of your output is bounded by the quality of one mask, and that a person who leaves the frame or is occluded is the case that decides whether the result is usable.

The demo removes that preparation step. The Gradio interface lets you drop in a video or an image, assign the target masks with a few clicks, and get the matting result, and the demo integrates the MatAnyone series with MatAnyone 2 as the default model and the original MatAnyone available through a model selection control.

Two ways to install, and one that gets the toolchain right

The README documents a conda path and a uv path, and the uv path is the newer one, added in March 2026 together with CLI and Hugging Face support.

The conda route clones the repository, creates an environment on Python 3.10 and installs the package in editable mode, with an optional extra for the Gradio demo:

git clone https://github.com/pq-yang/MatAnyone2
cd MatAnyone2

conda create -n matanyone2 python=3.10 -y
conda activate matanyone2
pip install -e .
# [optional] install python dependencies for gradio demo
pip3 install -r hugging_face/requirements.txt

The uv route creates a separate project and adds the package from the git repository:

uv init my-matting-project && cd my-matting-project
uv add matanyone2@git+https://github.com/pq-yang/MatAnyone2.git

The uv path has one detail that matters and is easy to miss. The project configuration declares a dedicated index for the PyTorch CUDA 12.8 wheels and points torch and torchvision at it, and it disables build isolation for the thinplate package, which comes from a git repository and therefore cannot be built in an isolated environment.

That dependency on a git source is worth knowing about, because it means an offline or restricted-network build will not work out of the box. The rest of the dependency list is the usual matting and tooling set: kornia, einops, hydra-core for configuration, gradio, av and imageio with ffmpeg for video, safetensors and huggingface_hub pinned at 0.36.2 for checkpoint handling, pycocotools for the evaluation side, and PySide6 with pyqtdarktheme for the interface.

Inference from the command line or from Python

The checkpoint is distributed as a release asset, matanyone2.pth, which you download into a pretrained_models folder. The model can also be downloaded automatically during the first inference, which saves a step if you are trying the tool on a single clip. The expected directory structure is a pretrained_models directory containing matanyone2.pth.

The inference script takes the video path and the mask path, and works with either a folder of frames or a single video file:

python inference_matanyone2.py -i inputs/video/test-sample1 -m inputs/mask/test-sample1.png
python inference_matanyone2.py -i inputs/video/test-sample2.mp4 -m inputs/mask/test-sample2.png

Results are written to a results folder, and the README says they include the foreground output video and the alpha output video. Two options are documented. --save-image writes per-frame images instead of, or as well as, video. --max-size sets a limit on the maximum input resolution, and the video is downsampled if the smaller of width and height exceeds it. By default no limit is set, which means a high-resolution clip is processed at full resolution.

The same work is available as a console script, since the package registers matanyone2 as an entry point:

matanyone2 -i inputs/video/test-sample1 -m inputs/mask/test-sample1.png
matanyone2 --help

For programmatic use the API is short, and the model loads straight from Hugging Face:

from matanyone2 import MatAnyone2, InferenceCore

model = MatAnyone2.from_pretrained("PeiqingYang/MatAnyone2")
processor = InferenceCore(model, device="cuda:0")
processor.process_video(
    input_path="inputs/video/test-sample2.mp4",
    mask_path="inputs/mask/test-sample2.png",
    output_path="results",
)

Note that the device is passed explicitly, so this is a GPU path by default. There is no CPU fallback mentioned in the documentation.

The demo runs locally as well as on Hugging Face

The interactive demo is served on Hugging Face, and the README also explains how to launch it on your own machine, which is the option to pick if your footage should not leave the host.

The local path is a directory in the repository and two commands:

cd hugging_face

# install GUI dependencies
pip3 install -r requirements.txt # FFmpeg required

# launch the demo
python app.py

The FFmpeg requirement is called out in the comment, which is reasonable for a tool that decodes and encodes video.

There is a second update worth knowing about, from June 2026: MatAnyone2Kit, described as a third-party Core ML and Swift package port of MatAnyone 2 for Apple platforms in real time. It lives in its own repository rather than this one, and the README points there for details. A real-time Apple Silicon port is a different requirement from a research inference script, and the fact that it exists outside the main repository tells you the main repository is not aiming at that deployment target itself.

The evaluation side has its own documentation, with the evaluation workflow described in docs/EVAL.md, and the repository holds an evaluation/ directory. The data pipeline the project used is documented in a diagram in the docs.

Three things are still not released

The TODO list in the README is the most important section for anyone planning to build on this, and it has three unchecked boxes.

Release training codes for the video matting model. That work is not published.

Release the checkpoint and the training codes for the quality evaluator model. The evaluator is the component named in the paper title and the reason the approach scales, and neither its weights nor its training code are available.

Release the real-world video matting dataset, VMReal, which the data pipeline diagram in the repository refers to.

What is released, then, is the inference path: the matting checkpoint, the inference code, the evaluation code, the Gradio demo, the CLI and the Python API, all added between December 2025 and March 2026.

That split shapes the audience. MatAnyone2 is currently usable as an applied tool, for matting video you have and a mask you can produce. It is not yet usable as a research platform, because you cannot retrain the matting model, you cannot train or evaluate your own quality evaluator, and the dataset the work was built on is not published.

The evaluation code being released without the evaluator checkpoint is a particular combination. It suggests the evaluation path is usable against the released model, and that anyone wanting to study the evaluator itself is out of luck for now.

Editorial conclusion

MatAnyone2 suits a team that needs to cut a moving subject out of video with a mask they can click once, and who will run inference on a GPU and download the released checkpoint. It does not suit a team that needs to retrain the model, train the quality evaluator, or work with the VMReal dataset, because none of those are released. Before you build on it, download matanyone2.pth and run the two documented inference commands on your own footage, decide whether you can live with the first-frame mask as the input contract, and check the licence in LICENSE.txt, which is a custom file rather than a standard identifier.

Frequently asked questions

Official sources

1. Issues
2. pq-yang/MatAnyone2 on GitHub
3. README
4. Releases