StereoCrafter turns a flat video into a 3D one with two diffusion passes

The problem is a missing right eye, and a missing right eye has holes

A flat video carries one view of the world. A stereoscopic 3D display needs two, one for each eye, and the two must disagree in the right way or the result looks wrong. StereoCrafter is a framework that converts 2D video into immersive stereoscopic 3D for devices such as 3D glasses, the Apple Vision Pro and 3D displays, and the documentation names the intended sources as movies, vlogs, 3D cartoons and AIGC video. The audience is therefore a studio or a research group with ordinary footage and no second camera.

The obstacle is that inventing a right eye is not a warping problem alone. If you take the left image and shift pixels along a depth estimate, the pixels that were hidden behind foreground objects have no source data, and shifting them produces smears and holes. Any approach has to deal with that occlusion problem, and StereoCrafter's answer is to spend a second diffusion model on it.

The project came out of Tencent AI Lab and ARC Lab at Tencent PCG, with a technical report on arXiv and a project page, both linked from the README. The examples in the README are presented in anaglyph 3D format, which is the cheap way to preview a stereo pair on a normal screen, and the pipeline also emits side-by-side video for devices that want that layout.

Stage one splats a right eye out of a depth sequence

The first stage is depth based video splatting. A separate script takes the input video and the depth model, produces a depth map, and forward splats the left view into a right view, writing a video that also carries an occlusion mask so you can see what could not be filled in. The script is `depth_splatting_inference.py`, and the documented invocation is:

python depth_splatting_inference.py --pre_trained_path [PATH] --unet_path [PATH]
                                    --input_video_path [PATH] --output_video_path [PATH]

The path arguments have concrete defaults in the documentation. `--pre_trained_path` points at the SVD img2vid weights, `--unet_path` at the DepthCrafter weights, and the input and output paths are the source video and the splatting result. One parameter controls the geometry directly: `--max_disp` sets the maximum disparity between the generated right video and the input left video, and its default is 20 pixels. That number is a real knob and a real limit, since disparity in pixels is what the display turns into depth, and a scene with a very near foreground needs more of it than a flat one.

The output of this stage is described as a video grid containing the input video, the visualized depth map, the occlusion mask and the splatting right video. Seeing all four at once is how you diagnose a bad result, and the mask tells you which pixels the next stage has to invent.

Stage two inpaints the occluded regions with the stereo model

The second stage is stereo video inpainting, run by `inpainting_inference.py`, and it consumes the splatting video produced by stage one. The documented command is:

python inpainting_inference.py --pre_trained_path [PATH] --unet_path [PATH]
                               --input_video_path [PATH] --save_dir [PATH]

Here `--unet_path` points at the StereoCrafter model weights instead of the depth model, and `--input_video_path` is the stage one result. The script writes into `--save_dir`.

The interesting parameter is `--tile_num`, the number of tiles in the width and height dimensions for tiled processing. The default is 1, a single tile, and the documentation is explicit about why the option exists: it allows handling high resolution input without requiring more GPU memory, and for input videos at 2K resolution or higher you can use more tiles to avoid running out of memory. That is the shape of the whole design. A single diffusion pass over a full 2K frame is a memory problem, so the work is cut into tiles, and the tiling is what makes the method reachable on a single consumer card rather than only on a research cluster.

The final artifacts are the stereo video in side-by-side format and in anaglyph 3D format.

Three checkpoints, fetched over git lfs from Hugging Face

StereoCrafter does not ship weights. It orchestrates three separate models, and the README documents exactly which. The SVD img2vid model provides the image encoder and the VAE, and the download is a git lfs clone into a `weights` directory in the project root:

mkdir weights
cd ./weights
git lfs install
git clone https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt-1-1

The depth model comes from `huggingface.co/tencent/DepthCrafter`, and the stereo generation model from `huggingface.co/TencentARC/StereoCrafter`, each cloned as its own repository. The dependency split matters for anyone planning disk, time or licensing. The image encoder and VAE come from Stability AI, the depth estimate comes from Tencent's DepthCrafter, and only the final stereo generation model belongs to this project. The README acknowledges the first two as open source projects it builds on, and credits Stable Video Diffusion as a latent diffusion model trained to generate video clips from image or text conditioning, and DepthCrafter as a method for generating temporally consistent depth sequences from videos.

The `git lfs install` step is not optional in practice. Without it the clone fetches pointer files instead of the large checkpoint blobs, and the failure appears later as a model that loads nothing useful.

The forward warp extension has to be compiled on your machine

The installation is four steps and one of them is a build. The environment is Python 3.8 with Cuda 11.8, and the documentation says you can use Anaconda or Docker to build it. The clone needs the recursive flag because the forward warping code arrives as a submodule:

git clone --recursive https://github.com/TencentARC/StereoCrafter
cd StereoCrafter

The Python dependencies are pinned in `requirements.txt` and installed with `pip install -r requirements.txt`. The pins are the ones you would expect from a diffusion stack of that vintage, including `torch==2.0.1`, `torchvision==0.15.2`, `diffusers==0.29.2`, `transformers==4.42.3`, `accelerate==1.0.1` and `xformers==0.0.20`, alongside video and plotting libraries such as `decord==0.6.0`, `av==12.3.0`, `mediapy==1.2.2` and `opencv-python==4.10.0.84`.

The custom piece is the forward warp package used for the splatting operation. It lives in the `dependency/Forward-Warp` directory of the clone and is installed by its own shell script:

cd ./dependency/Forward-Warp
chmod a+x install.sh
./install.sh

The `chmod` step is not cosmetic, since the script is not executable in a fresh clone. Expect this step to compile against the local CUDA toolkit, which is where the Cuda 11.8 requirement bites.

Running it, and where the memory wall sits

Inference is driven by a single shell script in the project root:

sh run_inference.sh

The README describes two main steps in that script, which correspond to the two python entry points above. The repository layout explains the rest: `pipelines/` holds the pipeline code, `source_video/` holds sample footage, `assets/` holds the images used in the README, and `.gitmodules` declares the submodules that the recursive clone pulls in.

The practical constraints are known. The `--max_disp` default of 20 pixels caps the depth separation you get without changing it, and `--tile_num` at its default of 1 means a 2K or larger input can exhaust GPU memory unless you tile. Both are documented, but both are the kind of detail that turns a demo into a pipeline problem, because a shop converting a feature length film is not working with the sample videos in `source_video/`.

The dependency pins make the second constraint sharper. `torch==2.0.1` and `xformers==0.0.20` are tied to the CUDA generation they were built for, so moving to a newer GPU means rebuilding that environment rather than installing a newer torch. The project also has no tagged releases on GitHub. The README's news entries record an update to the project on 2026/05/20 that added inference code and model weights in a v2 branch, the earlier release of inference code and weights on 2024/12/27, and the arXiv submission and project page on 2024/09/11. The last push to the repository was on 2026-05-27, so the tree is not frozen, but the work arrives as branch updates rather than as versioned releases with changelogs.

A custom licence file, not a standard one

The repository is not under a licence GitHub can classify. The entry for it is `License-Code.txt` rather than a standard LICENSE file, and the repository carries no recognised open source identifier. If you intend to use StereoCrafter in a commercial product, a film pipeline or a hosted service, that file is the first thing to read, because the terms are not visible from a badge or a label. Nothing in the README summarises them, so the obligations have to be read directly rather than inferred.

The dependencies bring their own terms into the same decision. The Stable Video Diffusion checkpoint and the DepthCrafter checkpoint are separate downloads with their own licences, and the paper and the arXiv submission carry the academic use expectations that usually apply to research code and weights. A team that treats the pipeline as research tooling can proceed without much thought. A team that wants to ship stereo conversion as a product needs to check `License-Code.txt`, the licence on the Stable Video Diffusion weights and the licence on the DepthCrafter weights, in that order, because a compliant orchestration script is worth nothing if the weights underneath it are not.

The upgrade path is straightforward and unstructured. There is no release history to plan against, so the practical approach is to track the `main` branch and the v2 branch that the 2026/05/20 news entry refers to, and to pin the specific model checkpoints you validated rather than assuming the newest clone is the one your results came from.

Editorial conclusion

StereoCrafter suits a researcher or post-production team that already has a CUDA 11.8 machine and needs stereoscopic output from flat footage without a capture rig, and it does not suit a pipeline that needs a maintained release cadence or a permissive licence. Before you build on it, check the two files that decide whether you can use it at all: `License-Code.txt`, which GitHub classifies as a custom licence, and `requirements.txt`, which pins torch 2.0.1 and diffusers 0.29.2, versions that will not lift to a newer CUDA stack without work.

Frequently asked questions

Official sources

1. Issues
2. README
3. TencentARC/StereoCrafter on GitHub