Code Video Model turns a prompt into a Three.js scene and then into a video

Four stages, and every intermediate file is kept

The pipeline is described as a sequence of inference steps, and the thing that makes it interesting is that each stage writes its output to disk in a form you can look at. The first stage turns a text prompt into an editable Three.js scene, and the README says the language model builds the codebase and then refines it through direct visual review, which is the part that distinguishes this from a one-shot generation. The second stage renders that scene to a proxy video and, optionally, produces a reference image from it. The third stage hands the proxy video, the reference image and the original prompt to the video model to produce the final clip. A fourth stage handles paired work, where an original scene and an edit instruction are combined. The examples directory follows the same structure, with a sports car, a tank boarding scene, a loft tour, a robot navigation scene and a chair placement pair, each holding scene code, a proxy video, a reference image and a prompt file. The quick start command takes a directory name and points at those three files, so running the pipeline is mostly a matter of choosing an example.

Stage one writes code and then looks at it

The scene generation step is a single module with a prompt and an output directory.

python -m inference.generate_threejs \
  --prompt "A red car follows a curved road through a sunlit town, with a smooth tracking camera." \
  --output outputs/threejs

The generated scene files are saved in the output workspace's case directory, and prompts can be typed in or read from a text file. Two follow-up commands act on that scene. A render command turns it into a video, and a viewer command serves it in a browser on a chosen port.

python -m inference.render video --scene outputs/threejs

python -m inference.viewer --scene outputs/threejs --port 8000

The viewer's description is worth reading, because it is aimed at inspection rather than at playing a game. You can play or scrub the timeline, step through frames, and select a source file to read its code, and there is a World Inspector where you drag to orbit, shift-drag to pan and scroll to zoom, with toggles for the camera frustum, the trajectory and the grid. That combination, a timeline, the source next to the render and a spatial inspector, is what makes the intermediate scene debuggable rather than a throwaway artefact, and it is the part of the repository an adopter should try first.

Stage two turns the proxy into a reference image

The reference image exists to give the video model something to anchor on, and there are two ways to make one. The first is strict alignment, where the first frame of the video is extracted and an image model is asked to create a 1536 by 864 reference with instructions to preserve the scene's composition and content.

python -m inference.generate_reference \
  --video outputs/threejs/case/reconstruction.mp4 \
  --output outputs/threejs/reference.png

An already extracted frame can be given instead of a video. The second is text guided, where the first frame is treated as loose visual context and a prompt guides the result, without requiring the same camera, composition or object placement.

python -m inference.generate_reference_guided \
  --video examples/loft_tour/code_video.mp4 \
  --prompt "A sunlit modern loft with a leather sofa and floating stairs, photographed at eye level." \
  --output outputs/loft_reference_guided.png

That mode also accepts an extracted frame and a scene description loaded from a file, and it produces the same 1536 by 864 output. The distinction between the two is the interesting design point, because strict alignment is for reproducing a specific shot while the guided mode is for changing it, and both end up feeding the same argument list to the final stage.

Stage three is the video model call

The final generation step takes the three artefacts, the proxy video from the scene, the reference image, and the prompt file, and runs the MiniMax-H3 checkpoint that was downloaded separately.

python -m inference.generate_video \
  --code-video outputs/threejs/case/reconstruction.mp4 \
  --reference-image outputs/threejs/reference.png \
  --prompt-file outputs/threejs/prompt.txt \
  --model-root models/MiniMax-H3-ref2va \
  --keep-steps 5 \
  --output outputs/threejs/video.mp4

The quick start and inference commands pass --keep-steps 5, and the checkpoint is fetched with a helper script that takes an output directory.

python download_h3.py --output models/MiniMax-H3-ref2va

The paired editing stage runs the same model with an anchor workspace and an edit file, and the example given is a chair placement pair, where the original and edited workspaces each hold scene code, a proxy video, a reference image and a prompt, with a pair type argument that selects the kind of pairing. The practical reading is that the pipeline is modular at each arrow, and you can substitute your own scene, your own reference or your own prompt at any of the four entry points.

The benchmark ships 22 prepared cases and a way to look at them

CodeVideoBench is published on Hugging Face and provides 22 cases across eight applications, with editable Three.js programs, proxy videos, reference images, prompts and a browser based viewer. The download uses the Hugging Face hub command line, and the gallery can be served with the standard library's HTTP server bound to the loopback address.

hf download code-video-model/CodeVideoBench \
  --repo-type dataset \
  --local-dir data/CodeVideoBench

python -m http.server 8001 --bind 127.0.0.1 --directory data/CodeVideoBench

A prepared case is run with the same three inputs as the third inference stage, using the code video, the reference image and the prompt file from the case directory. The value of shipping the benchmark with prepared scenes is that you can evaluate a change to the pipeline against fixed inputs, which is the difference between a research artefact and a demo. The evaluation script is separate and scores three dimensions, structure, style and quality, using a hosted model accessed through a router service, with one request per dimension and the scores saved.

python evaluate_video.py \
  --video outputs/sports_car.mp4 \
  --code-video examples/sports_car/code_video.mp4 \
  --reference-image examples/sports_car/reference.png \
  --prompt-file examples/sports_car/prompt.txt \
  --output outputs/sports_car_evaluation

The environment is pinned tightly, and two licences sit on top of it

The install instructions are explicit about the platform, Ubuntu 22.04 with a CUDA 12.6 compatible NVIDIA driver, and they build a conda environment with Python 3.10, install a GPL build of ffmpeg from conda-forge, upgrade pip, install a pinned PyTorch stack from the CUDA 12.6 wheel index, install the requirements file, and then install Chromium through Playwright.

git clone https://github.com/code-video-model/CodeVideoModel.git
cd CodeVideoModel

conda create -n cvm python=3.10 -y
conda activate cvm
conda install -c conda-forge "ffmpeg[version='>=6',build='gpl_*']" -y
python -m pip install --upgrade pip
python -m pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 \
  --index-url https://download.pytorch.org/whl/cu126
python -m pip install -r requirements.txt
python -m playwright install chromium

The requirements file pins the same PyTorch versions and adds a diffusers install taken from a specific git commit rather than a release, along with accelerate, a peft version, transformers, safetensors, the Playwright version and a video pipeline dependency. The licences are layered, since the code is Apache-2.0 while the example assets carry a Creative Commons 4.0 licence in their own licences directory, and the model checkpoint and the hosted models have their own terms. The environment file is documented as being for reference only, with the programs reading process environment variables, and it warns never to commit a real key.

Editorial conclusion

Code Video Model earns a place for a team studying how structured code and a reference image steer a video model, because the intermediate artefacts are all saved and inspectable, the scene is editable Three.js rather than a black box latent, and the benchmark ships with prepared cases and a viewer. It is the wrong tool for anyone wanting a self contained pipeline, since the scene generation, the reference image and the evaluation all call hosted models through an API key, and the video model checkpoint is downloaded from a separate model repository. Verify three things before you plan around it: which hosted model you are willing to pay for, since the scene and image stages route through a router service and the evaluation uses a separate hosted model, that the hardware matches the stated environment of Ubuntu 22.04 with a CUDA 12.6 compatible driver and the pinned PyTorch wheels, and how the licences divide, since the code is Apache-2.0, the example assets carry a Creative Commons 4.0 licence, and the model checkpoint has its own terms. The last push was on 2026-10-13 and the repository publishes no GitHub releases.

Frequently asked questions

Official sources

1. code-video-model/CodeVideoModel on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README