MVSplat turns sparse multi-view photos into Gaussian splats in one feed-forward pass

A feed-forward model, not a per-scene optimisation

MVSplat is a research project with a paper, and the paper title is the clearest description of the approach: Efficient 3D Gaussian Splatting from Sparse Multi-View Images. It was an ECCV 2024 oral, and the project page, the arXiv paper and a folder of pretrained models are all linked from the README.

The topics attached to the repository name the mechanism: cost-volume and feed-forward-gaussian-splatting. The word feed-forward is the one that changes what the software is for. A per-scene Gaussian splatting pipeline optimises a set of Gaussians to fit a particular scene, which takes minutes to hours on one GPU and produces a model that only describes that scene. A feed-forward model instead learns a mapping from images to Gaussians, so a new scene is handled by running the network once, with no optimisation loop per scene.

That is the entire research contribution claimed in the topic list, and it is why the repository is a training and evaluation codebase rather than a library with a friendly API. The software is the artefact that lets you reproduce the paper's numbers and train variants of it.

The author list is long and includes Marc Pollefeys, Andreas Geiger and Tat-Jen Cham, alongside Yuedong Chen, Haofei Xu, Chuanxia Zheng and Bohan Zhuang, and the project page lives at donydchen.github.io/mvsplat. The repository is MIT licensed, the primary language is Python, and the default branch is main.

Hydra config plus Lightning, with a shell script for the rest

The dependency list describes the stack, and it is a research stack rather than a packaged one.

pytorch_lightning
hydra-core
jaxtyping
beartype
wandb
timm
einops

pytorch_lightning is the trainer, and hydra-core is the configuration system, which is why every command in the README is a Hydra override on the command line rather than a set of flags. The pattern of a plus sign before an experiment name, as in +experiment=re10k, is Hydra's syntax for adding a new configuration group entry at runtime.

jaxtyping and beartype together give runtime shape checking on tensor operations, which in a codebase full of shape-sensitive view transformations catches a class of bug that would otherwise appear as a silent numerical wrong answer. timm is the model zoo, and the training procedure additionally downloads a UniMatch weight file into checkpoints. einops handles the tensor rearrangement, and wandb is the experiment tracker.

The rasterisation backend is not on PyPI. The requirements install a git URL directly.

git+https://github.com/dcharatan/diff-gaussian-rasterization-modified

This is the modified version of the CUDA rasteriser used for differentiable Gaussian rendering, and pinning a fork from a git repository rather than a released package means the build depends on a compatible CUDA toolchain at install time. It is also a maintenance consideration, since the code is tied to a specific fork rather than to a versioned release.

The remaining dependencies describe evaluation and visualisation: lpips for perceptual distance, scikit-image, opencv-python pinned at 4.6.0.66, moviepy and imageio for video output, sk-video, plyfile for writing point clouds, colorspacious, e3nn, svg.py, tabulate, matplotlib, tqdm, dacite, black and ruff. The repository top level shows the rest: config/ for the Hydra configuration tree, src/ for the code, assets/ for the tested evaluation indices, a more_commands.sh script, and two requirements files, one plain and one with versions.

Datasets arrive as chunks, and the conversion is not part of the project

The dataset section is where a new user will lose the most time, and the project is honest about where the responsibility lies.

MVSplat uses the same training datasets as pixelSplat, and the README quotes pixelSplat's instructions rather than writing its own. The reason is stated in the quote: pixelSplat was trained using versions of the RealEstate10k and ACID datasets that were split into roughly 100 MB chunks for use on server cluster file systems. Small subsets in that chunked format are linked for download, and to use them you unzip them into a newly created datasets folder in the project root.

If you would rather convert the original downloads, the README points at the scripts from dcharatan/real_estate_10k_tools, and the authors' own preprocessed versions are hosted separately. So there are three routes: a small subset in the chunked format, a conversion script maintained by another project, or the authors' hosted preprocessed data.

The DTU section is explicitly for testing only, and it is the one place where the repository ships its own tooling.

python src/scripts/convert_dtu.py --input_dir PATH_TO_DTU --output_dir datasets/dtu

There is also an optional step to generate the evaluation index.

python src/scripts/generate_dtu_evaluation_index.py --n_contexts=N

N is the number of context views, and the README notes that for N=2 and N=3 the tested versions are already provided under /assets. That last detail is a small kindness that saves real time, since the evaluation index defines which camera poses are sampled, and getting it wrong produces metrics that are not comparable with the paper.

The overall picture is that the dataset preparation is a distributed responsibility. The data itself comes from another project, the chunking convention comes from that project's cluster needs, and the conversion tools come from a third repository.

Evaluation and rendering are the same command with different overrides

The running instructions show a neat property of the Hydra setup, that rendering a video and computing metrics are the same command with different overrides.

Evaluation on RealEstate10K loads the checkpoint and switches the view sampler to the evaluation set.

python -m src.main +experiment=re10k \
checkpointing.load=checkpoints/re10k.ckpt \
mode=test \
dataset/view_sampler=evaluation \
test.compute_scores=true

The acid variant is identical except that it also points at an evaluation index file, dataset.view_sampler.index_path=assets/evaluation_index_acid.json.

To render a video instead of scoring, you swap the index for the video variant and flip two flags.

python -m src.main +experiment=re10k \
checkpointing.load=checkpoints/re10k.ckpt \
mode=test \
dataset/view_sampler=evaluation \
dataset/view_sampler.index_path=assets/evaluation_index_re10k_video.json \
test.save_video=true \
test.save_image=false \
test.compute_scores=false

The rendered novel views are written under outputs/test. Everything is an override, so the config tree in config/ is the real documentation of what the model does, and a full run is reproducible by naming the experiment and the checkpoint.

One thing to be aware of is the checkpointing.load path, checkpoints/re10k.ckpt. The pretrained models are distributed as a Google Drive folder and you save them into a checkpoints directory yourself, so the path is a convention you have to create, not something the code creates.

Training needs an A100 and a backbone you fetch yourself

The training section is short and has two parts, the download and the run.

wget 'https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmdepth-scale1-resumeflowthings-scannet-5d9d7964.pth' -P checkpoints
python -m src.main +experiment=re10k data_loader.train.batch_size=14

The UniMatch backbone weight is the MVSplat cost volume backbone, pretrained on the gmdepth scale1 resumeflowthings scannet checkpoint, and it has to be present in checkpoints before training starts. A training run that does not have it fails at the load step rather than at the loss step, which is at least a clear failure.

The hardware requirement is stated plainly. The models are trained with a single A100 GPU with 80GB of memory, and they can also be trained on multiple GPUs with smaller RAM by setting a smaller data_loader.train.batch_size per GPU. The default batch size in the example is 14, and the multi-GPU guidance is to lower it per GPU rather than to raise the total.

Multi-node training is supported through SLURM because the project is built on pytorch_lightning, and the README gives the job script directives and the srun invocation.

#SBATCH --nodes=2           # should match with trainer.num_nodes
#SBATCH --gres=gpu:2        # gpu per node
#SBATCH --ntasks-per-node=2

srun python -m src.main +experiment=re10k \
data_loader.train.batch_size=4 \
trainer.num_nodes=2

The comment about matching --nodes with trainer.num_nodes is a real requirement rather than a formality, since Lightning derives the world size from both. The optional exports, NCCL_DEBUG, HYDRA_FULL_ERROR, NCCL_SOCKET_IFNAME and NCCL_IB_GID_INDEX, are the usual suspects for multi-node NCCL communication problems, and the README links the relevant Lightning and NCCL issues for each.

Where MVSplat is the wrong tool

The first wrong case is a team without a large GPU. The models are trained on a single A100 with 80GB, the rasteriser is a custom CUDA fork, and the whole pipeline assumes a research cluster. Using the pretrained checkpoints for evaluation is a different proposition, since inference needs far less memory, but anyone who wants to train a variant on their own data needs that hardware class.

The second wrong case is a scene that does not match the training data. The models are trained on RealEstate10K and ACID, which are real estate and outdoor video datasets, and evaluated on DTU. A colleague's follow-up work points the same way, with DepthSplat described as covering more complex scenes and more input views. If your content is objects on a turntable, indoor rooms, or a scene type where the camera motion differs from handheld video, the model's assumptions may not hold and the cost of fine-tuning on a matching dataset is the real cost of the approach.

The third wrong case is a production requirement for a new scene at interactive rates with no training. Feed-forward inference is fast, but it still requires the right number of input views in the right configuration, and the output quality for a scene unlike the training distribution will be below what an optimised per-scene splat would give, because the optimisation fits the actual scene while the network generalises from what it has seen.

The fourth wrong case is expecting a maintained library. There are no GitHub releases, so there is no versioned artefact, and the last push to main was on 2026-05-06. The requirements install a rasteriser from a git fork, so a pip install of the requirements today is not guaranteed to be the environment the paper was produced in, and the scripts you would write against this code will need maintenance as the surrounding ecosystem moves.

How it differs from per-scene Gaussian splatting

The comparison worth making is with the original 3D Gaussian Splatting method, since MVSplat's claim is a change of regime rather than an incremental improvement.

In the optimisation approach, a set of anisotropic Gaussians is fitted to a set of posed images by gradient descent. The optimisation is what produces fidelity: every Gaussian ends up placed to explain actual observations, so the reconstruction is very accurate for that scene. The costs are time, one scene at a time, and a fresh optimisation for every new capture. A library that does this is a rendering and optimisation tool, and the interesting engineering is in the differentiable rasteriser.

MVSplat replaces the optimisation with a learned prior. The network takes a small number of input images and predicts the Gaussians directly, so producing a novel view synthesis for a new scene costs one forward pass. The training data is the price you pay, and the README makes that price visible by shipping the datasets and the training recipe rather than hiding them.

The difference in approach also shows in the evaluation protocol. Both approaches need the same evaluation indices and the same metrics, which is why the DTU index generation script exists, but the feed-forward model has to be evaluated across held-out scenes rather than on the scene it was fitted to, and that is a different statistical question from the optimisation approach, where the fit is always to the evaluation scene.

The third difference is where the work went. The optimisation method's contribution is in the representation and the rasteriser, and the codebase is mostly the rasteriser. MVSplat's contribution is in the architecture, the cost volume and the training recipe, so the codebase is mostly model code, a configuration tree and a training loop built on Lightning. Reading the src/ directory tells you which side of that line the paper is on.

Editorial conclusion

MVSplat fits a research team that needs novel view synthesis as a single forward pass over two or three input images rather than a per-scene optimisation loop, and it fits anyone who wants to train on their own data with the same pipeline the authors used. It does not fit a team without a large GPU, since the models are trained on a single A100 with 80GB of memory, and it does not fit someone expecting a turnkey product, since the dataset preparation, the UniMatch backbone download and the checkpoint selection are all manual steps. Before running it, confirm you can get the RealEstate10K and ACID chunks or the authors' preprocessed versions, download the backbone weight into checkpoints before training, and note that the repository has no GitHub releases and the last push was on 2026-05-06.

Frequently asked questions

Official sources

1. donydchen/mvsplat on GitHub
2. Issues
3. License: MIT
4. Project website
5. README