MILo extracts a mesh at every training step so the Gaussians learn to be a surface

The idea, and what changes because of it

The paper title states the approach: MILo is Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction, published at SIGGRAPH Asia 2025 in the Journal Track, which is the Transactions on Graphics venue.

The abstract describes the mechanism precisely. The method introduces a differentiable mesh extraction framework that operates during the optimization of 3D Gaussian Splatting representations. At every training iteration, it differentiably extracts a mesh, including both vertex locations and connectivity, only from Gaussian parameters.

That last clause is the constraint that makes it interesting. The mesh is derived from the Gaussians alone, with no separate surface objective, no depth supervision and no mesh in the loop from the outside.

The consequence is gradient flow from the mesh to the Gaussians, which allows the framework to promote bidirectional consistency between the volumetric and surface representations. The abstract then states the payoff: this approach guides Gaussians toward configurations better suited for surface reconstruction, resulting in higher quality meshes with significantly fewer vertices.

And the stated benefit is not only quality. The framework can be plugged into any Gaussian splatting representation, increasing performance while generating an order of magnitude fewer mesh vertices. The practical argument at the end of the abstract is that this makes the reconstructions more practical for downstream applications like physics simulations and animation.

That last point is what distinguishes this from a splatting method that happens to output a mesh. A physics simulation needs a watertight-ish surface with few vertices and a sane triangle budget. A splat with a million primitives is cheap to render and useless to simulate, so anything that makes the reconstructed surface directly usable downstream is the point of the whole exercise.

The project page is at anttwo.github.io/milo, the paper is on arXiv at 2506.24096, and the authors are Antoine Guédon and Diego Gomez, who contributed equally, with Nissim Maruani, Bingchen Gong, George Drettakis and Maks Ovsjanikov. The affiliations are Ecole polytechnique and Inria, Universite Cote d'Azur.

Running a reconstruction from images

The quickstart starts from COLMAP data, because that is what the training script consumes.

Please start by creating or downloading a COLMAP dataset, such as the project's own COLMAP run for the Ignatius scene from the Tanks&Temples dataset, and you can move the Ignatius directory to ./milo/data.

Then, with MILo installed, you reconstruct a surface mesh from images by going to the ./milo/ directory and running:

# Training for an outdoor scene
python train.py -s ./data/Ignatius -m ./output/Ignatius --imp_metric outdoor --rasterizer radegs

# Saves mesh as PLY with vertex colors after training
python mesh_extract_sdf.py -s ./data/Ignatius -m ./output/Ignatius --rasterizer radegs

The two commands are the whole pipeline. Training produces the Gaussians, and mesh_extract_sdf.py then writes the mesh as a PLY with vertex colors. The flags are worth reading. The source directory and the model output directory are given with -s and -m, and --imp_metric selects outdoor or indoor, and the README asks you to change --imp_metric outdoor to --imp_metric indoor if your scene is indoor, which is the physical-scene assumption the method makes about lighting and importance sampling. The --rasterizer flag picks the rasteriser implementation, with radegs being the default here.

The README also explains what these commands give you. They use the lightest version of the approach, resulting in a small number of Gaussians and a light mesh.

Two optional flags trade compute for quality. You can increase the number of Gaussians by adding --dense_gaussians, and improve the robustness to exposure variations with --decoupled_appearance:

# Training with dense gaussians and better appearance model
python train.py -s ./data/Ignatius -m ./output/Ignatius --imp_metric outdoor --rasterizer radegs --dense_gaussians --decoupled_appearance

# Saves mesh as PLY with vertex colors after training
python mesh_extract_sdf.py -s ./data/Ignatius -m ./output/Ignatius --rasterizer radegs

The separation of the two scripts is worth noting, because the README points at further sections covering how to use other rasterizers, how to train MILo with high-resolution meshes, various mesh extraction methods, and how to easily integrate MILo's differentiable GS-to-mesh pipeline to your own GS project. That last item is the plug-in claim from the abstract, and the fact that it has its own documentation section suggests the integration path is a real supported workflow rather than an aspiration.

The Ignatius scene is a useful default precisely because it is a hard case, since it is an outdoor scene from Tanks and Temples with a lot of foliage and reflective water, which is where a volumetric representation and a surface representation disagree most.

Installation, and the CUDA constraint

The installation has four steps and one hard constraint.

Clone the repository recursively:

git clone https://github.com/Anttwo/MILo.git --recursive

The --recursive flag matters because the submodules are where the rasterisers live.

Then create the environment:

conda create -n milo python=3.9
conda activate milo

Python 3.9 in a conda environment, which is the older convention this kind of CUDA extension stack usually follows.

Then point the build at your CUDA installation:

# You can specify your own cuda path (depending on your CUDA version)
export CPATH=/usr/local/cuda-11.8/targets/x86_64-linux/include:$CPATH
export LD_LIBRARY_PATH=/usr/local/cuda-11.8/targets/x86_64-linux/lib:$LD_LIBRARY_PATH
export PATH=/usr/local/cuda-11.8/bin:$PATH

And finally run the install script:

python install.py

The README says this installs all dependencies, including PyTorch and Gaussian Splatting submodules.

The constraint is stated twice and should be taken literally. By default the environment will be installed for CUDA 11.8, and if you are using CUDA 12.1 you can provide the argument --cuda_version 12.1 to install.py. Please note that only CUDA 11.8 has been tested.

That is a real limitation for anyone on a current machine. The detailed instructions section offers the manual route, with conda install lines pinning pytorch==2.3.1, torchvision==0.18.1, torchaudio==2.3.1, pytorch-cuda at 11.8 or 12.1, and mkl=2023.1.0 from the pytorch and nvidia channels, followed by pip install -r requirements.txt and pip install submodules/diff-gaussian-rasterization_m.

The comment on that last line explains what the submodules carry: different rasterizers, aggressive densification, simplification, and utilities.

The Python dependencies themselves are a short, conventional list, and they tell you what the geometry side of the pipeline uses. open3d at 0.19.0 for point cloud and mesh processing, trimesh at 4.6.8 for mesh handling, scikit-image at 0.24.0, opencv-python at 4.11.0.86, plyfile at 1.1 for the PLY output and tqdm at 4.67.1.

Licensing, and what the project is still missing

The licence is the part to read carefully, because the project builds on other people's code in a way that does not reduce to a single identifier.

This project builds on existing open-source implementations of various projects cited in the Acknowledgements section. Specifically, it builds on the original implementation of 3D Gaussian Splatting; as a result, parts of this code are licensed under the Gaussian-Splatting License, see the ./LICENSE.md file. The codebase also builds on various other repositories such as Nvdiffrast, and the README asks you to refer to the license files of the submodules for more details.

So the top-level licence file is described as custom by the repository metadata, because the code carries the Gaussian Splatting licence terms for the parts derived from it, and the submodules carry their own. For an academic project that is a normal arrangement. For a company planning to ship a renderer built on this code, the submodule licences are the part that needs a lawyer rather than a skim.

The to-do list is short and tells you the state of the project better than a roadmap would.

Implement a simple training viewer using the GraphDeco viewer is still open. Add the mesh-based rendering evaluation scripts in ./milo/eval/mesh_nvs is still open. Add T&T evaluation scripts in ./milo/eval/tnt/ is done. Add Blender add-on, for mesh-based editing and animation, to the repo is done. As are Clean code and Basic refacto.

The two open items are both about evaluation and inspection rather than about the method. A training viewer is a convenience for watching the optimisation, and the mesh-based rendering evaluation scripts would let you measure the new view-dependent synthesis path against the volumetric one. The fact that the viewer is not done is a small but real friction point for anyone who wants to see what is happening during training rather than only the final result.

The Blender add-on being done is more interesting for the downstream applications the abstract names. An add-on for mesh-based editing and animation is the tool you need if the point of the reconstruction is to rig and animate a character, which is exactly the physics simulation and animation use case the paper motivates.

Where MILo fits, and where it stops

The comparison worth making is with the usual two-stage pipeline and with the plain splatting codebase.

Against the two-stage pipeline, where you train Gaussians and then run a separate surface extraction tool over the trained model, the difference is where the optimisation pressure comes from. In the two-stage version the mesh extraction is a post-process and the Gaussians are free to end up in whatever configuration minimises the photometric loss. With the mesh in the loop, the loss can pull the Gaussians toward configurations that produce a good surface, and the vertex count falls as a consequence rather than as an explicit target.

Against the plain splatting codebase, the difference is that MILo is a modification rather than a replacement. The framework can be plugged into any Gaussian splatting representation, and the README describes a section on how to easily integrate the differentiable GS-to-mesh pipeline to your own GS project. The training script therefore reads like a splatting script with additional flags, which is the right shape for research code that people want to try on their own variant.

The limits are concrete. This is GPU-only work: the whole stack is CUDA, with a PyTorch build pinned to 2.3.1 and a rasteriser extension compiled against a specific CUDA version. Only CUDA 11.8 has been tested, and the environment pins are old enough that a current driver and a current PyTorch will conflict with them, so adopting this means pinning your environment rather than tracking it. The method also needs a COLMAP reconstruction as input, so you need a photogrammetry pipeline before you can use it at all, and the indoor versus outdoor distinction has to be declared rather than detected.

And the mesh is not free. The extraction happens at every training iteration, and that is the mechanism of the method rather than an implementation detail, so the cost of the method is paid in training time.

On maintenance, the picture is that of a paper release. There are no GitHub releases, the last push was on 2026-04-03, and the repository is not archived. The to-do list has two items still open, so the natural expectation is that the code stays as it is for the published results rather than evolving into a maintained tool.

For a researcher reproducing the paper or building on the idea, the code is what it needs to be. For anyone expecting a maintained, general surface reconstruction pipeline from a splatting model, the framing in the abstract about downstream physics simulation and animation is the interesting part, and the Blender add-on is the piece that would make it practical.

Editorial conclusion

MILo suits a graphics researcher who needs an explicit mesh from a splatting reconstruction rather than a point cloud to be run through a separate surface extraction step, since the mesh participates in the optimisation and the vertex count falls accordingly, and since the extraction script is separate from the training script so the pipeline can be reused. It is the wrong tool if your goal is a pure splatting renderer with no mesh, because the extra extraction adds cost to every iteration, and it is the wrong tool without an NVIDIA GPU, since the whole stack is CUDA and the README states that only CUDA 11.8 has been tested. Before you start, clone with --recursive and run install.py, since the Gaussian Splatting submodules carry the rasterizers the training scripts depend on.

Frequently asked questions

Official sources

1. Anttwo/MILo on GitHub
2. Issues
3. Project website
4. README