Matrix3D packs pose, depth and novel views into one masked model

Why one model for three photogrammetry tasks

Photogrammetry pipelines are usually assembled from separate specialised models. One network estimates camera poses, another predicts depth, a third renders novel views, and each brings its own training data, its own failure modes and its own checkpoint. Matrix3D, from Nanjing University, Apple and HKUST, takes the opposite position: a single model performs several photogrammetry subtasks including pose estimation, depth prediction and novel view synthesis. The repository described as the official implementation ships the model inference pipeline along with a modified 3D Gaussian Splatting pipeline for actual 3D reconstruction, and the project page is hosted at nju-3dv.github.io. The paper is on arxiv at 2502.07685. The interesting engineering is not the claim that one network can do three jobs, but the mechanism that lets a single set of weights switch between them: a masked inference scheme where the input states are described by indicator variables rather than by separate model heads. The repository also contains nerfacto_matrix3d and splatfacto_matrix3d directories, which is the usual sign that the authors intend the model to be driven from inside a nerfstudio pipeline rather than as a standalone script.

Masked inference: mod_flags and view_ids as the control surface

The mechanism that makes the unification work is masking. Instead of running different networks for different tasks, the diffusion process is steered by indicators named mod_flags and view_ids, which control the input states in the block the README points to as L48-L56. A modality flag selects which kind of input the model should treat as present, and the view identifiers tell it how many and which views to condition on. That is what allows dynamic compositions: the same weights can predict a novel view from two posed RGB images, or estimate poses, or predict depth from several posed images, without a separate checkpoint per job. The README is explicit that you must assign values to these indicators carefully, since a wrong assignment produces a wrong task rather than an error. There is also a hard ceiling: the model was trained with a maximum view number of 8, so view_ids must not exceed eight views. That limit is a design constraint baked into the training distribution, and it is the one piece of technical documentation in this repository that is both precise and easy to miss.

Building the pinned environment on Ubuntu 20.04

The project is tested on Ubuntu 20.04 with PyTorch 2.4 and Python 3.10, and the installation notes ask you to create a fresh conda environment and install matching versions. PyTorch, xformers and pytorch3d must agree with each other and with your CUDA version, and the notes take CUDA 11.8 as the worked example. The block below is the documented setup sequence: the PyTorch wheel comes from the cu118 index, pytorch3d comes from an extra index, requirements.txt pulls in the rest, and timm is pinned afterwards to resolve conflicts introduced by nerfstudio.

  conda create -y -n matrix3d python=3.10
  conda activate matrix3d
  # Here we take Pytorch 2.4 with cuda 11.8 as an example
  # If you install a different PyTorch version, please select a matched xformers/pytorch3d version
  pip install torch==2.4.0 torchvision==0.19.0 xformers==0.0.27.post2 --index-url https://download.pytorch.org/whl/cu118
  pip install --extra-index-url https://miropsota.github.io/torch_packages_builder pytorch3d==0.7.7+pt2.4.0cu118
  pip install -r requirements.txt
  # fixed the requirement conflicts from nerfstudio
  pip install timm==1.0.11

The README warns that some dependencies need CUDA matching the torch build in your system and that the installation may not work out of the box, pointing you at each dependency's own repository for troubleshooting. Two details deserve attention. The requirements file installs nerfstudio from a pinned git commit rather than from a release, and it pulls tiny-cuda-nn from its own repository as a git URL with the torch bindings subdirectory. That means a network fetch during setup, and it means your environment follows the commit recorded in the file rather than whatever is current on the branch.

Getting the weights into the checkpoints folder

The pretrained model is a single file, matrix3d_512.pt, hosted on an Apple CDN. You create a checkpoints folder and put the file into it. There is a second, optional weight: if you want single-view to 3D reconstruction, the project asks you to fetch isnet-general-use.pth from the DIS repository and place it in the same checkpoints folder. The IS-Net code is vendored in the repository under an IS_Net directory. Model weights and code are covered separately here. The repository carries both a LICENSE file and a MODEL_LICENSE file, and the licence field on the project is a custom one that GitHub cannot classify. In other words the code licence and the licence on the weights are two different documents, and anyone planning to ship a derivative model has to read both before assuming the permissions from one carry over to the other. The weights are not distributed through a package manager or a model hub, so there is no version tag and no checksum published with them.

Running the demo scripts on CO3Dv2

The examples are shell scripts, one per task, and all results land in a results folder by default. Novel view synthesis from a single input image is the first one to try.

    sh scripts/novel_view_synthesis.sh examples/co3dv2-samples/31_1359_4114

Pose estimation is a separate script, and it writes a png and an html file that show a visual comparison between predictions and ground truth values.

    sh scripts/pose_estimation.sh examples/co3dv2-samples/31_1359_4114

Depth prediction from several posed images is the third, and the back-projected ground truth and predicted point clouds are written into the folder.

    sh scripts/depth_prediction.sh examples/co3dv2-samples/31_1359_4114

The sample data under examples/co3dv2-samples and examples/unposed-samples comes from the CO3Dv2 and ARKitScenes datasets. Camera extrinsics in that data are stored as FOV values or in Blender camera coordinates, and the preprocessing code converts them into PyTorch3D cameras. The README names the exact place to change if you want a different camera convention, pointing at L654-659 in data/data_preprocessor.py and noting that the official pytorch3d.utils.cameras_from_opencv_projection conversion function can be swapped in for OpenCV cameras. That is a genuine invitation to adapt the pipeline rather than a note about a workaround.

Reconstruction paths: one image, or a few unposed frames

The 3DGS pipeline extends the same masked inference model into reconstruction. Single-view to 3D takes one image and runs a modified 3D Gaussian Splatting stage on the result.

    sh scripts/single_view_to_3d.sh single-view-to-3d examples/single-view/skull.png

The rendering output is written to results/single-view-to-3d/skull/3DGS-render-traj.mp4. Field of view defaults to 60 in this task, and you can override it by dropping a text file named after the image alongside it, which the data processor picks up automatically. The unposed few-shot route takes images with no known camera parameters and reconstructs from predicted poses.

    sh scripts/unposed_fewshot_to_3d_co3dv2.sh unposed-fewshot-to-3d examples/unposed-samples/co3dv2/31_1359_4114

Here the novel view trajectories are sampled on fitted splines derived from the predicted poses, and the design targets object centric scenes. An ARKitScenes variant exists for indoor scenes, differing only in how the splined cameras are generated, with the 3DGS stage unchanged. The README is candid about the cost of that difference: trajectory generation and reconstruction parameters need tuning per dataset to get higher performance. That is the pattern to expect for the whole project. The model supplies predictions, and the geometric quality of the result still depends on parameters you choose.

Where the intrinsics requirement bites

The single most consequential limitation for a new user is the camera intrinsics requirement, and it is stated in the pose estimation section rather than in a limitations section. The README strongly recommends providing the camera intrinsics saved in the txt files, because the model was trained with known camera intrinsics. Without them the processor falls back to a default fov of 60, and the documentation states plainly that performance may degrade. The fallback value can be changed by passing a default_fov argument. This is worth weighing before adoption, because intrinsics are the one input an unposed capture genuinely does not have, and the unposed few-shot path is exactly where a wrong field of view assumption propagates into the splined camera trajectories and then into the reconstruction. The single-view path has the same issue in a milder form, since it also defaults the field of view to 60. The README does not document a quality threshold below which the fallback becomes unusable, so anyone depending on it has to validate on their own data rather than rely on a stated boundary.

Maintenance state and the shape of the dependencies

The last push to the repository was on 2026-09-11, and the project is not archived. There are no GitHub releases, so versions are not tracked through tags and the upgrade story is a matter of tracking the default branch and the pinned nerfstudio commit inside requirements.txt. The dependency list is worth reading before you commit to it, because it reaches well beyond a single model: diffusers pinned at 0.31.0, accelerate, transformers, open3d, torchmetrics, deepspeed, opencv-python, numpy pinned at 1.26, plus tiny-cuda-nn built from source and nerfstudio from a specific commit. Both the tiny-cuda-nn binding and the nerfstudio pin mean an install touches two upstream repositories, so a working environment depends on those upstream sources staying available. On the licensing side there is no legal advice here, only the observation that a custom licence for the repository and a separate MODEL_LICENSE for the weights both need reading before use in a commercial pipeline, and that the two do not obviously grant the same permissions.

    sh scripts/unposed_fewshot_to_3d_arkitscenes.sh unposed-fewshot-to-3d examples/unposed-samples/arkitscenes/41069043

That variant swaps in splined camera generation suited to indoor scenes.

Editorial conclusion

Matrix3D suits research groups that already work in nerfstudio style pipelines, that can supply camera intrinsics for most captures, and that can assemble the pinned Ubuntu 20.04, PyTorch 2.4 and CUDA 11.8 environment without expecting a one line install. It is the wrong tool for a production photogrammetry service, for scenes with more than eight views, or for anyone unwilling to read the MODEL_LICENSE file that governs the pretrained weights, since the code licence and the weight licence are separate documents and only one of them is a standard open source licence. Before anything else, download matrix3d_512.pt into a checkpoints folder and run sh scripts/novel_view_synthesis.sh examples/co3dv2-samples/31_1359_4114, because that script exercises the whole inference path on data that already has ground truth poses. If that run produces the expected renders in the results folder, the environment is sound and the remaining risk sits in your own camera intrinsics rather than in the setup.

Frequently asked questions

Official sources

1. apple-aiml-research/ml-matrix3d on GitHub
2. Issues
3. Project website
4. README