ShapeR turns casual captures into a metric 3D reconstruction

Metric shape, not a novel view

ShapeR is research code for a paper on conditional 3D shape generation from casual captures. The mechanism is worth reading closely, because the words in the description carry specific commitments.

The input is an image sequence. Preprocessing extracts, for each object, metric sparse SLAM points, images, poses and captions, using off-the-shelf methods rather than anything built for this project. A rectified flow transformer operating on VecSet latents then conditions on these multimodal inputs to generate a shape code, which is decoded into the object's mesh. Applying the model object-centrically to each detected object yields a metric reconstruction of the entire scene.

Two properties distinguish this from the more common approach of generating a 3D asset from a text prompt. The first is metric. The shape is reconstructed in real units rather than normalised into a canonical box, so it can be placed in the scene it was captured in. The second is that the conditioning is genuinely multimodal, with geometry, imagery, camera poses and language all feeding the same latent.

The caption input is the part that surprises people. The model is conditioned on text, which means a single reconstruction can be steered by describing the object, and it means the same weights handle a scene that contains many objects because each object is processed separately with its own caption.

The result is a mesh per object, and the README makes clear that this is a research artifact with a paper, a project page, a video and a model on Hugging Face, not a productised scanner.

Three presets trade views and steps for time

The inference entry point is a single script, infer_shape.py, and its interface is deliberately small. The README gives the shape of the command:

python infer_shape.py --input_pkl <sample.pkl> --config balance --output_dir output

The config flag selects a preset, and the three presets are the interesting part because they make the cost of the result explicit. The quality preset uses 16 views and 50 steps and is described as the best quality and the slowest. The balance preset uses 16 views and 25 steps and is the recommended one. The speed preset uses 4 views and 10 steps and is the fastest.

So the number of views sampled from the input sequence and the number of denoising steps are the two knobs, and a user chooses between them deliberately rather than getting a fixed cost.

The other arguments are the usual post-processing switches. The input path is a preprocessed pickle file, relative to the data directory. The output directory receives the meshes and the visualisations. A flag transforms the output mesh into world coordinates, which is what you want when the reconstruction should sit in the captured scene rather than in its own local frame. Two flags are on by default: remove_floating_geometry strips disconnected mesh components, and simplify_mesh reduces mesh complexity. A final flag, save_visualization, is off by default and produces the comparison image that shows the input, the prediction and the ground truth side by side.

The README's worked example runs on a single named sample:

python infer_shape.py --input_pkl ADT1292__stool.pkl --config balance

and the output is a .glb mesh plus, when the visualisation flag is passed, a file named VIS__<name>.jpg.

The pickle files are the real dependency

Here is the constraint that decides whether you can use this project at all. The README states that the codebase assumes a sequence has already been processed using the Aria MPS pipeline, along with a 3D object instance detector, resulting in the pickle files with the preprocessed data required for ShapeR ingestion.

So the pipeline upstream of ShapeR is substantial and is not part of this repository. Aria MPS is Meta's project Aria glasses pipeline, and the object instance detector is a separate model. ShapeR consumes their output rather than providing it.

To make the format usable, the project releases the ShapeR Evaluation Dataset, which contains preprocessed samples from Aria glasses captures. Each sample is a pickle file with point clouds, multi-view images, camera parameters, text captions and ground truth meshes. The ground truth meshes are what makes the dataset an evaluation set rather than just training data, and they are also what the visualisation flag compares against.

The explore_data.ipynb notebook is the guide to all of this. According to the README it includes the complete pickle file structure with all keys and their dimensions, interactive 3D visualisation of point clouds and meshes, camera position visualisation, image and mask grid displays, DataLoader usage examples for both the SLAM and RGB variants, and an explanation of the view selection strategies. The last item ties back to the config presets, since the view selection strategy determines which frames the model sees.

The dataset directory in the repository shows how the code is organised around this. shaper_dataset.py holds the dataset and dataloader, image_processor.py handles view selection and image preprocessing, and point_cloud.py converts a point cloud into a SparseTensor.

The model directory shows the pipeline as code

The repository layout describes the architecture directly, and it is more informative than the prose summary.

Under model, the flow_matching directory holds shaper_denoiser.py, the flow matching denoiser that the rectified flow transformer consists of. Next to it, vae3d/autoencoder.py is the 3D VAE responsible for mesh generation, which is the decoder that turns a shape code into a mesh. pointcloud_encoder.py is described as a sparse 3D convolution encoder, and dino_and_ray_feature_extractor.py handles image feature extraction, consistent with a model that conditions on both imagery and geometry.

The preprocessing directory contains a single helper.py for fisheye rectification and camera utilities, which is the fisheye correction step that Aria glasses captures require. The postprocessing directory has its own helper.py for mesh cleanup and visualisation, matching the two flags that are on by default at inference time.

Two structural details are worth noting. The checkpoints directory is described as holding model weights that are downloaded automatically, so the weights are fetched on first use rather than committed. And the model directory is split into a denoiser, a VAE, a point cloud encoder and an image feature extractor, which is the structure you would expect from a model that consumes several input modalities through separate encoders and combines them in the denoiser.

The top-level entries also include an experimental directory and a docs directory alongside the evaluation directory, whose EVAL.md holds the evaluation scripts, and the INSTALL.md and CONTRIBUTING.md files at the root.

The licence rules out commercial use for most of the code

The licence is the decisive fact for anyone outside academia. The repository states that the majority of ShapeR is licensed under CC-BY-NC, with details in the LICENSE file, and that portions of the project are available under separate licence terms, with the NOTICE file pointing at them.

CC-BY-NC means Creative Commons Attribution, NonCommercial. For a research paper's code that is a common choice, and it does not restrict use, modification or redistribution in a research setting. For a company that wants to put a capture-to-mesh pipeline into a product, it is a blocker, and the NOTICE file needs reading to see exactly which portions fall under which terms, because the split means the answer is not uniform across the tree.

The repository metadata carries a licence value of Other, with a note that it is a custom licence GitHub cannot classify, which is consistent with a repository that mixes two licence regimes in one tree and cannot express that in a single SPDX identifier.

Everything else about the project is standard research practice. There is a code of conduct, a contributing guide, an INSTALL.md, and the paper is cited through a BibTeX entry pointing at arXiv 2601.11514. The model weights are on Hugging Face and the evaluation dataset is published as a Hugging Face dataset, both under the facebook organisation.

For a company, the honest summary is that the pipeline is available to read and study, and that the licence terms for the parts you would want to reuse are not permissive.

A branch that has not moved since April

The repository is not archived, but the last push was on 2026-04-30, which is more than five months before the present date. There are no GitHub releases either, so there is no tagged version to pin to and no changelog to read.

For a research repository that is normal, and it comes with the usual trade-off. The code and the paper are available, the weights are on Hugging Face, and the evaluation dataset is published, so the result is reproducible in principle. What is not available is any guarantee that the code still matches the weights, or that a future change to the environment will not break the pipeline, since there is no release to anchor to.

The model checkpoint is downloaded automatically into the checkpoints directory, which means the weights move independently of the code. Anyone running this should confirm the checkpoint and the code revision correspond, and the README does not document a version pairing scheme.

The installation path runs through INSTALL.md, which the README points to for detailed environment setup instructions, and the evaluation scripts are documented separately in the evaluation directory. Those two files are where the operational detail lives, since the README deliberately stays at the level of inference usage and data format.

Taken together, the shape of the project is a well-documented research release with a strong licence restriction and no release engineering. That is a reasonable thing to build on in research and a poor thing to depend on in production.

Editorial conclusion

ShapeR suits a computer vision researcher who wants to study or reproduce conditional 3D shape generation on Aria captures, and who can accept a non-commercial licence and a preprocessing pipeline the project does not run for you. It is the wrong tool for a product team, because the majority of the code is licensed CC-BY-NC, and it is the wrong tool for a team without access to the Aria MPS pipeline and the object instance detector, since the input format assumes both. Before you adopt it, read the LICENSE file and the NOTICE file, because the project is split between the non-commercial terms for the bulk of the code and separate terms for other portions, and read INSTALL.md, since the environment setup lives there rather than in the README.

Frequently asked questions

Official sources

1. facebookresearch/ShapeR on GitHub
2. Issues
3. README