Concerto: joint 2D-3D self-supervised pre-training for point clouds

What the repository is for, and what it is not

Concerto is the official project repository for the paper Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations, and the readme states the scope in its own first line. This repo provides joint 2D-3D self-supervised pre-trained Point Transformer V3 for 3D point cloud downstream tasks, modified from Sonata, the Meta Research project.

The key sentence is the second one. The repository is mainly used for providing pre-trained models, inference code and visualization demos. To reproduce the pre-training process of Concerto, the readme directs you to the Pointcept codebase instead.

That division is worth understanding before you clone anything. This is a distribution point for weights plus the code to use them, and a place to look at the results. The training pipeline, which is where the paper's actual contribution lives, is in a separate repository that is designed for the whole Pointcept family of models.

The homepage is pointcept.github.io/Concerto, the paper is on arXiv at 2510.23607, and the weights are on Hugging Face at Pointcept/Concerto. The project is Apache-2.0 licensed, with the setup.py carrying a Meta Platforms copyright header inherited from Sonata, which tells you how much of the underlying code is Meta's.

The repository listing is small: README.md, LICENSE, a concerto/ package, a demo/ directory, environment.yml and setup.py. There is no training code, no configuration files and no test suite in the tree, which is consistent with the stated purpose.

Two installation modes and what each one gives you

The installation section describes two ways in, and the choice is about what you plan to do with the code.

Standalone mode is recommended for users who want quick inference and visualization, and the environment is set up from a conda environment file that includes cuda and pytorch:

# Create and activate conda environment named as 'concerto'
# cuda: 12.4, pytorch: 2.5.0

# run `unset CUDA_PATH` if you have installed cuda in your local environment
conda env create -f environment.yml --verbose
conda activate concerto

The pinned versions matter here: CUDA 12.4 and PyTorch 2.5.0. The note about unset CUDA_PATH is for people who have a system CUDA installation that conflicts with the one the environment brings.

There is also a note about FlashAttention: it is installed by default but not necessary, and if it is not available in your local environment the readme says to check the Model section in Quick Start for a solution. That is the usual pattern for these research repositories, where the attention kernel is the first thing to break on an unusual CUDA setup.

Package mode is recommended for users who want to inject the model into their own codebase, and it requires CUDA and PyTorch to be present already:

# Ensure Cuda and Pytorch are already installed in your local environment

# CUDA_VERSION: cuda version of local environment (e.g., 124), check by running 'nvcc --version'
# TORCH_VERSION: torch version of local environment (e.g., 2.5.0), check by running 'python -c "import torch; print(torch.__version__)"
pip install spconv-cu${CUDA_VERSION}
pip install torch-scatter -f https://data.pyg.org/whl/torch-{TORCH_VERSION}+cu${CUDA_VERSION}.html
pip install git+https://github.com/Dao-AILab/flash-attention.git
pip install huggingface_hub timm 

# (optional, or directly copy the concerto folder to your project)
python setup.py install

Two things in there are worth flagging. The torch-scatter wheel is fetched from a URL keyed on the exact torch and CUDA version, so the two environment variables have to be right or the install fails with a version mismatch rather than a helpful message. And the flash-attention install goes to the GitHub repository rather than a release, which means it compiles from source.

The demo code needs a further set of packages, including open3d, fast_pytorch_kmeans, psutil, addict, scipy, camtools, natsort, opencv-python, trimesh, gradio and a pinned numpy:

pip install open3d fast_pytorch_kmeans psutil addict scipy camtools natsort opencv-python trimesh gradio numpy==1.26.4  # currently, open3d does not support numpy 2.x

That numpy pin is the kind of constraint that has bitten a lot of people: open3d does not support numpy 2.x, so the environment holds numpy back at 1.26.4 for the sake of one dependency. The setup.py declares the package as concerto with py_modules and find_packages excluding the demo directory, and version 1.0.

Running the demos and the three model sizes

The quick start begins with visualisation, and the demo folder mirrors Sonata's demos with additions.

Three model sizes are offered: concerto_small at 39M parameters, concerto_base at 108M and concerto_large at 208M. The visualisation scripts cover the similarity heatmap and the PCA projection, and there are separate variants for indoor and outdoor scenes:

# For visualization demo without color and normal, use python demo/xxx.py --wo_color --wo_normal
export PYTHONPATH=./
python demo/0_pca.py
python demo/0_pca_outdoor.py
python demo/1_similarity.py
python demo/2_sem_seg.py  # linear probed head on ScanNet

The demo files in the repository are 0_pca.py, 0_pca_outdoor.py, 1_similarity.py, 2_sem_seg.py, 3_batch_forward.py, 4_pca_video.py and visual_util.py. The sem_seg script is the interesting one for a downstream evaluation, because it runs a linear probe on ScanNet, which is the cheapest way to ask whether the learned representation transfers.

The --wo_color and --wo_normal flags matter for this particular paper. Because the pre-training is joint 2D and 3D, the interesting question is what the 3D model learned from the 2D image supervision, and running the demo without colour and normal input strips the point cloud down to geometry so you can see the spatial representation on its own.

The video demo is a separate matter because it needs VGGT to lift video into a point cloud first:

git clone https://github.com/facebookresearch/vggt.git
cd vggt
pip install -e .
cd ..
# Or you can copy the vggt folder in the vggt repository to ./demo
# Then you can run the demo
python demo/4_pca_video.py \
--input_video ${YOUR_VIDEO_PATH} \
--conf_thres 0 \
--prediction_mode "Depthmap and Camera Branch" \
--if_TSDF \
--pca_start 1 \
--pca_brightness 1.2 # the usage of these inpputs can be found in 'help'.

The Highlights section also records the release history. In October 2025 the pre-training code was released along with Pointcept and the inference demo and visualisation were provided here. In August 2026 a pretrained weight using SigLIP instead of DINOv2 was released on Hugging Face, which tells you the image branch of the joint training is itself configurable.

The data format and the transform pipeline

The instructions for running inference on custom data are the most useful part of the readme for anyone actually using the model.

Data is organised as a dictionary of numpy arrays. A single point cloud has coord, colour and normal arrays of shape N by 3, and an optional segment array of shape N. A batched version adds a batch array of shape N, and the readme points at the Pointcept documentation for the offset structure used for batched point clouds:

# single point cloud
point = {
  "coord": numpy.array,  # (N, 3)
  "color": numpy.array,  # (N, 3)
  "normal": numpy.array,  # (N, 3)
  "segment": numpy.array,  # (N,) optional
}

# batched point clouds

# check the data structure of batched point clouds from here:
# https://github.com/Pointcept/Pointcept#offset
point = {
  "coord": numpy.array,  # (N, 3)
  "color": numpy.array,  # (N, 3)
  "normal": numpy.array,  # (N, 3)
  "batch": numpy.array,  # (N,) optional
  "segment": numpy.array,  # (N,) optional
}

A sample point cloud can be loaded with a single call:

point = concerto.data.load("sample1")

The transform pipeline is shared with the Pointcept codebase and is constructed as a list of dictionaries passed to a Compose helper:

config = [
    dict(type="CenterShift", apply_z=True),
    dict(
        type="GridSample",
        grid_size=0.02,
        hash_type="fnv",
        mode="train",
        return_grid_coord=True,
        return_inverse=True,
    ),
    dict(type="NormalizeColor"),
    dict(type="ToTensor"),
    dict(
        type="Collect",
        keys=("coord", "grid_coord", "color", "inverse"),
        feat_keys=("coord", "color", "normal"),
    ),
]
transform = concerto.transform.Compose(config)

The steps are standard for the pointcept ecosystem and each does something specific. CenterShift recentres the cloud on the origin in x and y, with apply_z meaning z is included. GridSample voxelises at a grid size of 0.02 with fnv hashing, and the mode is train, which is worth noticing in an inference example. It returns the grid coordinates and the inverse mapping, and the inverse is what lets the model output be scattered back to the original point order. NormalizeColor scales the colour channel, ToTensor converts to tensors, and Collect gathers the fields the model actually consumes, with feat_keys naming which of them are features rather than coordinates.

Because the pipeline is a list of plain dictionaries, you can change it without touching Python code, and because the inverse mapping is preserved you can go back to the original points. The mode attribute on GridSample is the one to watch when you move from the demo to your own data, since the demo uses mode train while inference elsewhere may want a different setting.

Where Concerto is the wrong choice

The first limitation is that this is not the pre-training code. The readme is explicit that the repository is mainly for pre-trained models, inference code and visualisation demos, and that reproducing the pre-training process means going to the Pointcept codebase. If your research question is about how the joint 2D-3D objective works, whether it needs a particular data mix, or how sensitive the result is to the image branch, you need the training repository and this one will not answer those questions.

The second is the dependency on a specific stack. The conda environment pins CUDA 12.4 and PyTorch 2.5.0, the torch-scatter wheel has to match your exact torch and CUDA combination, flash-attention is built from a GitHub checkout, and numpy is held at 1.26.4 because open3d does not support numpy 2.x. Each of those is individually reasonable and together they mean the tool wants a particular environment rather than fitting into yours. On a shared cluster or a machine with a newer CUDA already working, that is real friction.

The third is that Concerto is a modification of Sonata rather than a separate implementation. That is a strength, in that you inherit a working codebase, and a limitation, in that the feature set you get is the one the authors needed. The readme lists what Concerto adds over Sonata, including visualisation without colour and normal input and the video demo lifted by VGGT, and the Aug 2026 SigLIP weight shows the image branch being swapped. What is not in that list is a general-purpose point cloud library.

The fourth is that the model is a backbone, not a task. The demo directory contains one downstream head, a linear probe on ScanNet, and the rest is visualisation. To use the representation for segmentation, detection or anything else you write the head yourself, and the repository gives you the pretrained weights and the transform pipeline to feed them.

The fifth is the release state. The repository publishes no GitHub releases, the last push was on 2026-08-18, and the setup.py carries version 1.0 for the package while the weights are tracked separately on Hugging Face. That split is normal for this kind of project, and it also means there is no release history to consult when a weight changes or when a dependency breaks.

Editorial conclusion

Concerto fits a point cloud researcher who wants to evaluate the joint 2D-3D self-supervised representation on their own downstream task, since three model sizes are published as weights on Hugging Face, the data format and transform pipeline are documented, and the visualisation demos show what the representation has learned before you run a single benchmark. It is the wrong choice if you want to reproduce the pre-training, because that lives in the Pointcept codebase, and it is the wrong choice if you need the full Sonata feature set, since Concerto is a modified version of it. Before downloading, check that your environment matches the conda environment, where CUDA 12.4 and PyTorch 2.5.0 are pinned, and note that the last push was on 2026-08-18 with the repository publishing no GitHub releases.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Pointcept/Concerto on GitHub
4. Project website
5. README