Open3D-ML: 3D machine learning that ships inside the Open3D wheel

An extension of Open3D rather than a separate package

The positioning statement is short and load bearing: Open3D-ML is an extension of Open3D for 3D machine learning tasks, built on top of the Open3D core library and extending it with machine learning tools for 3D data processing. The repository narrows that scope in the next sentence, saying it focuses on applications such as semantic point cloud segmentation and provides pretrained models that can be applied to common tasks as well as pipelines for training.

The consequence for anyone installing it is that there is nothing to install from this repository alone. Open3D-ML is integrated in the Open3D v0.11 and newer Python distribution. The README's installation section starts by telling you to upgrade pip and install open3d, with no separate step for the machine learning part. What lives in this repository is the source of the extension, the configs, the model zoo, the dataset readers and the tests.

The framework story is dual by design. The README says Open3D-ML works with TensorFlow and PyTorch to integrate easily into existing projects, and also provides general functionality independent of ML frameworks such as data visualization. That independence is real and useful: reading a dataset and looking at it does not require choosing a framework first, and every example imports either `open3d.ml.torch` or `open3d.ml.tf` through the same variable name.

The repository topics describe the intended audience precisely: 3d-object-detection, 3d-perception, lidar, rgbd, semantic-segmentation, pretrained-models, pytorch, tensorflow, datasets and visualization.

Installing and verifying, including the Linux TensorFlow caveat

The compatibility list the README gives is specific about versions: PyTorch 2.0 and newer, TensorFlow 2.13 and newer on macOS with a Linux note attached, and CUDA 10.1 or 11 on GNU/Linux x86_64 as an optional extra.

# make sure you have the latest pip version
pip install --upgrade pip
# install open3d
pip install open3d

The framework itself is then installed from whichever requirements file matches your situation.

# To install a compatible version of TensorFlow
pip install -r requirements-tensorflow.txt
# To install a compatible version of PyTorch
pip install -r requirements-torch.txt
# To install a compatible version of PyTorch with CUDA on Linux
pip install -r requirements-torch-cuda.txt
# To install a compatible version of PyTorch with XPU (Intel GPU) on Linux
pip install -r requirements-torch-xpu.txt

Those four files appear in the repository root alongside a plain `requirements.txt`, so the split between base packages and framework-specific pins is a deliberate part of the layout.

Verification is a one liner, and it is the shape you want for a library with a native half.

# with PyTorch
$ python -c "import open3d.ml.torch as ml3d"
# or with TensorFlow
$ python -c "import open3d.ml.tf as ml3d"

The one genuine trap is spelled out in the README. From v0.18 onwards on Linux, the PyPI Open3D wheel no longer has native support for TensorFlow, because of build incompatibilities between PyTorch and TensorFlow. macOS is unaffected, which is why the version list notes TensorFlow on Linux separately. The documented workaround is to build the wheel from source inside Docker with TensorFlow support but PyTorch turned off, and the README gives the exact environment variables for that build.

Reading a dataset before you train anything

The first worked example is not a model. It constructs a SemanticKITTI dataset, pulls a split, prints attributes and a point cloud shape, and then visualizes a hundred frames. That ordering is a deliberate statement about how to work with this library: look at the data before you trust anything derived from it.

import open3d.ml.torch as ml3d  # or open3d.ml.tf as ml3d

# construct a dataset by specifying dataset_path
dataset = ml3d.datasets.SemanticKITTI(dataset_path='/path/to/SemanticKITTI/')

# get the 'all' split that combines training, validation and test set
all_split = dataset.get_split('all')

The split name `all` is one of the things worth noticing, because it means the dataset object composes training, validation and test into one view rather than requiring you to know which file holds which. Each split exposes `get_attr` and `get_data`, and the data for a sample is a dictionary with a `point` key, which is the shape you would expect to flow into a model.

The visualizer is the part that generalizes beyond this example, since the README positions framework-independent functionality as a first class feature rather than an afterthought. `ml3d.vis.Visualizer` with a `visualize_dataset` call and an index range is enough to page through a dataset in a window.

The topics list includes datasets and visualization for a reason, and the v0.19.0 changelog shows the visualizer getting real maintenance: a fix to `visualizer.py` for the case where no GUI is available, which is exactly the import problem you hit on a headless training box.

YAML configs drive model, dataset and pipeline together

The configuration model is the piece of this library most worth understanding, because it decouples what you run from what you wrote. Configs for models, datasets and pipelines live in `ml3d/configs`, and the README notes that users can construct their own YAML files to keep a record of customized configurations.

The loading example shows how the pieces get assembled.

import open3d.ml as _ml3d
import open3d.ml.torch as ml3d # or open3d.ml.tf as ml3d

framework = "torch" # or tf
cfg_file = "ml3d/configs/randlanet_semantickitti.yml"
cfg = _ml3d.utils.Config.load_from_file(cfg_file)

# fetch the classes by the name
Pipeline = _ml3d.utils.get_module("pipeline", cfg.pipeline.name, framework)
Model = _ml3d.utils.get_module("model", cfg.model.name, framework)
Dataset = _ml3d.utils.get_module("dataset", cfg.dataset.name)

Two design choices stand out. First, class lookup goes through a module resolver that takes the framework as an argument, which is what lets one config file describe both a PyTorch and a TensorFlow run. Second, the dataset and pipeline constructors are fed keyword arguments unpacked from the config, and the dataset path is handled specially: `cfg.dataset['dataset_path']` is set and then popped in the same expression, so a machine specific path never has to live in the committed YAML file.

That pattern shows up again in the semantic segmentation example, where a config is loaded, the model is constructed from `cfg.model`, the path is overwritten and popped, and the pipeline is assembled with a device argument. It reads as boilerplate, and it is, but it is boilerplate that keeps your hyperparameters in version control and your paths out of it.

Pretrained weights come from the model zoo

The semantic segmentation walkthrough builds a pipeline from a pretrained checkpoint rather than training anything, and the checkpoint is named after a timestamp.

model = ml3d.models.RandLANet(**cfg.model)
cfg.dataset['dataset_path'] = "/path/to/your/dataset"
dataset = ml3d.datasets.SemanticKITTI(cfg.dataset.pop('dataset_path', None), **cfg.dataset)
pipeline = ml3d.pipelines.SemanticSegmentation(model, dataset=dataset, device="gpu", **cfg.pipeline)

Inference itself is two calls. `pipeline.load_ckpt` restores the parameters, `pipeline.run_inference` on a single data sample returns a dictionary with `predict_labels` and `predict_scores`, and `pipeline.run_test` evaluates over the test split while writing logs to the `./logs` directory.

Weights are not fetched through a package manager. The example builds a Google Cloud Storage URL for a specific checkpoint file, downloads it into a logs folder it creates if missing, and then loads it. So the artifacts are plain files hosted on a bucket, and the model zoo page is the index of which ones exist. Anyone mirroring or air gapping this needs to handle that download step themselves.

RandLANet on SemanticKITTI is the example the README returns to repeatedly, across the dataset reader, the config loading, and the inference run, which makes it the natural first thing to run if you want one working path through the whole library.

What the releases show about maintenance and platform support

v0.20.0 was published on 2026-09-16, matching the last push to the repository. Its changelog is almost entirely dependency and platform work: an update to PyTorch 2.7, a fix for API changes in TensorFlow 2.20, updated PyTorch, torchvision and TensorFlow version pins, and Intel GPU support through XPU. That is the shape of maintenance for a library whose hard part is staying buildable against two fast moving frameworks, and the XPU requirement file that appeared alongside it explains why `requirements-torch-xpu.txt` now exists in the repository root.

v0.19.0, published on 2025-01-08, shows the other kind of work: adding the TUM-FACADE dataset to the list, Python 3.12 support, a fix for multiple preprocessing being applied during PyTorch segmentation, a per requirements file for the CXX11 ABI, a fix for the visualizer import when there is no GUI, and the addition of a SECURITY.md with restricted GitHub Actions permissions. The dataset addition and the preprocessing bug fix are the two most directly useful to a user.

The gap between those two release dates is worth noting honestly. The changelog file exists at the repository root, but the release history is not continuous, and the repository carries 134 open issues against 2,349 stars, which is a high ratio for a project of this age. The README's contribution section is a single short paragraph asking for help, and there is a `ci/` directory with a style checker wired into a Makefile target that can apply formatting automatically.

The `docs/`, `examples/`, `scripts/` and `tests/` directories complete the layout, and `model_zoo.md` sits at the root next to the README rather than buried in the docs, which tells you how central pretrained weights are to how this project expects to be used.

Editorial conclusion

Open3D-ML is in an unusual position for a research library: it is not a separate product you install alongside Open3D, it is part of the Open3D distribution from v0.11 onwards, and its examples read end to end against real datasets and real checkpoints. The pieces that make it usable are the YAML config files that describe model, dataset and pipeline together, the model zoo that supplies weights without training, and a visualizer that lets you inspect data before trusting a result. The main friction is the framework split, where TensorFlow on Linux no longer fits in the prebuilt wheel and Intel GPU support arrived only in v0.20. Start by reading a dataset with the visualizer, then run a pretrained pipeline before you attempt any training.

Frequently asked questions

Official sources

1. isl-org/Open3D-ML on GitHub
2. Issues
3. README
4. Releases