nnDetection: self-configuring 3D medical object detection

The problem nnDetection automates: configuring detection per dataset

The framing in the README is clinical rather than technical. Simultaneous localisation and categorization of objects in medical images is of high clinical relevance because diagnostic decisions depend on rating objects rather than on pixels, and for that task the cumbersome and iterative process of method configuration is described as a major research bottleneck.

That bottleneck is the target. In a medical detection project, the same set of decisions has to be made again and again for every dataset: how the volume is cropped, how much context the network sees, how the loss is weighted, what normalisation is applied, how aggressively the data is augmented, how the anchor or the classification head is built. Each decision interacts with the others and with the dataset's own statistics, so tuning them by hand is slow and the tuning decisions rarely transfer.

nnDetection takes the route nnU-Net took for segmentation. The README says that nnU-Net has tackled the segmentation version of this challenge with success, and that following nnU-Net's agenda, the work systematises and automates the configuration process for medical object detection. The resulting method adapts itself without any manual intervention to arbitrary medical detection problems, and the paper behind it is Baumgartner, Jäger, Isensee and Maier-Hein from MICCAI 2021.

The evaluation story is stated as well: the method is demonstrated on two public benchmarks, ADAM and LUNA16, and ten further public data sets are proposed for a comprehensive evaluation of medical object detection methods. Those twelve data sets are what makes the framework assessable rather than merely clever, and the repository keeps a projects/ directory for exactly that kind of per-dataset setup.

Installing with Docker: one image, two mounted directories

The README calls the Docker route the easiest way to get started, with a provided Dockerfile. It asks you to install docker and nvidia-docker2 first, which tells you the container is GPU-oriented from the outset.

The build command from the base directory is:

docker build -t nndetection:0.1 --build-arg env_det_num_threads=6 --build-arg env_det_verbose=1 .

The two build arguments are optional and overwrite provided default parameters, and the tag naming scheme nnDetection:[version] is assumed by every project built on top of the framework, so the tag you choose here matters later.

Running it is a matter of two mounted directories:

docker run --gpus all -v ${det_data}:/opt/data -v ${det_models}:/opt/models -it --shm-size=24gb nndetection:0.1 /bin/bash

The container expects data in /opt/data and models in /opt/models, and the README suggests setting the environment variables det_data and det_models in the host system so the mounts point at the right places. The --shm-size=24gb flag is not incidental: the README warns that when running a training inside the container it is necessary to increase the shared memory, and links to the standard explanation of why Docker's default /dev/shm is too small for PyTorch data loaders with many workers.

The Dockerfile itself shows what the image contains. It starts from the NVIDIA PyTorch 21.11 container, sets the environment variables det_data, det_models, det_num_threads, det_verbose and OMP_NUM_THREADS=1, installs build tooling including cmake, ninja-build and build-essential, installs the requirements, and finishes with a forced CUDA install of the project itself with FORCE_CUDA=1 pip install -v -e .

Local installation: a pinned conda environment on Linux

The local path assumes conda and a Linux machine. The README notes that nnDetection was developed on Linux and that Windows is not supported, which is not a small restriction given the CUDA extensions in the build.

The environment is created with Python 3.8:

conda create --name nndet_venv python=3.8
conda activate nndet_venv

Then the toolchain and the deep learning stack are installed, with the compiler pointed at the conda toolchain so the C++ and CUDA extensions build against it:

git clone https://github.com/MIC-DKFZ/nnDetection.git
cd nnDetection

export CXX=$CONDA_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++
export CC=$CONDA_PREFIX/bin/x86_64-conda_cos6-linux-gnu-cc

conda install gxx_linux-64==9.3.0
conda install cuda -c nvidia/label/cuda-11.3.1
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch

The project is then installed in editable mode with the CUDA build forced on:

pip install -r requirements.txt  \
  && pip install hydra-core --upgrade --pre \
  && pip install git+https://github.com/mibaumgartner/pytorch_model_summary.git
FORCE_CUDA=1 pip install -v -e .

Two details in that sequence are deliberate. hydra-core is installed with --upgrade --pre, taking the pre-release of Hydra that the framework expects. And the final install forces the CUDA path with FORCE_CUDA=1, which the setup.py checks together with the availability of CUDA_HOME, so the compiled extension is built even when the GPU is not visible at install time, which is the normal situation in a container image build.

The source-level requirements are listed separately and are blunt about versions: nndetection requires Python 3.8+, and PyTorch 1.X should be used for now and not 2.0, with a minimum of PyTorch above 1.10 and a matching torchvision. CUDA above 10.1 and cudnn are required, with compatible versions selected deliberately, and TORCH_CUDA_ARCH_LIST may need setting depending on the GPU.

Environment variables that decide whether a run works

The variables deserve their own reading, because the README marks one of them as required and warns about it in unusually blunt terms.

det_data is required and is the path to the source directory where all the data will be located. det_models is required and is the path to the directory where all models will be saved. OMP_NUM_THREADS is also marked required, with the note that it needs to be set, otherwise bad things will happen, referring to the batchgenerators documentation. That is the sort of line that saves a day: an unspecified thread count interacts badly with the augmentation workers and produces failures that look like data corruption.

The recommended one is det_num_threads, the number of processes to use for augmentation, with a note that it should be at least 6 and defaults to 12. det_verbose is optional and can be used to deactivate progress bars, which are activated by default. MLFLOW_TRACKING_URI is optional and specifies the logging directory of mlflow, with a pointer to the mlflow documentation.

In the Docker image these are set as environment variables at build and run time, with det_data and det_models pointing at /opt/data and /opt/models. The same variables are also exposed as build arguments, which is how the Docker build lets you override the thread and verbosity defaults at image build time instead of at container start.

For performance, the README has a separate recommendation: CUDA 11.0 or later with cuDNN 8.1.X or later, and a locally compiled version of PyTorch 1.7 or later, which points at a real property of this framework. The heavy lifting is in custom C++ and CUDA kernels, not in Python, so the compiled extension has to be built well for the pipeline to hit the numbers the paper reports.

Registries and Hydra: how a pipeline is configured

The configuration model is the heart of the framework, and the README describes it in terms of registries.

nnDetection uses multiple Registries to keep track of different modules and to switch between them via the config files. The first registry named is the AUGMENTATION_REGISTRY, imported from the nndet.io.augmentation module path. The idea is that each stage of the pipeline, the augmentation policy, the network, the loss, the post-processing, is a component type that can be swapped by name in a configuration rather than hard-coded in a script.

The configuration files themselves live in nndet.conf and are composed with Hydra, which the README describes as a tool to dynamically configure and compose configurations. Hydra is what makes the self-configuring claim practical: a run is a set of composed configuration files, and overriding one value for one dataset does not require editing pipeline code.

The upstream project is nnU-Net, and the requirements pin it at version 1.7.1, which tells you the detection framework reuses the segmentation framework's data handling rather than reimplementing it. The rest of the requirements list the surrounding stack: pytorch_lightning at 1.3.1 to 1.4.2 for the training loop, batchgenerators for the data pipeline, torchmetrics for evaluation, hydra-core for configuration, mlflow for experiment tracking, SimpleITK below 2.1.0, dicom2nifti and python-gdcm below 3.0.26 for medical image formats, medpy for medical image processing, nevergrad for the automatic optimisation of the pipeline configuration, and loguru for logging.

The compiled side is handled in setup.py, which collects the C++ and CUDA sources from nndet/csrc, splits them into CPU and CUDA files, and builds them with torch's BuildExtension, choosing the CUDAExtension when CUDA is available or when FORCE_CUDA is set.

Verifying the install and where nnDetection stops being usable

The README has a hidden section for testing the installation, and the check it asks for is direct:

python -c "import torch; import nndet._C; import nndet"

The import of nndet._C is the one that matters, because that is the compiled C++ and CUDA extension, and the comment in the README says the command verifies that the compilation of the C++/CUDA code was successful. It also warns that the command must be run in the terminal and not in the PyTorch root folder, an odd but real failure mode. To test the whole installation, the README suggests running the Toy Data set example.

The limits are significant and mostly environmental. Linux only, with Windows explicitly unsupported. Python 3.8, which is past its end of life. PyTorch 1.x with an explicit instruction not to use 2.0, and a set of pinned dependencies such as pytorch_lightning below 1.4.2 and SimpleITK below 2.1.0 that mean a modern environment will conflict. A GPU is effectively required, since the installation forces the CUDA build.

On the project side, the release history is thin: v0.1-alpha in January 2023 and v0.1-alpha-zenodo in May 2023, with the last push to the repository on 2026-10-08 and the repository not archived. So the code is being touched, but there is no numbered release to pin, and the published artifacts are alpha tags.

The realistic alternative is to write the configuration by hand for your own dataset, which is what the framework automates away. If you need one dataset and can afford to tune it, a plain PyTorch Lightning script with your own preprocessing may be simpler than learning the registry and Hydra composition model. If you need twelve datasets, or you want a published baseline you can compare against, the automation is the entire value, and the environment constraints are the price.

Editorial conclusion

nnDetection earns its place for a research group that needs a working 3D detection baseline on a new dataset this week and would rather inherit an automated configuration than write one. It does not fit production inference on a modern PyTorch 2 stack, on Windows, or on hardware without a supported CUDA setup, and the pinned dependency versions make upgrading expensive. Verify first that your dataset is in one of the supported formats, that a GPU with a matching CUDA and cuDNN combination is available, and that OMP_NUM_THREADS is set to 1 in every environment that touches the code, because the project states plainly that other values cause failures.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. MIC-DKFZ/nnDetection on GitHub
4. README
5. Releases