BiomedParse turns one prompt into a mask, a box and a label

Three tasks, one prompt

BiomedParse is described as a foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities, published in Nature Methods, and the repository hosts the code and resources for it.

The design point is the consolidation. BiomedParse is designed for comprehensive biomedical image analysis and offers a unified approach to perform segmentation, detection and recognition across diverse biomedical imaging modalities. By consolidating these tasks, BiomedParse provides an efficient and flexible tool tailored for researchers and practitioners, facilitating the interpretation and analysis of complex biomedical data.

In practice that means you describe what you want in text and get back a mask, a box and a label rather than running three separate models and reconciling their outputs. For a workflow where a radiologist wants a specific organ segmented, bounded and named, that is a real reduction in plumbing.

The v2 architecture is built on BoltzFormer, and the notice at the top of the repository says what v2 brings: improved code and model architecture using BoltzFormer, and support for end-to-end 3D inference. There is a v1 branch for the original version.

The project is Python, Apache-2.0 licensed, with the last push on 2026-09-09. There are no GitHub releases, so the branches and the HuggingFace weights are the distribution route.

The v1 versus v2 question has a blunt answer

The documentation asks the question directly, Should I use v1 or v2, and then answers it in the short answer field. Use the released v2 weights for 3D modalities and v1 for the rest. For 2D training with the v2 existence classifier, see the 2D guide. Dedicated pretrained 2D weights will be released in the future.

The comparison table fills in the detail. Version 1 is 2D, covering CT, MRI, Ultrasound, X-Ray, Pathology, Endoscopy, Dermoscopy, Fundus and OCT, with 100 or more tasks, and its existence detection is a post-inference K-S test. Version 2 is 3D, covering CT, MRI, Ultrasound, PET and 3D Microscopy including EM and lightsheet, with 200 or more tasks, and its existence detection is a built-in ISD module.

So the split is not arbitrary. v1 is the 2D model with nine modalities and a statistical test applied after inference to check whether the thing you were told to find actually exists. v2 is the 3D model, and the modality list changes accordingly.

The built-in existence detection is the substantive difference. The v1 approach is described in the news as post-inference K-S test, and the v2 approach as built-in object existence detection for false positives, with no separate mask checking required.

That matters because a segmentation model will happily produce a plausible-looking mask for a structure that is not there, and a clinical or research pipeline that does not check will treat that as a finding. Baking the check into the model changes the failure mode from a silent error to a reported absence.

The v2 gains, and the Detectron2 removal

The news section records what changed between the paper and v2, and the most recent entries are about usability rather than capability.

On September 9, 2026, Detectron2 is no longer required for BiomedParse v2. The entry says this removes a recurring installation and deployment blocker while preserving compatibility with the released checkpoint and its predictions. For a research model that others have to install, removing a heavyweight dependency is one of the most valuable changes you can make, because the installation is where most attempts die.

The same day, the v2 architecture was tailored for 2D segmentation with built-in existence classification, with 2D inference and training and fine-tuning on your own segmentation data, including automatically generated negative image-prompt pairs. The negative pairs are the interesting part: a classifier for existence is only as good as its negatives, and generating them automatically from a segmentation dataset is a practical way to get them.

The four v2 bullet points in the release section are the broader picture. Larger pretraining data at million scale covering 200 or more anatomies across different modalities. Improved segmentation performance for small objects using the BoltzFormer architecture. SOTA 3D segmentation performance supporting end-to-end volumetric inference, with a link to the CVPR challenge. And the built-in existence detection.

The challenge result is also dated. On June 11, 2025, BiomedParse was number one in the CVPR 2025 Foundation Models for Text-guided 3D Biomedical Image Segmentation Challenge, after upgrading the model and finetuning on the challenge dataset with wider and more comprehensive coverage for 3D biomedical imaging data. The challenge dataset is on HuggingFace, and the model is available as a containerised docker image for direct inference.

Installation is a conda environment and a requirements file

The install path is documented in full and it is worth reading because the dependency notes are where a research model usually goes wrong.

The clone is of the v2 branch:

git clone --branch v2 https://github.com/microsoft/BiomedParse.git
cd BiomedParse

Then the environment:

conda create -n biomedparse_v2 python=3.10.14
conda activate biomedparse_v2

And the inference dependencies, with the explicit note that no Detectron2, Azure ML or DeepSpeed is required:

python -m pip install -r assets/requirements/inference.txt

The PyTorch version is pinned and the documentation is clear that you install the matching wheels for your hardware first if you need a different CUDA or CPU build. The validated versions are PyTorch 2.6.0 and torchvision 0.21.0.

For Azure ML fine-tuning and evaluation, the additional training stack comes from a second requirements file, plus two named packages:

python -m pip install -r assets/requirements/requirements.txt
python -m pip install azureml-automl-core opencv-python

The training requirements target CUDA 12.4, and the 2D training configuration at configs/finetune_biomedparse_2D.yaml reuses the existing training pipeline, with no separate framework required.

The repository layout backs this up. There is an assets/ directory holding the requirements files and the README assets, a configs/ directory, src/ for the model code, inference.py and inference_2d.py as separate entry points, process_2D.py, inference_utils/, biomedparse_datasets/, utils.py and a FINETUNING.md at the root, alongside a set of inference notebooks: inference_example_3D.ipynb and the earlier RGB, NIFTI and DICOM examples.

The weights come from HuggingFace, and there are two ways to get them

The weights are trained on the CVPR 2025 challenge dataset, and the documentation asks you to acknowledge the original challenge if you use that version of the model. It also refers to the original dataset for necessary image preprocessing.

Option one is the HuggingFace Hub, and the first step is the package:

pip install huggingface_hub

Then the download:

from huggingface_hub import hf_hub_download

# Download the checkpoint file
file_path = hf_hub_download(
    repo_id="microsoft/BiomedParse",
    filename="biomedparse_v2.ckpt"
)

print("Model weights downloaded to:", file_path)

Option two is a direct download with `wget` or `curl`, which the documentation describes as also possible.

The model is on the HuggingFace Hub as microsoft/BiomedParse, and the data is at microsoft/BiomedParseData on the Hub as well, described as extra datasets for finetuning uploaded in December 2024, the same month that the loading process of target_dist.json was optimised by automatic downloading from HuggingFace and that a random rotation feature was added for training.

The choice between the two download routes is mostly about whether your environment already has the Hub client. The Python route is better when you want the model cached in the standard Hub location, because the `hf_hub_download` call handles the cache and the resume. The command line route is better for an air-gapped setup where you are copying a file into place by hand.

One thing to be aware of is the size of the model, which is why the containerised image is offered for direct inference rather than asking everyone to assemble their own stack.

What the release history shows about how this project is maintained

The news section reads as a maintenance log, and it is worth reading as one because the pattern is instructive.

The entries start on November 18, 2024, when BiomedParse went officially online in Nature Methods, and they run through to September 2026. In between there are entries for adding inference notebook examples for RGB, NIFTI and DICOM, a negative prediction p-value example, uploaded extra datasets for finetuning, refined object recognition scripts with an example notebook, and the v2 release being complete in October 2025 with full support for inference and finetuning.

So the project has been actively maintained for about two years, with the gap between the v1 paper and the v2 model being worked on publicly rather than silently.

Two details in the older entries show how the tooling has improved. The December 2024 optimisation of target_dist.json loading to download automatically from HuggingFace is a small quality-of-life fix for the class files, and the random rotation feature added for training is an augmentation that directly addresses overfitting on small medical datasets, where you often have tens of volumes rather than thousands.

The repository also carries a SECURITY.md and a LICENSE, and the default branch is v2 with the v1 code still available on the main branch for anyone who needs the original model. That is a good state of affairs for a research model: the newer version is the default and the older one is not deleted.

The paper, the demo, the model and the data are all linked from the top of the README, and the citation block is a BibTeX entry, so if you use it in a publication the linking is already done for you.

Editorial conclusion

Adopt BiomedParse if you need to segment, detect and classify biomedical structures in one pass from a text prompt, and you are working with the modalities its released weights cover. Skip it if you need 2D weights from the v2 architecture today, because dedicated pretrained 2D weights are still promised for the future, and if you want a stable released version, because v2 sits on a branch with no GitHub releases. Verify first which version suits your data, since the documented rule is v2 weights for 3D modalities and v1 for the rest, and that your hardware matches the validated PyTorch 2.6.0 and torchvision 0.21.0 builds.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. microsoft/BiomedParse on GitHub
4. Project website
5. README