# GLaMM: a multimodal model that puts segmentation masks in the sentence

> GLaMM is a large multimodal model from the Mohamed bin Zayed University of AI group, published at CVPR 2024, that generates natural language answers with object segmentation masks attached to the phrases. The repository ships training and evaluation code, pretrained checkpoints, the GranD dataset and an annotation pipeline, which makes it a research codebase to run on a workstation rather than a library to depend on.

**mbzuai-oryx/groundingLMM** — [CVPR 2024 🔥] Grounding Large Multimodal Model (GLaMM), the first-of-its-kind model capable of generating natural language responses that are seamlessly integrated with object segmentation masks.

- Repository: https://github.com/mbzuai-oryx/groundingLMM
- Website: https://grounding-anything.com
- Stars: 969 · Forks: 56
- Language: Python
- License: Apache-2.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/mbzuai-oryx-groundinglmm

## The task the paper invents: Grounded Conversation Generation

GLaMM is an end-to-end trained large multimodal model whose contribution is a task as much as a model. The README describes it as providing visual grounding with the flexibility to process both image and region inputs, which enables a unified task of Grounded Conversation Generation combining phrase grounding, referring expression segmentation, and vision-language conversations.

That combination is the point. Phrase grounding means linking a phrase in a sentence to a place in an image. Referring expression segmentation means turning a phrase like the red car on the left into a pixel mask. Vision-language conversation is the ordinary image question answering capability. Grounded Conversation Generation asks for all three in one output, so a caption like a man in a red shirt next to a parked bicycle arrives with a mask attached to the man and another attached to the bicycle, and the phrases themselves live in the same sentence as the answer.

The value of that is for anything downstream of the model. A caption without masks has to be post-processed by a separate referring expression segmenter, which is an extra model, an extra pass and an extra source of error. Producing the mask and the phrase together means the alignment is part of the generation rather than a guess made afterwards.

The model accepts textual and optional visual prompts, at the image level and for a region of interest, which is what gives it the multiple granularity levels the README describes. The same conversation can be about the whole image, about a region the user points at, or about a phrase the user has already chosen.

The paper was released on arXiv in November 2023 and accepted to CVPR 2024, and the work comes from Mohamed bin Zayed University of AI together with Australian National University, Aalto University, Carnegie Mellon University, University of California Merced, Linköping University and Google Research.

## The repository layout and what each directory is for

The top level of the repository is the model, the training entry points and the data tooling, and reading the file names tells you most of what you need to know.

There are two training scripts, train.py and train_ft.py, the first for training and the second for fine-tuning, which matches the two-stage story the project tells with the pretraining annotations from GranD and the smaller GranD-f set for the fine-tuning stage. The model code is in the model directory, the evaluation code in eval, the dataset handling in dataset, and there is a GranD directory at the top level for the dataset itself. The tools directory holds the annotation and preprocessing utilities, and scripts holds the shell level entry points.

The mmdet directory is the one that tells you about the segmentation side. It is a vendored copy of an object detection framework, which is how the masks are produced for the automated annotation pipeline and how grounding outputs are evaluated. The app.py file is the demo application, and it is a Gradio app, which the dependency list confirms with gradio and gradio-client pinned in the requirements.

The docs directory is the real documentation and the README does not attempt to summarise it. It holds install.md for setting up the conda environment, datasets.md for downloading and arranging the data, GranD.md for the dataset and the annotation pipeline, model_zoo.md with links to all the pretrained checkpoints, training.md for the training instructions across the three capabilities, evaluation.md for the evaluation procedures, and offline_demo.md for running the demo locally. There is also a separate project site, a dataset site at grounding-anything.com, and an online demo served through an ngrok URL.

For a project of this size, that split is the reason it is usable at all. A research repository usually mixes the model, the data and the experiment scripts, and the result is that reproducing anything means reading the code. Here the pieces are separated and documented.

## GranD and GranD-f: the data is the larger contribution

The dataset behind GLaMM is called GranD, the Grounding-anything Dataset, and the README describes it as a large-scale densely annotated dataset with an automated annotation pipeline for detailed region-level understanding and segmentation masks. The scale it claims is 7.5M unique concepts anchored in a total of 810M regions, each with a segmentation mask.

Those two numbers describe a different kind of annotation from the usual referring expression data. Instead of a handful of expressions per image, every concept that appears in the image is grounded to a region, and the region carries a mask. That is what makes it usable as a pretraining corpus for a model that has to localise arbitrary phrases, and it is why the project needed an automated pipeline rather than human annotation at that scale.

The pipeline is documented in docs/GranD.md, which the README links from the March 2024 update entry announcing the release of the GranD dataset together with the automated annotation pipeline. A separate file describes preparing the pretraining annotations from GranD, so the two steps, using the pipeline and turning its output into pretraining annotations, are documented separately.

For the Grounded Conversation Generation task specifically, the fine-tuning data is GranD-f, which the README describes as about 214K image-grounded text pairs for higher-quality data in the fine-tuning stage. The gap between 810M grounded regions and 214K image-grounded text pairs is the gap between the pretraining corpus and the supervised task data, and the naming reflects that: GranD is the grounding corpus, GranD-f is the fine-tuning set built for the conversational output.

The data is published on the project site, and the model zoo document lists the downloadable pretrained checkpoints. That combination, data plus weights plus training code, is what distinguishes this repository from a paper page with a demo link.

## Training, evaluation and the three capabilities the docs cover

The training guide covers GLaMM for its various capabilities, and the README names three: Grounded Conversation Generation, region-level captioning, and referring expression segmentation. The evaluation guide covers the same three using pretrained checkpoints, reporting results as described in the paper.

That three-way split reflects how the model is put together. Referring expression segmentation is the region-level output, a mask for a phrase. Region-level captioning is the reverse direction, describing a region. Grounded Conversation Generation is the combination, producing an answer at image level while tying phrases to masks. A project that only needs one of the three can still use the released checkpoints, since the model zoo document lists the pretrained weights for each capability.

The evaluation protocol is described in the README as a contribution in its own right, a comprehensive evaluation protocol for the new task. That is not unusual for a paper that introduces a task, because there is no prior benchmark to compare against, so the authors have to define what counts as a correct grounded answer. The evaluation code in the eval directory, together with the pinned pycocotools and pycocoevalcap entries in the requirements, indicates the caption side of the evaluation leans on the standard COCO caption tooling, while the grounding side leans on the mmdetection code.

A later entry in the update log, from November 2024, records that VideoGLaMM has been released, extending the grounded conversation generation task to videos. That is a separate project with its own page, and it tells you something about the group's direction, but the GLaMM repository itself is still image focused.

## The dependency set is a late 2023 snapshot

The requirements.txt file in the repository root is worth reading closely, because it is the environment and it is pinned exactly. The pins are consistent with a project finalised around the end of 2023: gradio at 3.35.2, diffusers at 0.22.3, deepspeed at 0.12.5, accelerate at 0.25.0, peft at 0.5.0, numpy at 1.24.2, pillow at 9.4.0, huggingface-hub at 0.20.1, opencv-python at 4.8.0.74, transformers-era tooling around the same period, and a handful of evaluation packages including pycocotools and pycocoevalcap.

A representative slice of that file looks like this:

```python
deepspeed==0.12.5
diffusers==0.22.3
flash-attn==2.3.6
gradio==3.35.2
numpy==1.24.2
peft==0.5.0
```

Two of those deserve a warning. flash-attn at 2.3.6 is a compiled extension that needs a matching CUDA toolchain and a compiler, and it is the dependency most likely to fail on a machine that is not the one the pins were written for. deepspeed is present because the training scripts use it for distributed training, which means the dependency is not optional in practice even if you only want to run evaluation.

The practical consequence is that you are building a specific environment rather than installing a package. The installation guide is explicit that it provides a conda environment for training, evaluation and the demo, and the pinned versions mean the code was written against those exact library behaviours, so upgrading the stack is a compatibility project rather than a routine bump. Anyone planning to run this should set the environment up as documented first, and only then consider modernising the pins.

The demo is the lightest path in, since it needs the pretrained checkpoints and a Gradio server rather than a training cluster, and the offline demo guide exists for exactly that.

## Why this is a research codebase and not a component you can depend on

The first limitation is structural. There is no PyPI package here, no published GitHub release, and the repository root has no setup.py or pyproject.toml. The dependencies are pinned in a requirements file, the model weights are downloaded from the model zoo links, and the model code is imported from the model directory inside the repository. That is the shape of a research artefact, and it means there is no supported upgrade path and no compatibility promise between the code and any published version.

The second is the hardware. Training GLaMM is a distributed deep learning job, with deepspeed in the requirements, and the fine-tuning stage works over 214K image-grounded text pairs derived from a 810M region corpus. The pretraining stage is a different order of magnitude again. A team without GPUs, or with a single consumer card, is realistically limited to running the demo and the evaluation scripts with released checkpoints.

The third is the data pipeline dependency. The masks in GranD come from an automated annotation pipeline that depends on the mmdetection code vendored in the repository. Reproducing the annotations means running that pipeline, and the guide says so, but it also means the pipeline's output depends on the detector's behaviour, which is a moving target as detection models are replaced.

The fourth is the demo. The online demo is served through an ngrok URL, and the README points readers to it directly, so the model is reachable from a public endpoint in a way that a private system would not be. Running the offline demo locally, as the offline_demo.md guide describes, avoids putting your own machine's inference behind a tunnel.

None of this is criticism of the work. It is what an academic release looks like, and for the purpose of building on the research it is exactly right. It is a poor fit for a product that needs a supported, versioned dependency.

## Licence, provenance and the projects that followed

The licence is Apache-2.0, which is permissive, allows commercial use and modification, and includes an explicit patent grant. The LICENSE file sits at the repository root. This describes the published terms rather than legal advice.

The provenance is unusually broad for a research release, and it is worth knowing who is behind the code. The author list runs from the Mohamed bin Zayed University of AI group through Australian National University, Aalto University, Carnegie Mellon University, University of California Merced and Linköping University to Google Research, with the paper's first authors listed as Hanoona Rasheed and Muhammad Maaz. The dataset and the annotation pipeline are published as separate downloadable artefacts, and the project maintains a dataset site, a project page and an online demo.

The update log gives the timeline. The paper went out in November 2023, the online demo in the same month, and the training and evaluation code, the pretrained checkpoints and the GranD-f dataset followed on 2023-12-27. Acceptance at CVPR 2024 came in February 2024, and the GranD dataset with its automated annotation pipeline in March 2024. VideoGLaMM, the extension to video, was released in November 2024.

The repository is not archived and the last push was on 2026-09-05, so the code is still being touched. What the push dates do not tell you is whether the pinned environment still installs, and that is the question to answer first. The paper is on arXiv at 2311.03356, and the model zoo document in docs is the place to find the checkpoint that matches the capability you actually intend to use.

## Conclusion

Adopt GLaMM when your work is grounded vision-language research, when you need a segmentation dataset at a scale you cannot annotate by hand, or when you want a reference implementation of Grounded Conversation Generation to compare against. It is the wrong tool for a product integration, since the repository is a research codebase with pinned late-2023 dependencies, no published releases and no package on PyPI, and it is the wrong tool for a machine without a CUDA GPU. Before starting, read docs/install.md, create a conda environment matching the pinned requirements.txt, and confirm that your driver and CUDA version can build flash-attn 2.3.6, which is the dependency most likely to fail first.

## FAQ

### What is GLaMM and what task does it introduce?

GLaMM is an end-to-end trained large multimodal model that generates natural language responses with object segmentation masks attached to the phrases. The task it introduces is Grounded Conversation Generation, which combines phrase grounding, referring expression segmentation and vision-language conversation in one output.

### What is the GranD dataset and how large is it?

GranD, the Grounding-anything Dataset, is a large-scale densely annotated dataset with an automated annotation pipeline, comprising 7.5M unique concepts grounded in a total of 810M regions, each with a segmentation mask. The GranD-f subset for the Grounded Conversation Generation task holds about 214K image-grounded text pairs for fine-tuning.

### How do I set up the environment to run GLaMM?

The installation guide in docs/install.md sets up a conda environment for training, evaluation and the demo, and the repository pins every dependency exactly in requirements.txt, including gradio, diffusers, deepspeed, peft and flash-attn. The flash-attn pin in particular needs a matching CUDA toolchain to build.

### Where do I get the GLaMM checkpoints and how do I run the demo?

The model zoo document in docs/model_zoo.md lists the downloadable pretrained checkpoints, and the offline demo guide describes running a local Gradio demo. There is also an online demo linked from the README, served through an ngrok URL.

### What licence is GLaMM released under?

The repository is licensed Apache-2.0, which permits commercial use and modification, requires preservation of the notice and licence text, and includes an explicit patent grant. The LICENSE file at the repository root carries the terms.

## Sources

- [Issues](https://github.com/mbzuai-oryx/groundingLMM/issues)
- [License: Apache-2.0](https://github.com/mbzuai-oryx/groundingLMM/blob/main/LICENSE)
- [mbzuai-oryx/groundingLMM on GitHub](https://github.com/mbzuai-oryx/groundingLMM)
- [Project website](https://grounding-anything.com)
- [README](https://github.com/mbzuai-oryx/groundingLMM/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/mbzuai-oryx-groundinglmm