# EoMT turns a plain Vision Transformer into a segmentation model > EoMT is the Encoder-only Mask Transformer from TU Eindhoven, a CVPR 2025 Highlight that segments images by treating segmentation queries as tokens inside a Vision Transformer, with no adapters and no decoder. It suits researchers who want a fast, simple segmentation baseline built on pretrained backbones, and its released models cover panoptic, instance and semantic segmentation across COCO, ADE20K and Cityscapes. **tue-mps/eomt** — [CVPR 2025 Highlight] Official code and models for Encoder-only Mask Transformer (EoMT). - Repository: https://github.com/tue-mps/eomt - Website: https://www.tue-mps.org/eomt/ - Stars: 634 · Forks: 64 - Language: Jupyter Notebook - License: MIT - Published: 2026-10-21 · Updated: 2026-10-21 · Language: en - Canonical page: https://hysenlabs.com/projects/tue-mps-eomt ## Segmentation queries as tokens, with no decoder EoMT stands for Encoder-only Mask Transformer, and the idea is stated in one line in the README: a minimalist image segmentation model that repurposes a plain Vision Transformer to jointly encode image patches and segmentation queries as tokens. What is absent is the point. No adapters, no decoders, just the ViT. The README's summary is that architectural complexity is not necessary, and that for segmentation a plain Transformer is all you need, which is the claim behind the title, Your ViT is Secretly an Image Segmentation Model. The practical consequence is a different cost profile. Most segmentation architectures bolt a task-specific head onto a backbone, and the head is where the parameters and the engineering effort go. EoMT removes that layer by folding the queries into the encoder, so the model you train is close to the model you pretrained. The README states the result in terms of accuracy and speed, saying EoMT achieves accuracy similar to state-of-the-art methods that rely on complex, task-specific components, and is significantly faster thanks to its simplicity, for example up to 4 times faster with ViT-L. The work comes from Eindhoven University of Technology, the Polytechnic of Turin and RWTH Aachen University, with the paper published at CVPR 2025 as a Highlight, and the code and models are in this repository under the MIT licence. ## A conda environment, pinned dependencies, and a wandb login The installation path is the standard research setup. If Conda is not installed, Miniconda is fetched and run: ```bash wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh ``` Then the environment is created with a specific Python version and the dependencies installed: ```bash conda create -n eomt python==3.13.2 conda activate eomt python3 -m pip install -r requirements.txt ``` Weights and Biases is used for experiment logging and visualization, and enabling it means logging in with wandb login. That is worth noting before you start a long run, because the training script expects a logging backend. The requirements file is fully pinned, and the stack tells you what you are getting into: torch 2.7.0 with torchvision 0.22.0, lightning 2.5.1.post0 as the training framework, transformers 4.56.1, timm 1.0.15, torchmetrics 1.7.1, pycocotools 2.0.8 for the COCO evaluation, jsonargparse 4.38 for the configuration system, wandb 0.19.10, scipy, matplotlib, fvcore and an ipykernel for the notebook interface. There is also gitignore_parser, which suggests the data pipeline reads directory listings. The repository layout matches this. There is main.py as the entry point, a configs/ tree, models/, datasets/, training/, a model_zoo/ directory, docs/, and inference.ipynb for trying a checkpoint without training. ## Datasets are consumed as zip files, without unzipping The data preparation section has a detail that saves time: you do not need to unzip any of the downloaded files. You place them in a directory of your choice and provide that path through the --data.path argument, and the code reads the .zip files directly. For COCO the downloads are the usual archives: ```bash wget http://images.cocodataset.org/zips/train2017.zip wget http://images.cocodataset.org/zips/val2017.zip wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip wget http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip ``` ADE20K is described as needing one archive unpacked and repacked into a zip, because the instance annotations arrive as a tar: ```bash wget http://data.csail.mit.edu/places/ADEchallenge/ADEChallengeData2016.zip wget http://sceneparsing.csail.mit.edu/data/ChallengeData2017/annotations_instance.tar tar -xf annotations_instance.tar zip -r -0 annotations_instance.zip annotations_instance/ rm -rf annotations_instance.tar rm -rf annotations_instance ``` Cityscapes is the awkward one, since it is behind a login. The README gives a wget sequence that posts your credentials to the login form, saves the session cookies, and then uses those cookies with content-disposition to pull two packages, and asks you to replace the placeholder username and password. That credential handling is worth thinking about before you script it. A login form posted from the command line with your dataset account is fine for a one-off, but it is also a place where a mistake puts a password into a shell history file. ## Training a configuration, and the hardware it assumes Training EoMT from scratch means running main.py with a Lightning-style fit subcommand and a configuration file: ```bash python3 main.py fit \ -c configs/dinov2/coco/panoptic/eomt_large_640.yaml \ --trainer.devices 4 \ --data.batch_size 4 \ --data.path /path/to/dataset ``` The README explains what that specific configuration does: it trains the EoMT-L model with a 640 by 640 input size on COCO panoptic segmentation using 4 GPUs, with each GPU processing a batch of 4 images for a total batch size of 16. The guidance is to make sure the total batch size is devices times batch size, equal to 16 here. Switching to dinov3 in the configuration path selects the corresponding DINOv3 model. The compute note is the part to read before booking a cluster: this configuration takes about 6 hours on 4 NVIDIA H100 GPUs, each using about 26GB of VRAM. For fine-tuning a pretrained model, two flags are added: ```bash --model.ckpt_path /path/to/pytorch_model.bin \ --model.load_ckpt_class_head False ``` The second flag skips loading the classification head, and the README says why: because you are fine-tuning on a dataset with different classes, where keeping the old head would be wrong. This is a Lightning and jsonargparse setup, so every knob in the YAML is overridable on the command line, which is what makes the same script usable across the panoptic, instance and semantic tasks in the configs tree. ## Model zoo coverage across COCO, ADE20K and DINOv3 backbones The repository ships a model_zoo/ directory, and the DINOv3 model zoo has its own documentation file at model_zoo/dinov3.md listing the available EoMT configurations and performance benchmarks. The headline numbers for the DINOv3-based models are given in the README, and they are specific about resolution and model size. Panoptic segmentation reaches up to 58.9 PQ on COCO with EoMT-L at 1280 by 1280. Instance segmentation reaches up to 49.9 mAP on COCO with EoMT-L at the same resolution. Semantic segmentation reaches up to 59.5 mIoU on ADE20K with EoMT-L at 512 by 512. The repository also carries Papers with Code badges for state of the art results on COCO 2017 panoptic segmentation and on the BRAVO out-of-distribution benchmark. For deployment, there is a second distribution channel. EoMT with DINOv2 is available in Hugging Face Transformers, with the model documentation linked from the Transformers docs, and the model list is on the Hugging Face hub under the eomt library filter. The homepage for the project is at tue-mps.org/eomt/. One caution about the published numbers: they are the authors' own results, and the README presents them as headline figures for specific configurations. If you need an independent comparison, the leaderboards the badges point to are the place to look, since a repository cannot verify its own benchmarks. ## The same idea is now split across three repositories The README points to two follow-up projects, and the split tells you how the research direction has moved. VidEoMT, the CVPR 2026 paper, extends the EoMT philosophy to the temporal domain with an encoder-only video segmentation model that the README says is up to 10 times faster than competitors. It lives in a separate repository, tue-mps/videomt. PMT, Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders, is the latest model. The README describes it as reconciling the EoMT minimal philosophy with the need to preserve the features of frozen foundation models, by mimicking the last layers of EoMT and VidEoMT with a simple and fast decoder. It is in tue-mps/pmt. That PMT is described as having a decoder, while EoMT explicitly has none, is the interesting signal. The pure encoder-only formulation is being kept for the case where the backbone is fine-tuned, and a small decoder is reintroduced when the backbone is frozen and its features must be preserved. For someone adopting EoMT specifically, the practical consequence is that this repository is where the pure minimal design lives, and the interesting recent work sits next door. There are no GitHub releases in this repository, so the code is used from the branch, and the last push was on 2026-10-12. ## Conclusion Adopt EoMT if you want a segmentation model built on a pretrained Vision Transformer without the usual task-specific head machinery, since the design puts segmentation queries in the encoder and the repository ships checkpoints and configurations for panoptic, instance and semantic segmentation with both DINOv2 and DINOv3 backbones. Do not adopt it if you need video segmentation as a first-class feature in the same codebase, since the temporal version lives in a separate repository, or if you want a stable packaged library, because this is a research code release with no tagged versions. Before you start, check the compute the configuration demands, since the README states the reference training run takes about six hours on four H100 GPUs, confirm the dataset preparation steps for your target dataset, and note that weights are hosted on Hugging Face with the EoMT model also available in Transformers. ## FAQ ### What is EoMT and how does it do image segmentation? EoMT is the Encoder-only Mask Transformer, a minimalist segmentation model that repurposes a plain Vision Transformer to jointly encode image patches and segmentation queries as tokens. The README stresses what is not there: no adapters and no decoders, just the ViT. ### How do I install EoMT and its dependencies? Install Miniconda if needed, then create the environment with conda create -n eomt python==3.13.2, activate it, and run python3 -m pip install -r requirements.txt. The pinned requirements include torch 2.7.0, lightning 2.5.1.post0, transformers 4.56.1 and wandb 0.19.10, and you log in with wandb login to enable experiment logging. ### Do I need to unzip the dataset files for EoMT? No. The README says you place the downloaded files in a directory of your choice, pass that path with the --data.path argument, and the code reads the .zip files directly. The exception is the ADE20K instance annotations, which arrive as a tar and are repacked into a zip. ### What hardware does training an EoMT configuration need? The README states the reference configuration takes about 6 hours on 4 NVIDIA H100 GPUs, each using about 26GB of VRAM, and it trains EoMT-L at 640 by 640 on COCO panoptic segmentation with a total batch size of 16 across the 4 devices. ### Which backbones and tasks does the EoMT model zoo cover? The model zoo lists the available configurations, and the README reports DINOv3-based results for panoptic segmentation on COCO, instance segmentation on COCO and semantic segmentation on ADE20K with EoMT-L. EoMT with DINOv2 is also available in Hugging Face Transformers, and DINOv3 model configurations are documented in model_zoo/dinov3.md. ## Sources - [Issues](https://github.com/tue-mps/eomt/issues) - [License: MIT](https://github.com/tue-mps/eomt/blob/master/LICENSE) - [Project website](https://www.tue-mps.org/eomt/) - [README](https://github.com/tue-mps/eomt/blob/master/README.md) - [tue-mps/eomt on GitHub](https://github.com/tue-mps/eomt) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/tue-mps-eomt