TorchUMM: one interface for fourteen multimodal models and three task types

One toolkit for three different jobs

TorchUMM is described as a unified framework for unified multimodal model inference, evaluation, and post-training, and its scope covers multimodal models together with task-specific models for image understanding, which it writes as Image to Text, image generation, Text to Image, and image editing.

That three-way split is the problem the toolkit exists to solve. The models in the supported list are built on different tokenisers, different conditioning mechanisms and different output conventions, so running one and another on the same benchmark normally means two codebases. The repository structure shows the answer: a backbones/ directory of model adapters, one per family, sitting above a common inference pipeline, a common evaluation runner and a common config layer.

The five key features are stated compactly. A pluggable model architecture, where unified multimodal backbones and opt-in single-task extensions live under one inference interface. Comprehensive evaluation, with 10 or more benchmarks covering generation, understanding and editing. Post-training support for SFT, IRG, recA, UniCot and Unigame. Cloud-native execution that scales to cloud GPUs via Modal, with details in modal/README.md. And config-driven behaviour, where all behaviour is controlled through YAML configs and no code changes are needed to switch models or benchmarks.

The package is published under the name umm with a version of 0.1.0, requires Python 3.10 or newer, and exposes a single command, umm, mapped to umm.cli.main:main. The licence is Apache-2.0, the homepage is a GitHub Pages site, and there is a paper on arXiv at 2604.10784 alongside post-training weights on Hugging Face.

Thirteen backbones, and the capability matrix is the useful part

The supported models table is the most information-dense page in the README, and its value is in the four columns rather than the names. Each row gives the model, its parameter count, and three capability flags for Understand, Generate and Edit.

Bagel is 7B and does all three. DeepGen is 5B and does Generate and Edit but not Understand. OmniGen2 is 7B with all three. Emu3 is 8B with Understand and Generate. Emu3.5 is 34B with all three, and it is the largest model in the list by a wide margin. MMaDA is 8B with Understand and Generate. Janus comes in at 1.3B, Janus-Pro at 1B and 7B, and JanusFlow at 1.3B, all Understand and Generate only. Show-o is 1.3B and Show-o2 is 1.5B and 7B. BLIP3-o is 4B, Generate only. TokenFlow has no parameter count listed. Ovis-U1 is 3B with all three.

Reading the matrix tells you something a list of names cannot. The 1B to 3B models are the ones that are Understand and Generate only, and the models that also Edit start at 3B and go up to 34B. If you are choosing a backbone, that is a constraint you would otherwise discover by trying.

There is also a separate section for single-task models, which is the opt-in extension path the features list refers to, and each model links to a per-model guide under docs/models/ with usage instructions, configuration examples and the benchmarks it supports. The single task section is truncated in the README available here, so the model/ directory in the repository layout is the better place to see the full picture, and it lists BLIP3-o and TokenFlow among the external model repositories, along with evaluation toolkits such as UEval, Uni-MMMU, WISE, geneval and Step1X-Edit.

The layout separates framework code from vendored model repositories

The repository structure is given as a tree, and it is worth reading in full because it shows the boundary the project draws between its own code and everything else:

umm_codebase/
├── src/umm/                    # Core framework
│   ├── backbones/              # Model adapters (Bagel, BLIP3-o, DeepGen, Emu3, Emu3.5, Janus, Janus-Pro, JanusFlow, MMaDA, OmniGen2, Show-o, Show-o2, TokenFlow)
│   ├── cli/                    # CLI entry points (infer, eval, train)
│   ├── core/                   # Config, registry, interfaces
│   ├── data/                   # Datasets, collators, transforms
│   ├── evaluation/             # Evaluation runners and metrics
│   ├── inference/              # Inference pipeline (batching, generation)
│   ├── models/                 # Model builders, heads, processors
│   ├── post_training/          # Post-training methods (SFT, IRG, recA, UniCot)
│   └── serving/                # Serving APIs
│
├── model/                      # External model repos & evaluation toolkits (submodules)
│   ├── Bagel/, BLIP3o/, deepgen/, Emu3/, Emu3.5/, MMaDA/, OmniGen2/, Show-o/, TokenFlow/
│   └── UEval/, Uni-MMMU/, WISE/, geneval/, Step1X-Edit/
│
├── configs/                    # YAML configurations
│   ├── inference/              # Per-model inference configs
│   ├── eval/                   # Benchmark evaluation configs (modal_*, amd_*, and local)
│   └── posttrain/              # Post-training configs
│
├── modal/                      # Modal cloud infrastructure (see modal/README.md)
├── docs/                       # Per-model usage documentation
├── eval/                       # Evaluation runner scripts
├── scripts/                    # Utility scripts
└── output/                     # Evaluation results

The model/ directory holds the external model repositories and the evaluation toolkits as submodules, which is a deliberate split. Your own code stays in src/umm and the third party code stays in model/, and the first is versioned with the toolkit while the second is pinned separately.

Inside src/umm the separation of concerns is clean. Core holds config, registry and interfaces, which is the extension point. backbones holds the adapters. models holds builders, heads and processors, which is where the model-specific parts of a transformer live. data holds datasets, collators and transforms. inference holds the pipeline with its batching and generation. evaluation holds runners and metrics. serving holds the serving APIs, and cli holds the three entry points: infer, eval and train.

The configs/ directory then splits into inference, eval and posttrain, and the eval configs are named by execution target, with modal_ and amd_ prefixes alongside local ones.

The CLI is the only entry point the manifest declares

The Python packaging is minimal, and it tells you how the project expects to be used. The build system is setuptools with a wheel, the project name is umm, the version is 0.1.0, the description is Unified multimodal model framework, and the dependencies list is empty.

An empty dependency list is worth pausing on. It does not mean the toolkit has no dependencies, because the model repositories in model/ carry their own, but it does mean the core package declares no third party requirements of its own, which is consistent with a research codebase that is always run from a checkout with the submodules present rather than installed into a clean environment.

The one script declared is umm, mapped to umm.cli.main:main, and the CLI entry points in the source tree are infer, eval and train. That trio is the whole workflow surface: run a model, score it, train it. The documentation sections mirror it, with Local Execution through the CLI, AMD HPC Execution, and the Python API as three ways of driving the same thing.

The AMD HPC mention is a specific deployment target rather than a generic cloud claim, and the eval configs follow the same pattern with separate modal_ and amd_ prefixed files. Running a benchmark on a cluster with an InfiniBand interconnect is a different performance problem from running it on a rented GPU, and having separate configs for each is the practical way to handle that.

Evaluation is a three-way split with editing as its own category

The evaluation results are organised by benchmark family, and the split tells you what the toolkit is trying to be a fair referee for.

Generation benchmarks and understanding benchmarks are the two obvious ones. Editing benchmarks are the third, with GEdit-Bench and ImgEdit-Bench named separately, which is right because editing is a different task shape: you are not producing a new image from a prompt, you are changing an existing one under instructions, and the metric has to measure instruction following rather than aesthetic quality.

There is also a Uni-MMMU Benchmark section, and the model/ tree lists UEval, Uni-MMMU, WISE and geneval as the evaluation toolkits pulled in as submodules, with Step1X-Edit there as well. So the benchmark implementations live outside the framework and are called from it, which is the same submodules-not-invented-here pattern as the model repositories.

Two other sections are about trust in the numbers. Post-Training Models covers the released checkpoints, and there is a Detailed Sub-scores section for the per-metric breakdown rather than a single aggregate figure. Reproducing Results is its own section, which for a research toolkit matters as much as the results themselves, since a result you cannot regenerate is not a result.

The news entry is short but telling: in July 2026 the project added MMStar evaluation benchmark support. Adding a benchmark to an existing framework is the cheap operation here, and the fact that it is the only news item suggests the model side is stable and the benchmark side is what moves.

Extending the framework is a documented three-way operation

The Extending TorchUMM section is the part that decides whether this is a library you can build on or a script you run.

There are three documented extension points: adding a new model, adding a new benchmark, and adding a new post-training method. Each maps onto a directory in src/umm. A new model is a new adapter in backbones/ plus whatever the models/ builders need, and the config layer picks it up. A new benchmark is a new evaluation config plus a runner in evaluation/. A new post-training method is a new entry in post_training/.

The post-training methods are named in the features list as SFT, IRG, recA, UniCot and Unigame, and the directory comment lists SFT, IRG, recA and UniCot, so the set is close to but not exactly the same as the list in the features section, which is a small inconsistency worth checking against the current tree.

The registry in core/ is the mechanism that makes this work, and the config-driven claim follows from it. If behaviour is entirely controlled through YAML, then a new model is registered once and then selected by name in a config file, with no code changes needed to switch models or benchmarks. That is the property that makes thirteen backbones and 10 plus benchmarks tractable, and it is also the property that makes the tool usable from a shell script on a cluster rather than only from a Python REPL.

The sections for Disclaimers and Citation come last, which is the order a research project normally uses, and the repository root has an mkdocs.yml, so the documentation site is built with mkdocs.

Editorial conclusion

Adopt TorchUMM if you are running multimodal research that has to compare understanding, generation and editing models on the same benchmarks, and you can live with the fact that this is a research codebase still at version 0.1.0. Do not adopt it as a production inference service, since the package version is 0.1.0, the dependencies list in the manifest is empty, and the documentation is organised around papers and benchmarks rather than operations. Before you start, read the docs for the specific model you care about under docs/models, and check whether the external model repositories in the model/ folder are present as submodules in your checkout, because the evaluation and generation benchmarks depend on them.

Frequently asked questions

Official sources

1. AIFrontierLab/TorchUMM on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README