TorchMultimodal: published multimodal models as PyTorch building blocks, not wrappers

Three layers: blocks, model classes, and example scripts

TorchMultimodal is described as a PyTorch library for training multimodal multi-task models at scale, covering both content understanding and generative models. The README structures it in three layers, and the layering is the design decision that matters.

The first is a repository of modular and composable building blocks: fusion layers, loss functions, datasets and utilities. The second is a collection of common multimodal model classes assembled from those blocks, with pretrained weights for canonical configurations. The third is a set of examples that combine the blocks with components and infrastructure from elsewhere in the PyTorch ecosystem, described as replicas of state-of-the-art published models and as baselines for ongoing research.

So a model class is not a black box. It is one arrangement of the same blocks you would assemble yourself, and the examples show the arrangement. That is what makes the library useful for research and what makes it awkward when you only want to call something.

The model list is a catalogue of architectures, not a ranking

The README enumerates the models with a class path and a paper link for each, and reading the paths tells you how the library is laid out.

Each architecture has its own module: torchmultimodal/models/albef/model.py, torchmultimodal/models/blip2/blip2.py, torchmultimodal/models/clip/model.py, torchmultimodal/models/coca/coca_model.py, torchmultimodal/models/flava/model.py, torchmultimodal/models/masked_auto_encoder/model.py for both MAE and Audio MAE, and torchmultimodal/models/mdetr/model.py. DALL-E 2 is the exception to the pattern: it lives under diffusion_labs rather than models, at torchmultimodal/diffusion_labs/models/dalle2/dalle2_decoder.py, and the entry is labelled a model rather than a model class.

That diffusion_labs placement is worth noticing. Generative architectures are not integrated the same way as understanding architectures, so if your interest is image generation rather than retrieval or classification, check which side of the repository you land in.

Papers are linked per model: ALBEF, CLIP, CoCa, DALL-E 2, FLAVA, MAE, Audio MAE and MDETR each have a reference, except BLIP-2, whose paper link in the README is empty.

A zero-shot FLAVA script shows the intended shape of use

The getting started section gives two minimal scripts. The first loads a pretrained FLAVA and does zero-shot classification against candidate labels.

from torchmultimodal.models.flava.model import flava_model
from torchmultimodal.transforms.bert_text_transform import BertTextTransform
from torchmultimodal.transforms.flava_transform import FLAVAImageTransform

image_transform = FLAVAImageTransform(is_train=False)
text_transform = BertTextTransform()
zero_shot_model = flava_model(pretrained=True)
img = Image.open("my_image.jpg")  # point to your own image
predict(zero_shot_model, img, ["dog", "cat", "house"])

Three conventions are visible here. The model factory is a function, flava_model, taking a pretrained flag rather than a checkpoint path you supply. Transforms are separate classes, with an image transform that has a training mode flag and a text transform with no configuration at all. And the prediction path calls encode_image and encode_text separately with projection=True, then scores with a matrix product of the two feature tensors.

The comment in the example points the path at your own image file, so the script is meant to be edited rather than run as shipped. The commented sample output shows a tensor of label probabilities and the winning label, which is the shape of result you get back rather than a hard coded expectation.

Example scripts are organised by task, not by model family

The examples table maps models to the tasks they can be trained or evaluated on, and the directories under examples/ show the same split: albef, cnn_lstm, common, flava, mdetr, mugen, omnivore and utils.

For ALBEF the supported tasks are retrieval and visual question answering. FLAVA covers pretraining, fine-tuning and zero-shot evaluation. MDETR covers phrase grounding and visual question answering, with a fine-tuning script referenced directly. MUGEN covers text-to-video retrieval and text-to-video generation. Omnivore covers pre-training and evaluating pretrained weights.

There is also a DDPM entry for training and inference, and it is a notebook under torchmultimodal/diffusion_labs rather than a script in examples/. A cnn_lstm directory sits alongside the multimodal architectures, which suggests the examples tree also carries a baseline for comparison.

The examples/common and examples/utils directories are shared infrastructure, which matters if you copy an example into your own repository: the pieces you would most want to reuse are separated from the parts that are specific to one run.

Packaging still runs through setup.py, and the version comes from a regex

The build metadata is old school in a way that has practical consequences. There is a setup.py at the root alongside pyproject.toml, and pyproject.toml contains only a usort configuration with first_party_detection set to false, which is a code formatter setting rather than a build system declaration.

The version is not written in the manifest. setup.py reads version.py from disk and extracts it with a regular expression:

version_regex = r"__version__ = ['\"]([^'\"]*)['\"]"

So version.py is the single source of truth, and version.py is a top level file in the repository alongside setup.py.

There is a nightly path too. The script parses a --package_name argument, and when the name contains nightly it substitutes a date based version built from the current year, month and day, formatted as year.month.day. That is how the torchmultimodal-nightly package referenced by the PyPI version badge gets its version, and it means the nightly version changes every day whether or not anything was committed.

The runtime requirements file is short and pins precisely where it matters: DALL-E==0.1, iopath, attrs==23.1.0, ftfy and regex. dev-requirements.txt is separate. Tooling is spread across .flake8, .coveragerc, mypy.ini, codecov.yml and a pre-commit config.

Where TorchMultimodal is the wrong choice

The title says beta release, and that is the first thing to weigh. There are no GitHub releases; the repository does include a CHANGELOG.md file, while the package version is read from version.py. The last push was on 2026-10-12.

The second boundary is architectural. These are model classes for research, with pretrained weights for canonical configurations. If your requirement is a multimodal inference endpoint for an application, you want a serving setup with batching, timeouts and a stable contract, and none of that is what this repository is.

The third is dependency risk. requirements.txt pins attrs to an exact version and DALL-E to 0.1, and a library whose transitive dependencies pin exact versions tends to collide with whatever else is in the environment. Combining it with a newer PyTorch stack or with other research code in the same virtualenv is where that shows up.

The positive case is specific: if you want to reproduce a published multimodal architecture's training setup and see how its components fit together, having the blocks and the assembled class in one repository, with the example scripts beside them, saves rebuilding that from papers.

Editorial conclusion

Adopt TorchMultimodal if you need to train or fine-tune a published multimodal architecture inside your own PyTorch code and want the checkpoint loading and transforms handled for you. Do not adopt it as an inference service, because the repository ships model classes and training examples rather than a server, and the README describes it as a beta release. Verify first which architectures you actually need to be available as classes, since BLIP-2 is listed in the models section without a paper reference, and check the pinned dependency versions in requirements.txt before assuming a current PyTorch environment will resolve cleanly.

Frequently asked questions

Official sources

1. facebookresearch/multimodal on GitHub
2. Issues
3. License: BSD-3-Clause
4. README