4M and the any-to-any training recipe behind it

Masking is the common interface across modalities

4M is a framework for training any-to-any foundation models. The premise is that a model which has to treat images, text, bounding boxes, captions, depth, segmentation and more as first class citizens needs an interface that works for all of them, and the project picks tokenization and masking as that interface. Every modality is turned into tokens, and training becomes a masked prediction problem over a shared vocabulary.

That design choice is what produces the behaviour the project advertises. A model trained this way can perform a wide range of vision tasks, transfer to unseen tasks and modalities, and act as a steerable multimodal generative model. The published lineage is the 4M paper from NeurIPS 2023 as a Spotlight, followed by 4M-21 at NeurIPS 2024, and the repository ships code and pre-trained models for both.

The people behind it are EPFL's lab and Apple, and the project is presented as the official implementation. For an engineer the relevant question is not whether the idea is interesting but what the repository actually contains: training scripts, configuration files, tokenizer training code, a model zoo, and a small demo path for trying a checkpoint. The rest of this review is about that repository, since the code is what you have to run.

Two checkpoint families, seven or twenty-one modalities

The model zoo distinguishes two lines. 4M-7 covers seven modalities, 4M-21 covers twenty-one, and the naming in the configuration files follows the same pattern, with paths such as cfgs/default/4m/models/main/4m-b_mod7_500b.yaml and cfgs/default/4m/models/main/4m-b_mod21_500b.yaml. Every model listed carries a parameter count and a training data mixture, and the configurations are plain YAML files in the repository rather than something you have to reconstruct.

The sizes in the table are modest for a foundation model. 4M-B has 198M parameters and 4M-L has 705M. The data mixtures are CC12M, COYO700M and C4, with the twenty-one modality variant listed as trained on all three together. Checkpoints are published as safetensors files, which is the safer format for loading untrusted weights, and each entry in the table links both a direct checkpoint URL and a Hugging Face Hub repository under the EPFL-VILAB organisation.

There are also two specialised families in the zoo beyond the general models: 4M text-to-image specialist models and 4M super-resolution models. That is worth dwelling on, because those specialists are the honest alternative to the flagship. A specialist is trained for one output and will usually be a better fit for that output than a general any-to-any model, but it gives up the thing 4M is for, which is transfer to a task or modality the model never saw.

Installing means conda, an editable install and a CUDA check

The documented setup is a clone, a Python 3.9 conda environment and an editable install of the package:

git clone https://github.com/apple/ml-4m
cd ml-4m
conda create -n fourm python=3.9 -y
conda activate fourm
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

The pyproject.toml names the package fourm at version 1.0.0 and declares requires-python of at least 3.8, so the environment recipe is a tested choice rather than the only possible one. The next step is to confirm that PyTorch can see the GPU, since every model in the zoo is a large vision model:

import torch
print(torch.cuda.is_available())  # Should return True

If that prints False, the instruction is to reinstall PyTorch following the official PyTorch install instructions, and xFormers is called out as optional, needed for faster tokenizers, with the caveat that its CUDA version has to match. Notebook users can also expose the environment as a kernel:

pip install ipykernel
python -m ipykernel install --user --name fourm --display-name "4M (fourm)"

The demo sampler turns one image into every modality

The fastest way to see what the checkpoints do is the demo wrapper, and it is short enough to read in full. A Demo4MSampler is constructed with a Hugging Face model identifier, moved to the GPU, and then called with a dict whose key is the modality name and whose value is the input:

from fourm.demo_4M_sampler import Demo4MSampler, img_from_url
sampler = Demo4MSampler(fm='EPFL-VILAB/4M-21_XL').cuda()
img = img_from_url('https://storage.googleapis.com/four_m_site/images/demo_rgb.png')
preds = sampler({'rgb@224': img.cuda()}, seed=None)
sampler.plot_modalities(preds, save_path=None)

The comment on the img line says what the tensor is: a 1x3x224x224 ImageNet-standardised PyTorch tensor, so the resolution and the normalisation are part of the contract, not incidental. The output key uses the modality and resolution together, rgb@224, which hints at how the model handles inputs that are not 224 pixels.

Swapping the conditioning input turns the same sampler into a caption-to-all model, with a special token in the text:

preds = sampler({'caption': 'A lake house with a boat in front [S_1]'})

For anything beyond this, the repository splits the documentation by task: README_GENERATION.md for inference and generation, README_DATA.md for preparing aligned multimodal datasets, README_TOKENIZATION.md for training modality-specific tokenizers, README_TRAINING.md for training, and a generation notebook at notebooks/generation_4M-21.ipynb.

Training is a family of scripts, not one entry point

The repository root shows how wide the training surface is. run_training_4m.py is the main script, run_training_4m_fsdp.py adds a distributed variant, and three more cover specific objectives: run_training_divae.py, run_training_vqcontrolnet.py and run_training_vqvae.py. Two more utilities sit alongside them, save_vq_tokens.py for caching visual tokens and train_wordpiece_tokenizer.py for the text tokenizer.

The configuration lives in cfgs/, with the default tree under cfgs/default/4m/models/main/. This is a YAML-driven setup where a run is defined by pointing a script at a config path, which is the right shape for a research codebase: the same script serves the seven modality and twenty-one modality variants because the config carries the difference.

The training data path is the part with the largest practical cost. README_DATA.md covers preparing aligned multimodal datasets, and the dependency list includes webdataset, datasets, boto3, albumentations and wandb, which points at a pipeline that streams image-text corpora from object storage with augmentation and experiment tracking. Building an aligned corpus across twenty-one modalities is not a weekend project, and no dataset ships in the repository. The framework is open; the data work is yours.

Pinned dependencies are the real upgrade cost

Look at the dependency list and the age of the codebase shows. numpy is capped below 2.0.0, albumentations is pinned to exactly 1.4.0, diffusers to exactly 0.20.0, ftfy to 6.1.0, and huggingface_hub is constrained to at least 0.20.0 and at most 0.24.0. torch and torchvision come with a floor of 2.1.0 and 0.16.0 respectively.

These are not incidental. Exact pins on diffusers and albumentations usually mean an API the code calls was changed in a later release, and the numpy ceiling means the environment predates the numpy 2 migration, which in practice means a separate environment from anything else modern in your stack. Anyone trying to install this into an existing research image with a shared dependency set will hit conflicts, and the fix is a dedicated environment, which is exactly what the conda recipe creates.

There is one optional extra. The fast dependency group installs xformers, described in the README as being for faster tokenizers, and the installation notes warn that its CUDA version has to match the one you are using. A separate warning applies to the CUDA check: if torch.cuda.is_available() is False, the documented remedy is to reinstall PyTorch from the official instructions rather than debug the 4M code, because the environment is the first suspect in this setup.

Apache-2.0 for the code, a separate file for the weights

The repository root carries two licence files, LICENSE and LICENSE_WEIGHTS, next to ACKNOWLEDGEMENTS.md. The project is presented as Apache-2.0, and the pyproject metadata agrees. The presence of a separate weights licence is a detail that matters for anyone planning to ship a model: the code licence and the licence covering the published checkpoints are not the same instrument, so a team should read both before redistributing a fine-tuned 4M model in a product.

The repository is not archived. The last push was on 2026-09-11, and there are no GitHub releases at all, so there is no tagged version to pin and no changelog of published versions to consult. The version in pyproject.toml is 1.0.0. The practical consequence is that reproducibility depends on commit hashes rather than tags, which is a normal situation for a research repository that publishes model weights on the Hugging Face Hub rather than as release artifacts.

The other thing to know about the project is where the artefacts live. Pre-trained checkpoints and tokenizers are distributed as safetensors and can be loaded through the Hugging Face Hub, the website at 4m.epfl.ch collects the papers and models, and a demo space is published on the Hub under EPFL-VILAB. Everything the project promises is downloadable; what it does not provide is a packaged inference library, so adopting 4M for production means carrying the repository and its pinned environment yourself.

Editorial conclusion

Adopt 4M if your research goal is a single model that spans many modalities and transfers to unseen tasks, and if you are prepared to build your own aligned multimodal dataset. Do not adopt it as a production inference library, since the shipped experience is a research demo and a set of training scripts, and do not expect a smooth dependency upgrade, because numpy, albumentations, diffusers and huggingface_hub are all pinned. Verify three things first: that your hardware meets the CUDA requirement stated in the installation steps, that the licence situation is acceptable given the separate LICENSE_WEIGHTS file next to the Apache-2.0 LICENSE, and that the checkpoint you want actually exists in the model zoo for your modality count. The last push was on 2026-09-11 and the repository has no GitHub releases, so pin a commit rather than tracking main.

Frequently asked questions

Official sources

1. apple-aiml-research/ml-4m on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README