lmms-engine: one training engine for vision-language, diffusion and language models

A small version number and a wide model list

lmms-engine describes itself as a simple, unified multimodal models training engine, lean, flexible and built for hacking at scale, and the project metadata repeats the phrasing with the addition of hacking and hacking at scale.

The version tells you where it stands. The manifest is at 0.1.4, and the releases listed are 0.1.2 in October 2025, described as the first official release with new models and feature support, 0.1.3 in January 2026 with more model and parallelism coverage, and 0.1.4 in August 2026. The last push was on 2026-10-18.

What the project covers is the interesting part. The model support section says twenty or more architectures spanning vision-language, diffusion and language models, and the examples directory backs that up with directories for a language model, several vision-language models, the omni variants that handle image, audio and text, a mixture-of-experts vision-language model, a video generation model, a diffusion language model, a representation autoencoder, an interpolant transformer, a model autoencoder family, a lightweight audio-language model, and the LLaVA family in two versions.

So the ambition is a single harness for training text models, vision-language models and video models on the same cluster, with the same launcher, the same configuration format and the same parallelisation options. That is what makes the optimisation table in the README worth reading closely, because the table is where the project admits that the models are not interchangeable.

The project is Apache-2.0 licensed, the documentation is on readthedocs, and the requirements are Python 3.11 or newer and PyTorch 2.0 or newer.

Installation: uv, an editable checkout, or a Docker image with the stack prebuilt

The installation section gives you a choice of routes, and the ordering tells you which the author considers primary.

The source route is a clone and an editable install with the extras:

git clone https://github.com/EvolvingLMMs-Lab/lmms-engine.git
cd lmms-engine
uv pip install -e ".[all]"

There is a plain editable install for a lighter setup, and an install of the published stable release straight from the package index. The dependency sync is done with uv, and for Linux there is a helper script that auto-detects the platform:

bash uv_sync_linux.sh

The README says that script is the recommended path on Linux, with plain sync, and finally a pip install of the requirements file, offered as the fallback when the sync fails. Two optional performance extras are listed separately, because both need a compiler and are the two things most likely to fail on a new machine:

uv pip install flash-attn --no-build-isolation
uv pip install liger-kernel

The Docker route is the one to look at if you do not want to assemble the stack yourself. There are published images with pre-built PyTorch, CUDA, FlashAttention and training dependencies, and the run command mounts your working directory into the container and drops you into a shell on the tagged image.

The reason to prefer the container is visible in the dependency list of the project file. DeepSpeed, bitsandbytes, Triton, the PyTorch stack, PEFT for parameter-efficient finetuning, wandb for experiment tracking, hydra-core and omegaconf for configuration, librosa and decord for audio and video data, and the two Qwen helper packages. That is a large surface to assemble by hand, and a container turns it into one download.

Three launch paths, one configuration argument

Training is launched through a module with a single configuration argument, and the README offers three ways to run it, which is a useful sign of how the project thinks about distributed training.

The recommended one is torchrun, the native PyTorch launcher, with the process count, the node count, the rank, the address and the port given on the command line, followed by the module and the configuration file.

The alternative is Accelerate with the fully sharded data parallel option, which is the path if your cluster is already set up for it.

And for a single GPU, the launcher is invoked with the interpreter directly:

python -m lmms_engine.launch.cli config_yaml=examples/qwen3_vl/example_config.yaml

That last form is the one to remember, because the same module and the same argument work in all three cases. The configuration is a YAML file, and the hydra-core and omegaconf dependencies in the project file are there to resolve it, which means a run configuration can be composed from command line overrides without editing the file.

The practical consequence is that moving from one GPU to eight is a change of launcher rather than a change of script, and that the example configuration in the examples directory is the thing you copy. Every example directory has a run script, and the README says each of them contains the detailed setup instructions, the prerequisites and the configuration options, which makes the examples the real documentation for a project whose README is a table.

The optimisation matrix is where the project is honest

The featured examples table has one column per model and one row per optimisation technique, and the tick and cross marks are the most informative thing in the README.

The techniques are named in a legend. FSDP2 is fully sharded data parallel version two, for distributed training. TP is tensor parallelism, sharding model compute across GPUs. USP is Ulysses sequence parallel, for long contexts. Muon is an advanced optimiser using Newton-Schulz orthogonalisation. Liger is Triton fused kernels, covering cross entropy, RMSNorm, RoPE and SwiGLU, and the README claims a thirty per cent memory reduction for it. Packing is first-fit bin packing, which the README says can peak at thirty-five to forty per cent model FLOPs utilisation against twenty to twenty-five per cent without it, in the case of Qwen2.5-VL finetuning. NSA is native sparse attention for long-context work. And EP is expert parallelism, for mixture-of-experts models, sharding the experts across GPUs.

The marks are uneven, and that is the point. Most of the models have FSDP2 and packing. Tensor parallelism appears for exactly one listed model, Qwen3-VL. Expert parallelism appears only on the mixture-of-experts models, and only on some of them. Liger is available for some architectures and not others, and the video generation and diffusion models have the sparsest coverage of all.

There is a companion document for this. The announcements section mentions an efficiency report from October 2025 with model FLOPs utilisation metrics across architectures for various architectures and training configurations, and the README points at a reference page with the detailed benchmarks. So the claims are backed by numbers in the repository rather than left as adjectives.

For an adopter, this table is the single most valuable thing in the README, because it tells you in one glance which of the techniques you were planning to use will work with the model you want to train.

Twenty-odd architectures, one dependency stack, and a 0.1 version

A few closing observations about what it means to build on this.

The dependency list in the project file is the honest measure of what a run costs. Datasets and a transfer helper for the model weights, the transformers library pinned to a range, accelerate, PEFT, deepspeed, bitsandbytes, Triton, wandb for tracking, torchdata, a compiled attention kernel and the Liger kernel, and the Qwen helper packages. The optional extras in the install instructions mirror two of those, which tells you the authors consider the compiled attention kernel and the fused kernel library the two things worth installing separately.

The documentation lives in a docs directory with a reference section, including the efficiency report, and the repository also carries a continuous integration directory, a tools directory, a test directory and a helper script for the Linux sync. The examples directory holds a run script at its root alongside the per-model directories, which is the fastest way into the project: copy an example directory, edit its configuration, and run it.

The version number is the caveat. A 0.1.4 engine with a release every few months is a project in the phase where the interfaces are still moving, and the table of supported optimisations is the best evidence of that, since a technique that works for one architecture and not another is usually a sign that the underlying integration is model-specific rather than generic.

So the sensible way to use it is as a reference implementation and a starting point rather than as a dependency to pin, and to read the efficiency report before believing any performance claim, including the ones in this article's description of the project.

Editorial conclusion

Adopt lmms-engine if you are training vision-language, diffusion or mixture-of-experts models and want one harness with the same launch command, the same configuration style and a clear table of which optimisations apply to which architecture, since that matrix saves a lot of trial and error. Do not adopt it if you need a stable, slow-moving dependency, because the project is at version 0.1.4 and the dependency list pulls in a deep stack including deepspeed, bitsandbytes, triton and a compiled attention kernel. Before you plan a run, check the matrix for your specific model, because most of the listed models have no tensor parallelism support at all, and decide whether you need the stable release from the index or the editable checkout.

Frequently asked questions

Official sources

1. EvolvingLMMs-Lab/lmms-engine on GitHub
2. Issues
3. Project website
4. README
5. Releases