Model or dataset
UbiquitousLearning/mllm avatar
UbiquitousLearning/mllm

mllm sets two CMAKE_BUILD_TYPE values and calls pytest a runtime dependency

Fast Multimodal LLM on Mobile Devices

1,638 stars222 forksC++MIT

At a glance

What is it?
A C++ inference engine for phones and edge hardware, shipped with Python bindings and four console scripts. The build file disagrees with itself on the compiler flags, the Jetson headline number does not match the table printed beside it, and the test framework is a hard install requirement.
Who is it for?
The inference work here is substantial and the numbers published for it are specific enough to check, which is more than most projects in this space offer. Two things to verify before depending on it: which build type your wheel was actually compiled with, since the file names it twice, and whether the throughput you care about is prefill or decode, since only prefill shows a gain and the multimodal table tops out near half the figure quoted in the prose.
Can I use it commercially?
Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
Is it still maintained?
Yes. The repository last received commits 46 days ago.
What is it written in?
Mainly C++, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.

From daily snapshots

Star history

  • UbiquitousLearning/mllm1,638 ★ · +31
Star history
Repository2026-09-102026-11-15
UbiquitousLearning/mllm1.6k1.6k

31 Stars in the last 66 days, measured from the daily GitHub snapshot. Compare with other projects

Editorial analysis

The Jetson headline and the table printed under it are two measurements

The prose claims up to a 3.12x prefill speedup over llama.cpp on AGX Orin 32GB with Qwen3-VL-2B W8A8, and about 12243 tok/s of prefill throughput. The table immediately below reports mean TPS across repeated runs at `input_len=2048` for the multimodal path, and its highest value anywhere is 6443.59.

Those are not the same measurement, and the file says so: `bench_one_batch --image` covers vision encoding plus image and text token prefill, while the quoted figure comes from the text serving path. A reader who takes the first number as a description of the second will be off by roughly a factor of two, and nothing in the layout signals that the two tables describe different work.

The table also shows quantization is not free. On AGX Orin 32GB with Qwen3-VL-2B, W4A16 at 4700.28 is below FP16 at 4875.75, while W8A8 at 6443.59 is above both. On Orin NX 16GB the ordering flips, with W4A16 at 2494.89 edging FP16 at 2438.27. Decode is described as generally close to llama.cpp, with small wins or losses by model, device, and quantization.

The build file names two different CMAKE_BUILD_TYPE values

One is passed as a CMake argument:

code
cmake.args = [
  "-DCMAKE_BUILD_TYPE=Release",
  "-DMLLM_ENABLE_PY_MLLM=on"
]

The other sits under the build configuration key:

code
cmake.build-type = "RelWithDebugInfo"

Both are in the same project file, for the same wheel. Which one reaches CMake depends on how the arguments and the key are combined, and the file resolves it silently. A package whose selling point is speed is the case where an unintended debug-info build type costs the most, and there is no comment marking either line as the intended winner.

Two adjacent settings add to the picture. `wheel.cmake = true` forces the CMake build, with a note that it can be set to false or disabled through `SKBUILD_WHEEL_CMAKE=false` to skip it. The wheel is also declared ABI-agnostic with `wheel.py-api = "py3"`, so nothing in the build output tells you which backend produced the binary you installed.

pytest and pytest-html are declared as runtime dependencies

The dependency list for the Python package is not a short one. Alongside `torch`, `torchao`, `modelscope`, `fastapi`, `uvicorn`, `typer`, `openai`, `packaging`, `pyyaml`, `pyfiglet`, and `termcolor`, it contains `pytest` and `pytest-html`. There is no test dependency group in this file, so those two are what every install of `pymllm` pulls.

The consequence is that a user who only wants to run inference on a phone or an edge box installs a test runner and an HTML report writer, and gets them as hard requirements rather than extras. An optional extras table exists in the file and holds exactly one group, `cuda`, carrying `tilelang`, `flashinfer-python`, and `pyzmq`, which is the group that is genuinely backend-specific.

The build requirements also pin `apache-tvm-ffi == 0.1.8.post2` to an exact version, and that same exact pin appears again in the runtime dependency list. Duplicating an exact pin in two places is the kind of thing that ages badly, because a future bump has to find both.

Four backends are claimed and the support table has columns for three

Unified hardware support is listed as Arm CPU, OpenCL GPU, QNN NPU, and Ascend NPU. The v2 model support table has three columns: CPU, Hexagon NPU, and Ascend NPU.

So OpenCL GPU, one of the four named backends, has no column at all. CUDA has no column either, which is notable because the Jetson benchmarks, the Jetson serving support, and the Triton and CUTLASS paths are all CUDA work, announced in the news entries for March through June 2026.

The two NPU columns are also thin in a specific way. Hexagon carries a single entry, W4A16 on SM8650 for Qwen3-1.7B. Ascend carries a single entry, W8A8 for Qwen3-0.6B. Every other model in the table, from Qwen3-4B through Qwen3.5-4B, MiniCPM5-1B and 2B, LFM2.5-2.6B, and DeepSeek-OCR, has a CPU entry and blank NPU cells.

The CPU column is the one that is filled in, and its links point at w4a8 artifacts on ModelScope plus per-model example directories in the tree.

Five news entries postdate the newest tag, and pyproject is two patches ahead

The two published releases are 2.0.0 on 2026-02-16 and 1.0.0 on 2024-01-24. The Python package version in the project file is `2.0.2`. So the distribution you install from a package index is two patch releases ahead of the newest tag on the repository, and neither of those patches has a release entry.

The news list runs past the tag as well. Entries dated 2026 Mar 18, Apr 30, May 02, Jun 08, and Sep 08 all sit after 2026-02-16, covering CUDA on Jetson Orin and Thor, compressed-tensors AWQ and INT8 paths, the Ascend NPU backend, and Qwen3.5 coverage. Those are the changes the Jetson benchmark table describes.

The version announcement runs the other way. The news says MLLM v2 released on 2025 Nov 23, and the tag for 2.0.0 is dated 2026-02-16, so the release was announced about three months before it was cut. The last commit is dated 2026-09-30. Nothing here is archived.

V1 retirement integrated one feature and deferred another

The 2025 Aug 28 entry announces that support for MLLM V1 is ending, that GPT-OSS would be integrated into V1 before its retirement, and that the project would then move to V2, which could be viewed on a V2 branch. The listed V2 capabilities were a more Pythonic model authoring approach with eager execution, compilation support for easier NPU integration, support for parallel execution of multiple models, and a refined engineering implementation.

So the V1 line received exactly one named feature and a retirement date, and the default branch of the repository is `main` today, which means the branch arrangement described in that entry no longer describes where the code lives.

The QNN backend entry from 2025 Jul 30 shows the other pattern, a feature landing with a deferral attached: rotation quantization was added and Qwen-2-VL 2B was supported, with ViT profiling marked as integrating in v2. Deferred work is signposted rather than hidden, which is worth having in a project whose model table has this many empty cells.

Dependencies are declared in four files and the root one has three lines

The file named `requirements.txt` at the repository root contains:

code
safetensors
pyyaml >= 6.0.2
cmake-format == 0.6.13

Three lines, one of which is a CMake formatting tool rather than anything the library imports at runtime. It is nowhere near the sixteen entries in the project file, so it is not the set you install to use the Python package, and nothing in the file says what it is for. Two more variants sit alongside it, `requirements-mini.txt` and `requirements-qnn-aot.txt`, which together with the project file make four places where a dependency can be declared.

The Python side also carries two formatters, `black` with `target-version` covering py310 through py312 and `usort` configured with `first_party_detection = false`. The C++ side carries three, `.clang-format`, `.clang-tidy`, and `.clangd`, plus a `.clang-tidy.ignore`. Around those sit `.editorconfig`, a `.codespellrc` spelling dictionary, a pre-commit configuration, `tox.ini`, and a `task.py` with a `tasks/` directory.

Four console scripts for one package, two of them servers

Installing the Python package installs four executables:

code
[project.scripts]
pymllm = "pymllm.__main__:main"
mllm-convertor = "pymllm.mobile.utils.mllm_convertor:main"
mllm-service = "pymllm.mobile.service.tools:cli_app"
pymllm-server = "pymllm.server.launch:main"

`mllm-convertor` is the ingestion path the architecture section names, taking PyTorch and SafeTensors models, quantizing them, and converting into mllm format for the runtime to load. The other two are network-facing: a service entry point under `mobile.service.tools` and a server launcher. Three of the four names use the `mllm` prefix and one uses `pymllm`, and the package directory itself is `pymllm`.

That matters on a phone, which is the stated target. The Android build does not use JNI to call the engine directly; it introduces an in-app server layer written in Go and shipped as `mllm_server.aar`, so the UI and the inference are separated by a network boundary inside the app. Given that, having the Python side also expose two server entry points is coherent rather than accidental.

Editorial conclusion

The inference work here is substantial and the numbers published for it are specific enough to check, which is more than most projects in this space offer. Two things to verify before depending on it: which build type your wheel was actually compiled with, since the file names it twice, and whether the throughput you care about is prefill or decode, since only prefill shows a gain and the multimodal table tops out near half the figure quoted in the prose. Treat the NPU columns as a single verified configuration each rather than broad support. Anyone packaging this should expect to pin the compiler backend themselves, since the model table covers CPU, Hexagon, and Ascend while the Jetson work lives in CUDA.

Frequently asked questions

What is UbiquitousLearning/mllm?

A C++ inference engine for multimodal models on mobile and edge devices, MIT licensed, with Python bindings distributed as `pymllm` and a model conversion tool. It sits between optimization methods such as speculative decoding, pruning, and quantization, and compiler or runtime layers including CANN, CUDA, and MLIR.

What hardware backends does mllm support?

Four are named: Arm CPU, OpenCL GPU, QNN NPU, and Ascend NPU. The v2 model support table has columns for CPU, Hexagon NPU, and Ascend NPU only, with one Hexagon entry for Qwen3-1.7B W4A16-SM8650 and one Ascend entry for Qwen3-0.6B W8A8.

How does mllm compare to llama.cpp on Jetson Orin?

On prefill, up to 3.12x with Qwen3-VL-2B W8A8 on AGX Orin 32GB and about 12243 tok/s, while decode throughput stays broadly close to llama.cpp. The W4A16 path uses AWQ compressed tensors with Marlin GEMM, and the W8A8 path uses Triton per-token activation quantization with CUTLASS INT8 GEMM.

How do you convert a model for use with mllm?

Through `mllm-convertor`, which ingests PyTorch and SafeTensors checkpoints, quantizes and converts them into mllm format for the runtime to load and run. It is installed as one of four console scripts alongside `pymllm`, `mllm-service`, and `pymllm-server`.

What is the relationship between mllm V1 and V2?

Support for V1 was announced as ending, with GPT-OSS to be integrated into V1 first, after which the project moved to V2. V2 was described as bringing Pythonic model authoring with eager execution, compilation support for NPU integration, parallel execution of multiple models, and a refined engineering implementation.

Official sources

  1. License: MIT
  2. Project website
  3. README
  4. Releases
  5. UbiquitousLearning/mllm on GitHub
Add this badge to your README

If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.

Add this badge to your README

markdown
[![Hysen Labs](https://hysenlabs.com/badge/ubiquitouslearning-mllm.svg)](https://hysenlabs.com/projects/ubiquitouslearning-mllm)

Star history chart for your README

markdown
[![Star history](https://hysenlabs.com/chart/ubiquitouslearning-mllm.svg)](https://hysenlabs.com/projects/ubiquitouslearning-mllm)

Was this useful?