mllm: a C++ multimodal inference engine for phones, NPUs and Jetsons
Fast Multimodal LLM on Mobile Devices
At a glance
- What is it?
- mllm is an MIT-licensed C++ runtime that runs multimodal LLMs on Arm CPUs, Qualcomm Hexagon NPUs and Ascend NPUs, with a Python authoring layer called pymllm and an on-device Android server. It is a build-it-yourself project, not a drop-in SDK.
- Who is it for?
- Adopt mllm if you are shipping a multimodal model onto Arm CPUs, Hexagon NPUs or Ascend hardware and you are willing to build the C++ runtime and convert checkpoints yourself. Do not adopt it if you need a stable Python-first serving stack or a single vendor-neutral binary: the v1 to v2 transition is still in progress and the Ascend and Jetson CUDA paths are recent additions.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 8 days ago.
- What is it written in?
- Mainly C++, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap mllm is aimed at: multimodal models on hardware without a datacenter GPU
Most inference engines assume a server. mllm assumes the opposite. The README describes it as a fast and lightweight multimodal LLM inference engine for mobile and edge devices, and the model support table is organised around that assumption: Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B and Qwen3.5-0.8B, each listed against a CPU column, a Hexagon NPU INT8 column and an Ascend NPU column. The target reader is an engineer who has a phone, an Orin board or an Ascend device and needs a vision-language model to run on it rather than in a rack. The multimodal part matters here. Vision encoding plus image and text token prefill is a different workload from text-only generation, and the repository treats it as a separate benchmark path through bench_one_batch --image. If you only need a text model on a phone, this project is carrying weight you will not use.
Two execution paths: the C++ runtime and pymllm
The repository contains two things that are easy to confuse. The first is the C++ engine, which loads mllm-format checkpoints and executes them. The second is pymllm, a Python-facing layer used for serving on Jetson and other devices. The README's Jetson section describes pymllm running Qwen3, Qwen3-VL and Qwen3.5 with BF16, W4A16 and W8A8 serving. The W4A16 path uses AWQ compressed tensors with Marlin GEMM; the W8A8 path uses Triton per-token activation quantization with CUTLASS INT8 GEMM. That is a specific stack, not a generic wrapper. The C++ side is where the mobile deployment story lives, and the two do not share a single code path. Pick which one you are integrating before you start reading the build files, because the answer changes which dependencies you install.
How a checkpoint becomes an mllm binary
The README shows a conversion step in the workflow diagram description: mllm-convertor ingests PyTorch and SafeTensors checkpoints, quantizes and converts them into mllm format, and the mllm runtime loads that format. This is the mechanism that makes the rest work. The engine does not read Hugging Face safetensors directly at inference time; it reads its own converted artifact. Practically, that means your deployment pipeline has a build-time stage that is separate from your runtime stage, and a model that works in PyTorch is not automatically supported. The support table lists specific quantized checkpoints, for example Qwen3-0.6B w4a8 on CPU and Qwen3-1.7B W4A16-SM8650 for Hexagon NPU, published on ModelScope. Those links are the real compatibility surface. If your model is not in that table, the conversion path is the first thing you have to validate, and the README does not document a general recipe for arbitrary architectures.
The Android demo runs a Go server inside the app
The Android integration is the most distinctive design choice in the repository. Instead of the usual JNI call from the UI thread into native inference code, mllm ships an in-app server layer written in Go, packaged as mllm_server.aar. The README states the goal plainly: decouple the UI from the heavy inference computation, and it calls this a client-server architecture entirely on-device. Two consequences follow. First, the app now contains a second runtime, so the APK carries a Go server alongside the C++ engine. Second, token streaming becomes a network-shaped problem even though nothing leaves the device, which is presumably how the November 2025 note about stable Qwen3 and DeepSeek-OCR streaming on Android was achieved. Whether that trade is worth it depends on your team. If you already have a JNI pipeline and a working threading model, replacing it with an in-process server is a large refactor for a streaming benefit you may be able to get another way.
Backend coverage is uneven, and the README says so
The v2 support table is the most honest document in the repository, because the empty cells are as informative as the filled ones. Qwen3-1.7B has a Hexagon NPU W4A16-SM8650 entry; Qwen3-4B has no NPU entry at all. Qwen3-0.6B has an Ascend W8A8 entry; Qwen3-1.7B does not. CPU coverage is the broadest, with w4a8 variants for the Qwen3 sizes. This means the phrase unified hardware support in the feature list describes an architectural ambition, not a per-model guarantee. The QNN backend also has an ahead-of-time execution path, announced in February 2026, which the README links to a quick start and a technical report. AOT compilation for full graph execution on NPU is a meaningfully different workflow from the CPU path, and the documentation for it lives outside the README. Budget time for that if Hexagon is your target.
The v1 to v2 transition is the main adoption risk
In August 2025 the project announced that support for MLLM V1 would end soon, with V2 living on a separate branch. V2 was described as bringing a more Pythonic model authoring approach with eager execution, compilation support for NPU integration, parallel execution of multiple models, and what the README calls a more refined engineering implementation. Version 2.0.0 was released in February 2026. That is a long overlap period, and it means any tutorial, example or third-party writeup older than late 2025 may describe an API that is being retired. The examples directory is the place to check: if the example you need has not been ported, you are reading V1 material. This is the single largest source of wasted integration time for a new adopter, and it is not a defect in the code so much as a consequence of a deliberate rewrite.
Where mllm is the wrong tool
If your deployment target is a desktop or a server GPU and you want one binary that runs many models with minimal conversion work, mllm adds a conversion stage and a C++ build for no benefit. The Jetson numbers in the README are explicitly a comparison against llama.cpp, and the README is candid that decode throughput is generally close between the two, with small wins or losses depending on model, device and quantization. The measured advantage is concentrated in prefill, where the README reports Qwen3-VL-2B W8A8 reaching up to 3.12x prefill speedup on AGX Orin 32GB at input_len=2048 and output_len=128. That is a specific claim from the project's own benchmarks, not an independent measurement, and it is a prefill claim. If your workload is decode-heavy, interactive chat with short prompts for example, the case for switching is much weaker. Similarly, if you need a mature Python serving ecosystem with broad model coverage, pymllm covers Qwen3, Qwen3-VL and Qwen3.5 on Jetson, which is a narrower set than a general-purpose server would give you.
Licence and the cost of keeping up
mllm is MIT licensed, which permits commercial use and modification with the usual requirement to preserve the copyright notice and permission text. That is permissive and places few obligations on how you ship. It says nothing about the licences of the model weights you convert, and those are separate: the checkpoints linked from the support table live on Hugging Face and ModelScope under their own terms, so a permissive engine licence does not make a model licence permissive. On maintenance, the release history is sparse by design. There are two tagged releases, 1.0.0 in January 2024 and 2.0.0 in February 2026, but the news list shows continuous backend work between them: Ascend NPU in May 2026, Jetson CUDA in March 2026, QNN AOT in February 2026, Android streaming in November 2025. Upgrades arrive as backend additions rather than as tagged versions, so pinning to a release tag will leave you behind on hardware support, and tracking main means tracking a moving target across four backends. Neither choice is free. Whichever you pick, the mllm-convertor step is the part most likely to need re-running when you upgrade, since converted artifacts are tied to the format the runtime expects.
Editorial conclusion
Adopt mllm if you are shipping a multimodal model onto Arm CPUs, Hexagon NPUs or Ascend hardware and you are willing to build the C++ runtime and convert checkpoints yourself. Do not adopt it if you need a stable Python-first serving stack or a single vendor-neutral binary: the v1 to v2 transition is still in progress and the Ascend and Jetson CUDA paths are recent additions. Before committing, verify that your exact model and quantization appear in the v2 support table, and check which backend your target device actually uses, since CPU, Hexagon NPU and Ascend NPU coverage differ per model.
Community notes