MLC LLM: compiling LLMs for phones, browsers and desktop GPUs

What MLC LLM is for, and who it is aimed at

The README describes MLC LLM as "a machine learning compiler and high-performance deployment engine for large language models", with a mission of letting people "deploy AI models natively on everyone's platforms". That sentence is the whole product. The interesting half is "natively". MLC LLM is not a Python wrapper around a CUDA kernel. It takes model weights, compiles them through the TVM stack into platform-specific code, and ships a runtime called MLCEngine that executes the result on the device.

The audience follows from that. If you are writing an iOS app that needs a chat model without a network round trip, or an Android app that must work offline, or a browser page that runs inference in WebGPU, MLC LLM is aimed at you. If you are standing up an inference server behind a load balancer, it is not. The README's support table lists AMD, NVIDIA, Apple and Intel GPUs across Linux, Windows, macOS, iOS, Android and the browser, which is a wider target list than most inference projects attempt, and it is the reason the compiler exists at all: writing one kernel per platform does not scale, generating them from a tensor program IR does.

The project is Apache-2.0 licensed and packaged as the PyPI distribution mlc_llm. The pyproject.toml declares version 0.20.0.dev0 and a Development Status classifier of "4 - Beta", while the only GitHub release on record is v0.1.dev0 from 2023-04-29. The last push to main was on 2026-08-17, so the repository is not dormant, but the release page is not where you track this project. Track the PyPI version and the docs site instead.

How the compiler and MLCEngine fit together

There are two halves, and conflating them is the usual source of confusion.

The first half is the compiler. Model weights are converted into an MLC format and then compiled ahead of time into a shared library or a set of device kernels for a named target. This is a build step, not a runtime step. The output is a platform-specific artifact: a Metal library on Apple silicon, a Vulkan or CUDA or ROCm module on Linux and Windows, an OpenCL kernel set for Adreno and Mali GPUs on Android, WebGPU plus WASM for the browser. The README's table maps each platform to the backend it uses, and that mapping is the practical constraint you plan around.

The second half is MLCEngine, which the README calls "a unified high-performance LLM inference engine" spanning those platforms. It exposes an OpenAI-compatible API in several bindings: a REST server, Python, JavaScript, iOS and Android. Because the API surface is OpenAI-compatible, code written against the OpenAI client library can be pointed at a local MLCEngine instead, which is a smaller migration than it sounds.

The data flow is therefore: weights on disk, a compile step bound to a target, a compiled artifact, and then MLCEngine loading that artifact and serving tokens. The compile step is the part that costs you time and disk. It is also the part that makes the deployment step cheap, because the device never has to compile anything.

The project builds on TVM, TensorIR and MetaSchedule, all cited in the README's reference block. If you have used TVM before, the mental model transfers directly. If you have not, budget time for the target and quantization flags, which are the two places where a build most often fails.

Installing MLC LLM and running a first chat

The README does not inline installation steps. It points at the documentation: an Installation page, a Quick start page and an Introduction page, all under llm.mlc.ai/docs. The package name declared in pyproject.toml is mlc_llm, so the install path is the Python package of that name. Because the README itself gives no command, treat the documentation as the authority for the exact invocation and confirm it before you script anything.

The shape of a first run is a two-step sequence: install the package, then have it fetch a model and start a chat session. The README's Get Started section routes you to the Quick start page for the second step, and the repository ships a python examples directory alongside a rest examples directory, which is a fair hint that the two supported entry points are a Python session and an HTTP server. The dependency list in pyproject.toml includes openai, fastapi and uvicorn, which is consistent with a FastAPI server that speaks the OpenAI schema.

pip install mlc-llm

The distribution name on PyPI and the import name are not always identical, so if that fails, check the Installation page rather than guessing at variants. Once the package is present, the documentation's quick start drives model download and chat from a single command; the Python examples directory in the repository is the fallback reference if the docs and the code have drifted apart.

For the server path, the repository keeps an examples/rest directory, and the README states that MLCEngine exposes an OpenAI-compatible REST API. That means an existing OpenAI client can be repointed at the local server without rewriting the call sites. The README does not document the default port or the launch flag for that server, so read the REST example in the repository before you write deployment scripts around it.

Where MLC LLM is the wrong choice

The compile step is the first real cost. Every target you support needs its own compiled artifact, and every model change means recompiling. If your deployment is a single Linux box with an NVIDIA GPU and the model changes weekly, you have taken on a build pipeline for no benefit. A runtime that loads weights directly is simpler and faster to iterate on.

Memory is the second constraint. On-device inference is bounded by the device, and the README's support table includes mobile GPUs and integrated Intel graphics. A model that runs comfortably on a discrete GPU will not necessarily fit on a phone, and quantization is the lever you pull. The README does not publish memory requirements per model size, so that budget has to come from your own measurement on the target hardware.

Platform support is uneven by nature. The table shows NVIDIA GPUs on macOS as N/A, Apple GPUs on Linux and Windows as N/A, and Android support limited to OpenCL on Adreno and Mali. If your target is not in that table, the project does not claim to support it, and you should not assume a backend exists because a similar one does.

The release history is the fourth thing to weigh. The only listed GitHub release is v0.1.dev0 from 2023-04-29, while pyproject.toml carries 0.20.0.dev0. Anyone who pins dependencies by reading the GitHub Releases page will pin something three years stale. Pin from PyPI and read the docs changelog instead.

MLC LLM compared with vLLM and llama.cpp

The two comparisons people actually search for are vLLM and llama.cpp, and they land on opposite sides of MLC LLM.

vLLM is a server. Its design goal is throughput on a machine you control, with paged attention and continuous batching to keep a GPU busy across many concurrent requests. MLC LLM's design goal is reach: the same model running natively on a phone, a browser and a desktop GPU. If your problem is a hundred simultaneous users and a rack of GPUs, vLLM is the shape of the answer and MLC LLM is not, because MLCEngine is built to serve the device it runs on.

llama.cpp is closer in spirit. It is a C/C++ inference implementation with hand-written kernels and GGUF weights, and it also targets phones, desktops and browsers. The difference is the mechanism. llama.cpp ships kernels that humans wrote and optimizes for the hardware it knows. MLC LLM generates kernels from a tensor program IR through TVM and MetaSchedule, so a new backend is a compiler target rather than a new kernel file. That is a real advantage for a platform nobody has written kernels for, and a real disadvantage when you want to read and patch the kernel that is slow.

A practical tiebreaker: if you want the widest model catalog with the least ceremony, llama.cpp's GGUF ecosystem is broader. If you want one compiled artifact per platform and an OpenAI-compatible runtime across all of them, MLC LLM's model is the one that scales to more targets without more hand-written code.

Licence, maintenance and upgrade cost

MLC LLM is Apache-2.0, and both the LICENSE file and the pyproject.toml license field say so. The Python source files carry the standard Apache header, and the repository includes a NOTICE file, which is the Apache convention for third-party attribution. If you redistribute a compiled artifact, read the NOTICE file and the licences of the model weights you compiled, because the model licence is separate from the engine licence and is the one that usually bites. This is a description of what the repository contains, not legal advice.

Upgrade cost concentrates in two places. The compiled artifact is tied to the compiler version that produced it, so an engine upgrade generally means a recompile of every target you ship. And the model conversion format can move between versions, which means re-converting weights as well. Neither is documented in the README as a compatibility guarantee, so treat the version pairing of compiler and engine as something you pin deliberately.

The repository is not archived, and the last push to main was on 2026-08-17. That is recent enough that the codebase is moving, but the GitHub release list has not kept pace, so the PyPI version is the number that matters when you decide whether to upgrade.

Editorial conclusion

Adopt MLC LLM when the model has to run on the user's hardware: an Android or iOS app, a browser tab, or a desktop with a GPU you do not control. Do not adopt it if you are serving many concurrent users from one machine, because that is the problem vLLM is built for, or if you want a one-line download-and-chat experience, which is what Ollama gives you. Before committing, verify three things: that your target platform appears in the support table, that the model has a prebuilt MLC weight set on Hugging Face, and that the compiled artifact fits in the memory budget of the weakest device you intend to ship to.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. mlc-ai/mlc-llm on GitHub
3. Project website
4. README
5. Releases