LLM-TPU: Running Quantized LLMs and VLMs on SOPHGO BM1684X, BM1688 and CV186X

The problem LLM-TPU solves is chip-specific, not generic

Most teams that reach this repository already own a SOPHGO BM1684X, BM1688 or CV186X board. They are not shopping for an inference runtime. They have a model checkpoint in HuggingFace or GGUF format and no obvious path to the TPU. LLM-TPU exists to close that gap: it is the official SOPHGO-maintained repository for deploying generative models on those chips, and the README frames it as one-click deployment of mainstream LLMs and multimodal models. The audience is therefore narrow and identifiable. It is engineers building edge or appliance products on SOPHGO silicon, plus the integrators who support them. It is not a general-purpose serving stack, and it does not pretend to be one. The model list is the real product surface: Qwen1.5 through Qwen3.5, QwQ-32B, DeepSeek-R1-Distill-Qwen at 1.5B, 7B, 14B and 32B, Llama2 and Llama3.2-Vision, InternVL2 and InternVL3, MiniCPM-V-2.6 and MiniCPM-V-4.6, Gemma3 and Gemma4, ChatGLM and Phi. That breadth is the argument for the repository. If your model is on that list and your chip is on that list, you have a documented starting point. If either is missing, the repository offers you nothing beyond the compilation toolchain underneath it.

How the pipeline works: weights in, bmodel out, demos on top

The README's pipeline diagram describes a two-stage flow. Stage one takes HuggingFace or GGUF weights and runs them through llm_convert.py, which sits on top of tpu-mlir, and produces a quantized bmodel. Stage two deploys that bmodel through Python or C++ demos onto the target chip. The bmodel is the artifact that matters. Once you have it, the chip does not need the original checkpoint, and the README states that pre-compiled bmodels are available for download so that compilation is optional. The runtime features the README lists are AWQ and GPTQ quantized model support, dynamic compilation, KV Cache, and multi-chip parallelism. Multi-chip parallelism is what makes the 32B entries plausible on this class of hardware; the QwQ-32B and DeepSeek-R1-Distill-Qwen-32B demos are described as multi-chip. Multimodal coverage is handled by composing sub-models rather than by a single graph. The Mage-VL entry is described as Mage-ViT plus Qwen3-4B plus a StreamMind Gate, which is a vision encoder, a language model and a routing component shipped as one demo. That composition detail matters because it tells you where the failure will be when a multimodal demo misbehaves: the vision tower and the language model are separately compiled artifacts, so a mismatch between them is a real debugging scenario that the repository structure implies but the README does not walk through.

Getting a demo running: run.sh and the model flag

The documented quick start is two commands. Clone the repository, then run ./run.sh --model qwen3.5. The README's one-command table gives three concrete values for the model flag: qwen3 for Qwen3-4B, qwen3.5 for Qwen3.5-2B, and internvl3 for InternVL3-2B. The flag value is a directory name under models/, which is why the table maps display names to short slugs rather than to HuggingFace identifiers. Compilation is a separate concern. The README states that compiling models requires the TPU-MLIR environment and that either a Docker image or a source build works. That is the whole of the setup instruction in the material available here: the repository does not document a pip install, a requirements file, or a version pin for tpu-mlir in the README text. If you intend to compile rather than download, treat the TPU-MLIR repository as the authoritative source for environment setup, because LLM-TPU delegates that step. The pre-compiled route avoids the toolchain entirely, and for a first evaluation it is the lower-risk path: it separates whether the chip runs the model from whether your build environment can produce a bmodel.

The per-model directories are the real documentation, and they are uneven

Look at the supported models table closely and a pattern appears in the One-click Compile column. Mage-VL, Step3_VL, Falcon-Perception and LocateAnything-3B are marked with a dash rather than a checkmark. Those are four of the newest entries in the news table, dated between July and September 2026, and they are Python-only demos. The older, more established families (Qwen3.5, Qwen3-VL, Qwen2.5-VL, InternVL3, Gemma4) carry the checkmark and, for the Qwen entries, both Python and C++ implementations. That is a reasonable reading of project maturity: new model support arrives first as a Python demo and gains compilation support and a C++ path later, if at all. The practical consequence is that the One-click Compile column is a better predictor of production readiness than the model name. A model with a dash in that column is a demo you can run and study; it is not yet a model the repository claims to compile for you. Chip coverage is uneven in the same way. Mage-VL, Step3_VL and Falcon-Perception list BM1684X only, while LocateAnything-3B and MiniCPM-V-4.6 list both BM1684X and BM1688. If you are on a BM1688, check the row before you check the model.

Where it stops being the right tool

The clearest limitation is scope. LLM-TPU targets three SOPHGO chips. Nothing in the material suggests it runs anywhere else, and the repository has no homepage or hosted service behind it. If your deployment target is a GPU server or an x86 box without a SOPHGO accelerator, this project is irrelevant regardless of how attractive the model list looks. A second limitation is that the repository is a collection of per-model demos, not a unified serving layer. Each model lives under its own directory under models/, and the README points to those directories for details rather than describing a common API. There is no documented OpenAI-compatible endpoint, no batching scheduler, no request queue in the material provided. If you need concurrent multi-tenant serving, you are building that layer yourself on top of the demos. A third issue is licence ambiguity. GitHub reports the repository licence as NOASSERTION, while the README badge links to a LICENSE file and displays Apache-2.0. Those two signals conflict, and the model directories reference checkpoints from many upstream publishers (Qwen, DeepSeek, Meta, Google, NVIDIA, OpenGVLab, Microsoft) whose terms are separate from the repository's own. The README does not consolidate those terms. That is not a legal opinion, it is an observation that the material does not resolve the question, and the answer differs per model.

Alternatives, and what actually differs

The direct alternative is to use TPU-MLIR without LLM-TPU. TPU-MLIR is the compiler the README names as the compilation prerequisite, and it is the layer that turns a model graph into something the TPU executes. Choosing TPU-MLIR alone means you own the model conversion scripts, the quantization configuration and the demo harness for every model you ship. LLM-TPU's value is precisely that it has already done that work for the models it lists, with llm_convert.py as the entry point and a run.sh wrapper for the common cases. The trade-off is control: a model that LLM-TPU does not cover is not blocked by TPU-MLIR, only by the absence of a ready-made conversion script, and teams with unusual quantization requirements may prefer to work at the compiler layer directly. A second alternative is a general-purpose inference runtime on the host CPU or a GPU, which sidesteps the chip question entirely. That is not a like-for-like swap. It changes the hardware, the power envelope and the deployment shape, so the comparison is really about whether the SOPHGO board is a fixed constraint. If it is fixed, LLM-TPU and TPU-MLIR are the two realistic layers, and the choice between them is about how much of the per-model conversion work you want to inherit.

Maintenance cost and what to verify before you commit

The news table shows a steady cadence of new model additions through 2025 and 2026, with entries roughly monthly in the middle of that period. That is a maintenance signal in both directions: the project tracks new model releases, and the list of models you might be asked to support keeps growing. Each addition is a new directory with its own demo, its own bmodel artifacts and its own upstream licence. The cost of adopting LLM-TPU is therefore not a one-time integration. It is the ongoing cost of tracking which models have a checkmark in the One-click Compile column, which have a C++ path, and which chips each row covers, because those attributes change between entries and the table is the only consolidated view. The repository ships no releases, according to the material retrieved, so there is no versioned artifact to pin against; you track the main branch. Concretely, before you commit to a model, open its directory under models/ and confirm three things: that the chip listed in the table matches your board, that the demo language you need exists (several recent VLMs are Python only), and that the licence for that specific checkpoint is acceptable for your use. The repository-level licence discrepancy between the GitHub NOASSERTION value and the Apache-2.0 badge is worth raising with SOPHGO directly, because it affects the terms you are relying on and the material here does not settle it.

Editorial conclusion

Adopt LLM-TPU if you already have BM1684X, BM1688 or CV186X hardware and need a supported path from HuggingFace weights to a bmodel, starting from a pre-compiled bmodel rather than a compile. Do not adopt it if your target is a GPU server, an x86 CPU-only box, or a model family absent from the models/ directory, because nothing in the repository covers that case. Before committing to a model, verify three things: that the model directory lists your exact chip, that it has the language demo you need (several recent entries are Python only), and that the licence file in that directory permits your use. The repository's own LICENSE carries a NOASSERTION identifier on GitHub while the README badge shows Apache-2.0, and that mismatch is the first thing to resolve with SOPHGO.

Official sources

1. Issues
2. README
3. sophgo/LLM-TPU on GitHub