TPU-MLIR: an MLIR compiler that turns PyTorch, ONNX and HuggingFace models into Sophgo bmodel files

The gap TPU-MLIR fills between a trained model and a Sophgo chip

A trained network is a graph of framework operations. A Sophgo TPU does not execute that graph. Something has to lower it to an instruction stream the chip understands, and the artifact that carries that stream is a bmodel file. TPU-MLIR is that lowering step. The README describes it as converting pre-trained networks from mainstream frameworks into bmodel files that run efficiently on TPUs. The audience is therefore narrow and concrete: engineers who have already picked Sophgo hardware and now need their model to run on it. If you are still comparing accelerators, this project is not a factor in that decision, because the output format only targets this vendor's chips. The front ends listed are PyTorch, ONNX, TFLite and Caffe, with the README noting that other frameworks reach the compiler through ONNX. That ONNX bridge matters more than it looks: it means the practical set of supported models is whatever can be exported to ONNX cleanly, and export quirks in your framework become compiler problems downstream. The project also carries a paper reference (arXiv 2210.15016) in its badge row, which is where the design rationale lives if the README is too thin for you.

Top and Tpu dialects, layer-group memory planning, and why quantization is inside the compiler

The architecture is MLIR, and the README names two dialects: Top and Tpu. The naming is a fair summary of the pipeline. A Top-level representation is the framework-agnostic graph after import, and the Tpu dialect is the point where operations map onto what the chip can actually do. Between them sit pattern rewrites and what the README calls layer-group memory planning. That last item is the one worth pausing on. Memory planning at the layer-group level implies the compiler decides how intermediate tensors are laid out and reused across fused groups of layers, rather than emitting each operation independently and letting a runtime figure it out. For a compiler targeting an accelerator with limited on-chip memory, that decision is the difference between a model fitting and a model spilling. Quantization lives in the same pipeline rather than in a separate tool. The README lists F32, BF16, F16 and INT8 (symmetric and asymmetric), plus passthrough for AWQ, GPTQ and AutoRound checkpoints, calibration and QAT. Putting calibration inside the compiler means the accuracy you measure and the artifact you deploy come from one build, which removes a class of mismatch between a separately quantized model and the graph the compiler actually lowers. It also means you cannot easily swap in a quantization scheme the project has not implemented.

Installing TPU-MLIR: a privileged Docker container, then a wheel or a source build

The supported environment is a prebuilt Docker image, not a bare host. The README gives the pull command as docker pull sophgo/tpuc_dev:latest, with a fallback that downloads a tarball (tpuc_dev_v3.4.tar.gz) and loads it with docker load -i. The container is started with docker run --privileged --name tpu-mlir -v $PWD:/workspace -it sophgo/tpuc_dev:latest. Note the --privileged flag: this is not a sandboxed build container, and if your policy forbids privileged containers you have a problem before you compile anything. Inside the container there are two install paths. The fast one is pip install tpu_mlir, which the README says requires Python 3.10 or newer on Ubuntu 22.04, already satisfied in the image. The other is a source build, which the README marks as recommended: clone with --depth 1, pip install -r requirements.txt, source ./envsetup.sh, then ./build.sh. The envsetup.sh step is not optional decoration; it is what puts the local build on the path ahead of any installed wheel. Choose source build if you intend to patch the compiler or track master, and the wheel if you only need to convert models.

llm_convert.py and the --use_history_kv decision that shapes your whole build

The LLM path is a single script, llm_convert.py, and the README frames it as two scenarios selected by --use_history_kv. Without that flag, the compiler emits two instruction groups, block_ for prefill and block_cache_ for decode, and the README recommends this for single-turn conversations with short context, giving 4K as the rough boundary. With the flag, a third group appears: block_kv_, described as prefill with history. That variant is recommended for multi-turn conversations, long contexts such as 8K, or when you are unsure, and the README calls it more flexible with good overall performance. The chunking behaviour is the part most likely to surprise people. --chunk_length sets the segment length for chunked inference, and the README's own example is explicit: with 1K chunks, a 7K input runs prefill as block_ followed by seven block_kv_ invocations. Decode is segmented by KV-cache length too, so the README states that performance differs at 1K, 2K, 4K and 8K lengths. In other words, one compiled bmodel does not have a single performance number; it has a curve, and your chunk_length choice moves you along it. Required arguments are the model path (-m), sequence length (-s), chip (-c, one of bm1684x, bm1688, cv186ah) and output directory (-o). --max_input_length defaults to -s when omitted. The README also flags --dynamic for dynamic-shape compilation and recommends it for all new conversions, noting Qwen3.5 forces it.

Where TPU-MLIR is the wrong tool, and what ONNX Runtime does differently

The clearest limitation is portability of the output. A bmodel targets Sophgo chips, and nothing in the material suggests it runs anywhere else. If your deployment plan involves more than one accelerator vendor, you are maintaining a separate compiled artifact per vendor, and this compiler only covers one of them. A second limitation is the environment: the documented install path is a privileged Docker container on Ubuntu 22.04 with Python 3.10 or newer. That is a heavier footprint than a pip install into an existing environment, and the --privileged requirement is a real constraint in locked-down CI. The natural alternative is ONNX Runtime. The difference in approach is fundamental rather than incremental. ONNX Runtime takes the ONNX graph and executes it through vendor-specific execution providers, so the same model file can run on CPU, GPU or a supported accelerator by swapping the provider at runtime. TPU-MLIR instead compiles ahead of time to a vendor-specific binary. You trade runtime portability and the ability to change hardware without a rebuild for a compiled artifact with layer-group memory planning and quantization decisions baked in. If your model is small enough to run acceptably on CPU, or if hardware flexibility matters more than peak efficiency on one chip family, ONNX Runtime is the simpler answer and TPU-MLIR adds a build step you do not need.

Release cadence, the NOASSERTION licence, and what a version bump costs you

The release history shows v1.27 in February 2026, v1.28.1 in April 2026, and v1.30.2 in August 2026, with the last push to master on the same day as the v1.30.2 tag. That is a steady cadence of roughly one minor release every few months, with patch releases in between. For a compiler, that pace cuts both ways. You get fixes for newly supported models, which matters if you are chasing HuggingFace releases, but you also inherit a moving target. Because the wheel is published to PyPI as tpu_mlir, pinning a version is straightforward, and pinning is the right default for a compiler whose output you have already validated. Rebuilding on a new minor version means re-running your accuracy validation, since quantization and lowering decisions can shift between releases. The licence field on the repository reports NOASSERTION, which means GitHub could not map the licence file to a recognised identifier. That is not the same as saying there is no licence, and it is not a statement about your rights. It means you cannot read the licence terms off the repository metadata, and anyone planning to redistribute a bmodel or link this compiler into a product should read the actual licence file in the tree and get it reviewed. Treat that as a prerequisite, not a formality, and do not take this article as legal advice on what the terms permit.

The auxiliary tooling that decides whether you can debug a bad conversion

Accuracy validation is the step that turns a compiler from a black box into something you can work with, and the README lists model_runner, model_tool, accuracy validation, a simulator and a visualizer. The visualizer and simulator are the interesting pair. When a converted model produces wrong output, the question is whether the graph was lowered incorrectly or the quantization was too aggressive, and those are different bugs with different fixes. A simulator lets you inspect behaviour without hardware in the loop, and a visualizer lets you read the intermediate representation. Neither is described in enough detail in the README to say how much they cover, and the quick-start documentation site is where the detail would live. The honest position is that the README names these tools but does not demonstrate them, so their usefulness is something you have to establish on your own model. The same applies to the quantization passthrough for AWQ, GPTQ and AutoRound: the README recommends starting from a pre-quantized build, which sidesteps the calibration toolchain entirely and is the fastest route to a working bmodel. If your model has no pre-quantized checkpoint available, you are on the calibration and QAT path, and that is where the accuracy work actually begins.

Editorial conclusion

Adopt TPU-MLIR if you are already committed to Sophgo silicon (bm1684x, bm1688 or cv186ah) and need to move HuggingFace LLMs or vision models onto it; the Docker image plus llm_convert.py is the intended path and there is no realistic substitute for it. Do not adopt it if your target hardware has not been chosen yet, because the compiler is bound to Sophgo chips and a bmodel is not portable. Before you start, verify three things: that the licence terms in the repository are acceptable to your legal team, that a pre-quantized AWQ, GPTQ or AutoRound checkpoint exists for your model so you can skip the quantization toolchain, and whether your workload needs --use_history_kv, since that flag decides how many instruction groups get compiled and how prefill behaves at 8K context.

Official sources

1. Issues
2. README
3. Releases
4. sophgo/tpu-mlir on GitHub