Model or dataset
sophgo/tpu-mlir avatar
sophgo/tpu-mlir

tpu-mlir recommends its more complex compile path exactly when you are unsure

Machine learning compiler based on MLIR for Sophgo TPU.

1,022 stars231 forksC++NOASSERTION

At a glance

What is it?
An MLIR-based compiler that turns models from PyTorch, ONNX, TFLite and Caffe into bmodel files for three Sophgo chip targets, documented around a Qwen walkthrough. The docs steer you to the three-instruction-group layout when you do not know which to pick, the chunk length multiplies the compiled artifact, the install runs a privileged container with your working directory mounted, and one pinned dependency has been removed from PyPI.
Who is it for?
TPU-MLIR is the right tool if you have a trained model and Sophgo hardware and want a compiler with a real lowering pipeline, a quantization toolchain and accuracy tooling behind it. Three things to check before committing.
Can I use it commercially?
Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
Is it still maintained?
Yes. The repository last received commits 12 days ago.
What is it written in?
Mainly C++, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.

From daily snapshots

Star history

  • sophgo/tpu-mlir1,022 ★ · +43
Star history
Repository2026-09-102026-11-15
sophgo/tpu-mlir9791k

43 Stars in the last 66 days, measured from the daily GitHub snapshot. Compare with other projects

Editorial analysis

Two instruction-group layouts, and the docs pick the bigger one for undecided users

LLM compilation is a single switch, `--use_history_kv`, and it changes the shape of the output. Without it you get two instruction groups: `block_` for prefill and `block_cache_` for decode, which the page recommends for single-turn conversations with short context, giving 4K as the example.

With it you get three: `block_` for prefill, `block_kv_` for prefill with history, and `block_cache_` for decode. The recommendation for that layout is multi-turn conversations, long contexts at 8K, and then the phrase that decides most cases: or when unsure, because it is more flexible with good overall performance.

So the documented default is the larger artifact. A user with no strong opinion is steered toward three instruction groups instead of two, on the strength of a flexibility claim with no measurement attached. That is a defensible default for a compiler, since a recompile is expensive compared with carrying a spare group, and it is also the reason the two paths produce files that are not drop-in replacements for each other.

The commands show the shape difference directly. The no-history example compiles to `qwen3.5_4b`, and the history example to `qwen3.5_4b_history`.

Chunk length is a multiplier on the compiled artifact, not a tuning knob

`--chunk_length` is documented as the segment length for chunked inference, and the page then does the arithmetic that makes it matter:

> with 1K chunks, a 7K input runs prefill as `block_` + 7 x `block_kv_`; decode is also segmented by KV-cache length, so performance differs at 1K / 2K / 4K / 8K lengths.

So the prefill instruction count is the input length divided by the chunk length, and each unit is a separate `block_kv_` group in the compiled output. A smaller chunk length does not make the model faster, it multiplies how many instruction groups the compiler emits, which is why the page notes that performance differs across the four chunk lengths it names. Chunk length is an artefact-size decision that then shows up as a runtime decision.

The same segmenting applies to decode, by KV-cache length, which means the two phases can be segmented on different boundaries. That is a detail worth holding onto, because it is where a compiler with a chunked LLM path usually hides its complexity and this page states it in one sentence.

The history-KV example uses a chunk length of 1024 with a sequence length of 8192, so on the page's own arithmetic that is `block_` plus eight `block_kv_` groups before decode begins.

The walkthrough names the same model three different ways

The LLM quick start is a copy-paste sequence, and the identifiers drift across it. Step one installs git LFS and clones a repository at `https://huggingface.co/Intel/Qwen3.5-2B-int4-AutoRound`.

Step two then compiles a model at the path `/workspace/Qwen3.5/Qwen3.5-2B-int4-AutoRound`. The directory root there is `Qwen3.5`, not the cloned `Qwen3.5-2B-int4-AutoRound`, and nothing in the page renames or moves the clone into that shape.

Then the output. Both compile commands pass `-o qwen3.5_4b` or `-o qwen3.5_4b_history`, naming a 4b output for a 2b model. The model is 2B in the clone URL and in the model path, and 4b in both output names.

None of this changes the compiler. It changes whether the commands run. A user who copies the clone, then the compile line, gets a path that does not exist and an output directory labelled with a size the model does not have, and the page offers no note reconciling either. The chip flag is the one identifier that stays consistent, at `-c bm1684x` in both examples.

The chip argument is also where the platform choice lives. Its three accepted values are `bm1684x`, `bm1688` and `cv186ah`, and it is marked required, while the examples only ever show the first.

The install path is a privileged container with your working directory mounted

Everything in this compiler runs inside a prebuilt image, and the documented launch line is this:

shell
docker run --privileged --name tpu-mlir -v $PWD:/workspace -it sophgo/tpuc_dev:latest

Two things in one line. The container runs privileged, and the host working directory is bind-mounted into it. The build-from-source instructions then operate on that mount, at `cd /workspace/tpu-mlir`, so the compile and the model weights are written back to the host from a privileged process. For a team building its own models that is a reasonable trade. For a CI runner that pulls models from elsewhere, it is a container with full host privileges and a writable mount.

The image is also on a floating tag. The pull is `sophgo/tpuc_dev:latest`, and the fallback when the pull fails is a tarball at a dated asset path:

shell
wget https://sophon-assets.sophon.cn/sophon-prod-s3/drive/25/04/15/16/tpuc_dev_v3.4.tar.gz
docker load -i tpuc_dev_v3.4.tar.gz

That filename is `tpuc_dev_v3.4` and the path is dated 2025-04-15, while the tag you would otherwise pull is `latest`. So the documented recovery path installs a versioned image rather than the newest one, and the page does not say how far apart they are. The assets also come from a vendor host rather than a registry you control.

One pinned dependency has been removed from PyPI and is kept alive by a find-links index

The Python requirements file is fully pinned, roughly sixty packages with exact versions, and its own header explains the two index hacks it needs. The torch stack comes from the PyTorch CPU index so the install does not pull multi-gigabyte CUDA packages, which is why torch, torchvision and torchaudio all carry a `+cpu` local version.

The other entry is the interesting one:

code
--find-links https://www.paddlepaddle.org.cn/whl/linux/mkl/avx/stable.html

The comment above it says that PyPI no longer provides the pinned `paddlepaddle==2.5.0`, that pip reports no matching distribution found, and that the exact Linux wheel remains on PaddlePaddle's official archive, so the source is kept. A dependency pinned to a version its primary index has dropped, sustained by a second index, is a fragile arrangement that has to be re-checked every time the pin moves.

Two more pins in that file read like a different era. `nose==1.3.7` and `enum34==1.1.10` are Python 2 era packages sitting in a list whose companion installation step declares Python 3.10 as the floor. Neither is exercised by the documented build, and both are the kind of entry that lingers because removing it was never tested.

The file also says it is a canonical list mirrored to the repository root, with an instruction to keep both in sync. Two copies of a sixty-line pin set, kept aligned by hand, is the second thing most prone to drift in this repository.

The wheel is one command and the source build is the recommended one

There are two ways to get the compiler, and the page marks the slower one as recommended. The prebuilt wheel is a single line with a note that it needs Python 3.10 or higher on Ubuntu 22.04, which the Docker image already satisfies:

shell
pip install tpu_mlir

The source path is a shell script, an environment source, and a build:

shell
# git clone with --depth 1
cd /workspace/tpu-mlir
pip install -r requirements.txt
source ./envsetup.sh
./build.sh

The first line of that block is a comment. The instructions assume the source is already present, and the note about the clone is commented out rather than removed, so a reader following the sequence needs to supply the missing step themselves. The path also assumes the mount from the container launch, so the source build only makes sense inside the image.

The two paths are not equivalent in what they give you. `build.sh` compiles a C++ project with CMakeLists.txt at the root, alongside a `capi/` directory, a `bindings/` directory, `python/`, `tools/` and a `docker/` directory, so the wheel is a build of a large tree and the source path is the only way to change it. Recommending the source build while also publishing a wheel is a normal choice for a compiler, and the page does not say whether the two produce the same dialect behaviour.

For context on the size of that tree, the root also carries five separate release scripts: `release.sh`, `release_doc.sh`, `release_pip.sh`, `download_toolchains.sh` and `build.sh`, plus `test/`, `unittests/`, `regression/` and `experimental/` directories.

Four required flags, eight with no short form, and one flag one model family forces

The `llm_convert.py` argument table is the densest part of the page and the most useful. Four parameters are required: `model_path` (short `-m`), `seq_length` (short `-s`), `chip` (short `-c`) and `out_dir` (short `-o`).

The rest have no short form at all, marked with a dash: `max_input_length`, `use_history_kv`, `chunk_length`, `dynamic`, `do_sample`, `max_pixels`, `embedding_disk` and `lora_max_rank`. So a realistic invocation mixes two-letter flags with long ones, and the long-only set is the larger half.

Three of those are worth understanding rather than just copying. `dynamic` is dynamic-shape compilation and is recommended for all new conversions, with the parenthetical that Qwen3.5 forces it, which means a user on any other model has to add the flag manually or compile a fixed-shape artefact. `embedding_disk` stores word embeddings in a `.bin` file and runs them on CPU, which trades on-chip memory for host work. `max_pixels` caps image size for vision-language models and defaults to per-model values when unset, which is how the same script serves both a text LLM and a VLM.

One default is quietly conservative: `do_sample` enables random sampling, so a compiled model is deterministic unless you ask otherwise. And `lora_max_rank` sets a maximum LoRA rank, with setting it compiling a LoRA-enabled version, so adapter support is a compile-time choice rather than a runtime one.

The compiler versions itself but not the artefact, and the license metadata resolves to nothing

A few repository-level facts do not add up neatly. The release tags are specific, v1.30.2 on 2026-08-31, v1.28.1 on 2026-04-29 and v1.27 on 2026-02-06, and the compiler is not archived with a last push dated 2026-08-31. But the artefact the whole toolchain produces is called `bmodel` with no version qualifier anywhere, across three different chip targets, so nothing on the page tells you which compiler version a given bmodel file came from or which version can read it.

The license is a similar story. The repository records no recognised license identifier, while a `LICENSE` file sits in the root next to `SECURITY.md` and `CONTRIBUTING.md`, and the readme's table of contents reserves a License section. Two sources, neither of which you can resolve from the metadata alone.

The header also carries two identical PyPI badges pointing at the same `pypi.org/project/tpu_mlir` page, which is a markup artifact rather than anything substantive.

One claim in the highlights list deserves a note rather than a complaint. The entry reads bilingual docs and active community, followed by English and Chinese manuals, papers and video tutorials. The documentation claim is verifiable on the page, since `README_cn.md` sits beside `README.md` and the manual links come in both languages. The community claim carries no figure, no contributor count and no date, and the only activity signal available is the last push date.

Editorial conclusion

TPU-MLIR is the right tool if you have a trained model and Sophgo hardware and want a compiler with a real lowering pipeline, a quantization toolchain and accuracy tooling behind it. Three things to check before committing. The language around the LLM compile path is advisory rather than prescriptive, and the project's own advice for an undecided user is to take the three-instruction-group layout, so if you care about instruction count or artefact size, decide the KV strategy yourself rather than following the default. The documented install gives your Docker daemon a privileged container with the working directory bind-mounted, which is a real exposure if that machine builds untrusted models. And the Python dependency list pins a package PyPI no longer serves, kept alive by a find-links archive, which tells you the pinning strategy predates the current ecosystem.

Frequently asked questions

What does tpu-mlir compile and for which chips?

It converts pre-trained networks from PyTorch, ONNX, TFLite and Caffe into bmodel files, with other frameworks reachable through ONNX. The chip argument takes three target values, bm1684x, bm1688 and cv186ah, and both quick start examples compile for bm1684x.

When should I use --use_history_kv in tpu-mlir?

Without it you get two instruction groups, block_ for prefill and block_cache_ for decode, which suits single-turn conversations with short context around 4K. With it you get three groups, adding block_kv_ for prefill with history, and the page recommends it for multi-turn, 8K contexts, or when unsure because it is more flexible.

How do I install tpu-mlir?

Everything runs inside the prebuilt image `sophgo/tpuc_dev:latest`, launched with `docker run --privileged --name tpu-mlir -v $PWD:/workspace -it sophgo/tpuc_dev:latest`. Inside the container you can `pip install tpu_mlir` or build from source, which the page labels recommended, with `pip install -r requirements.txt`, `source ./envsetup.sh` and `./build.sh`.

What does --chunk_length do in tpu-mlir?

It sets the segment length for chunked inference and is only meaningful with use_history_kv. The page states the arithmetic: with 1K chunks a 7K input runs prefill as block_ plus 7 times block_kv_, and decode is segmented by KV-cache length, so performance differs at 1K, 2K, 4K and 8K.

Which quantization formats does tpu-mlir support?

The highlights list F32, BF16, F16 and INT8 in both symmetric and asymmetric forms, plus AWQ, GPTQ and AutoRound passthrough, along with calibration and quantization-aware training. The three LLM methods are described as passthrough rather than re-quantized by the compiler.

What auxiliary tools does tpu-mlir ship?

The production tooling line names model_runner, model_tool, accuracy validation, a simulator and a visualizer. The repository also holds bindings/, capi/, python/, tools/, test/, unittests/, regression/ and experimental/ directories, plus separate release, release_doc and release_pip scripts and a download_toolchains.sh.

Official sources

  1. Issues
  2. README
  3. Releases
  4. sophgo/tpu-mlir on GitHub
Add this badge to your README

If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.

Add this badge to your README

markdown
[![Hysen Labs](https://hysenlabs.com/badge/sophgo-tpu-mlir.svg)](https://hysenlabs.com/projects/sophgo-tpu-mlir)

Star history chart for your README

markdown
[![Star history](https://hysenlabs.com/chart/sophgo-tpu-mlir.svg)](https://hysenlabs.com/projects/sophgo-tpu-mlir)

Was this useful?