TransArch: A Monorepo of Attention-Rewrite Methods, Not a Single Tool
Design hardware-friendly model architectures and migrate existing LLMs with minimal performance loss
At a glance
- What is it?
- MuLabPKU's TransArch bundles six research projects that convert or compress attention so pre-trained LLMs run better on specific hardware. The README frames it as a collection, and that framing is the most important thing to understand before you try to install anything.
- Who is it for?
- Adopt TransArch only if you are evaluating or reproducing one of its named papers and can work from the arXiv listings, because the README documents no install command, no config keys and no licence. Teams that need a supported conversion utility with a versioned release should not start here.
- Can I use it commercially?
- Not without permission. GitHub finds no licence file in the repository, and without a licence all rights are reserved by default: you may read the code but not reuse it. Check the README, or ask the authors, before using it.
- Is it still maintained?
- Yes. The repository last received commits 64 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What TransArch actually is: six papers behind one repository name
The README opens by describing TransArch as "a collection of research projects" that design hardware-friendly architectures and migrate pre-trained models into them. The projects table lists six entries: CLOVER (ICML 2025), TransMLA (NeurIPS 2025 Spotlight), TPLA (ASPLOS 2026), HISA (COLM 2026), MISA (preprint) and GQLA (preprint). Each has its own arXiv identifier and its own citation block at the bottom of the README. That structure matters more than any individual method, because it tells you the unit of consumption here is a paper, not a package. There is no single `transarch` entry point described, no shared abstraction the README names, and no version string. The repository topics are attention, deepseek and mla, which points at the common thread: all six projects rewrite or compress the attention mechanism. The problem they address is stated in the first paragraph, that modern LLMs are "increasingly bottlenecked by communication rather than computation." If your workload is compute-bound rather than communication-bound, the premise does not apply to you.
TransMLA: the GQA-to-MLA conversion path and its Absorb compatibility claim
TransMLA is the project with the clearest migration story in the README. It converts GQA models into DeepSeek-MLA, and the README names LLaMA, Qwen and Mixtral as source families. The phrase to read carefully is "full Absorb compatibility." Absorb is the trick that folds the up-projection matrix into the output projection so decoding does not pay for it, and compatibility with it is what separates a conversion that speeds up inference from one that merely reshapes weights. The README states up to 10.6x inference speedup, a figure that comes from the paper, not from any benchmark run in the repository. The news section claims adoption by Ant Group's Ling-2.5-1T model, described as a 1T parameter model, and the README's own gloss is that this "demonstrates the robust scalability of TransMLA in ultra-large-scale LLMs." Treat that as a vendor-side adoption claim with a Hugging Face link attached, which is checkable, rather than as independent validation. What the README does not give is the conversion command, the expected checkpoint format, or a list of which GQA configurations survive conversion without retraining.
CLOVER, TPLA, HISA, MISA, GQLA: what each one changes
CLOVER is cross-layer SVD pruning of Q-K and V-O pairs inside attention heads, and the README says it is combined with TransMLA to reach up to 11.1x speedup over LLaMA-2-7B. Note that this number is a combination result, so CLOVER alone does not carry it. TPLA partitions latent representations across devices for tensor parallelism, with 1.79x and 1.93x reported on DeepSeek-V3 and Kimi-K2; its code is still listed as unreleased in the to-do section, so anyone drawn to TPLA today has a paper and no implementation. HISA is a two-stage hierarchical indexer for fine-grained sparse attention, reported at 2x to 4x speedup in the 32K to 128K context range, and the news section says its indexing is used in LongCat-2.0's LongCat Sparse Attention. MISA routes over DSA indexer heads with a mixture-of-experts scheme, described as matching dense DSA with 8x or 4x fewer indexer heads and roughly 3.82x kernel speedup on H200. GQLA is the most unusual entry: one set of weights with two decoding paths, MQA-absorb for H100 and GQA plus multi-token prediction for H20, with up to 8-way tensor parallelism. GQLA is a single-author preprint, which is a different maturity signal from the conference-track entries.
The hardware-specificity is the design, and also the constraint
GQLA is the clearest illustration of what "hardware-aware" means in this repository. The same weights decode differently depending on whether you are on an H100 or an H20, because the two GPUs reward different attention shapes. That is a deliberate trade: you accept a more complex serving path in exchange for throughput on a known device. The same pattern runs through the other entries. MISA's speedup figure is tied to H200, TPLA's to DeepSeek-V3 and Kimi-K2, CLOVER's to LLaMA-2-7B. None of these numbers transfer to a different accelerator or a different base model without re-measurement, and the README does not offer a portability table. If your deployment target is not one of the named devices, or your base model is not one of the named families, you are outside the documented envelope. There is also a structural cost that the README does not discuss: converting a GQA checkpoint to MLA changes the KV cache layout, so any serving stack, quantisation scheme or cache-eviction policy written against GQA needs to be revisited.
Getting it running: what the README does and does not tell you
This is the weakest part of the material. The README contains no installation command, no `pip install` line, no `requirements.txt` reference, no config keys, no CLI flags and no example invocation. What it does give you is a directory pointer, `HISA_COLM_2026/`, in the news entry announcing HISA's acceptance, which suggests the layout is one directory per paper rather than one installable package. The citation blocks give you the paper titles and venues to search for. The to-do list contains exactly one item, "Release TPLA code," which tells you the maintainers track code release as a checklist item and that at least one project is not yet available. If you want to run any of this, the realistic path is to clone the repository, read the per-paper directory, and follow the instructions inside that directory, because the top-level README is an index rather than a manual. I could not confirm from the supplied material whether any per-paper directory contains a working training or conversion script.
Licence, maintenance and the cost of tracking a moving monorepo
The licence is unknown. The repository metadata supplied does not include a licence identifier, and the README does not mention one, so anyone planning to ship TransMLA-derived weights in a product has an unresolved question that only the repository itself can answer. Check for a `LICENSE` file at the root and inside each per-paper directory before you build on anything, because a monorepo can carry different terms per subdirectory and the paper licences may differ from the code licences. On maintenance: the last push is dated 2026-07-14, the news entries run from 2025.05 through 2026.07.09, and there are no retrieved releases, so there is no tagged version to pin against. Upgrading means tracking `main`. For a research monorepo that is normal; for something you vendor into a serving stack it means every pull is an unreviewed change. The paper cadence also implies churn: five of the six projects have appeared or been accepted within roughly a fourteen-month window, and new entries can arrive without notice.
Alternatives: what you would use instead, and why the approach differs
If your goal is long-context inference rather than attention re-architecture, FlashAttention is the obvious comparison and the difference is philosophical. FlashAttention keeps the attention math identical and rewrites the kernel to avoid materialising the full score matrix in high-bandwidth memory; TransArch's methods change the architecture itself, so the model you serve is not the model you downloaded. That means FlashAttention drops into an existing serving stack with no weight conversion, while TransMLA and CLOVER require you to produce a new checkpoint and validate that quality held. The other comparison is to the original MLA implementation from DeepSeek, which is a single architecture rather than a conversion path: DeepSeek's MLA is trained as MLA, whereas TransMLA's contribution is the claim that you can get there from an already-trained GQA model. If you are training from scratch and have already chosen MLA, TransMLA has nothing to offer you. If you have a fleet of GQA checkpoints and no budget to retrain, that is precisely the gap it targets.
Editorial conclusion
Adopt TransArch only if you are evaluating or reproducing one of its named papers and can work from the arXiv listings, because the README documents no install command, no config keys and no licence. Teams that need a supported conversion utility with a versioned release should not start here. Before writing any code against it, verify three things directly in the repository: whether a licence file exists, which subdirectories actually contain runnable code versus paper artefacts, and whether the TPLA code release has landed, since the to-do list still shows it unchecked.
Community notes