Chitu: A Chinese-Built LLM Inference Engine That Targets Domestic GPUs and MoE Models
High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.
At a glance
- What is it?
- Chitu (赤兔) is a Python-based inference framework for large language models, with a focus on DeepSeek and MoE architectures, and explicit support for Chinese GPU vendors. This review assesses its positioning, mechanism, and limitations based on the repository material.
- Who is it for?
- Chitu is for teams that need to deploy large MoE models like DeepSeek-R1 or Qwen3 on domestic Chinese accelerators (Ascend, Moore Threads, Hygon, MetaX) and want a framework that explicitly supports those targets. It is not for those seeking a general-purpose, vendor-neutral inference engine or who require extensive community support.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 6 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What Chitu Solves and Who It Is For
Chitu positions itself as a production-grade inference engine, but its real niche is narrower than that phrase suggests. The release milestones tell the story: support for DeepSeek-R1 671B, FP4 quantization, CPU+GPU hybrid inference for single-card deployment, and adaptations for Chinese GPU vendors like Moore Threads, Ascend, Hygon, and MetaX. The target user is an enterprise in China that needs to serve a large MoE model, often a DeepSeek variant, on domestic hardware. This is not a framework for hobbyists with an NVIDIA RTX card; it is built for organizations that must comply with local hardware procurement or want to avoid reliance on NVIDIA. The README's acknowledgments list Huawei, Moore Threads, Hygon, and other Chinese vendors, confirming that this project is embedded in that ecosystem. If you are outside that context, you may still benefit from its NVIDIA support, but the primary value proposition is the breadth of Chinese accelerator coverage.
How It Works: The Mechanism Behind the Claims
The repository does not expose a detailed architecture diagram in the cleaned README, so the mechanism must be inferred from the release notes and feature lists. The framework appears to be a custom inference stack built in Python, with optimized kernels for specific operations. The v0.3.0 release mentions efficient operators for online conversion from FP4 to FP8 and BF16, which is a concrete mechanism for running quantized models like the FP4 version of DeepSeek-R1. The v0.2.2 release adds CPU+GPU heterogeneous inference, which likely means the framework can offload parts of the model to CPU memory while keeping compute on the GPU, a technique used to fit a 671B model on a single card. The v0.5.0 release focuses on cluster performance, and v0.6.0 introduces chitu.run, a single executable that can launch multi-node, multi-instance, and PD (prefill-decode) separated tasks. PD separation is a known architecture pattern where prefill and decode phases run on different resources to optimize throughput. The framework also borrows ideas from DeepSeek, FlashAttention, FlashInfer, and others, per the acknowledgments, so expect a hybrid of established techniques rather than a wholly novel approach.
Getting It Running: From Releases to chitu.run
Installation is not via pip or a source build in the main README. The recommended path is to download chitu.run from the Releases page on GitHub. The v0.6.0 release notes say that chitu.run is a single file that can start complex tasks like multi-node, multi-instance, and PD separation. The README directs users to the DEVELOPMENT.md document for full instructions, but that file is not included in the cleaned material. The supported models list is in SUPPORTED_MODELS.md, which is also referenced but not provided. For those who want to build from source, the development manual is the entry point, but the README does not list any concrete commands like pip install or git clone. This is a gap for a technical audience: the framework's own documentation is the only source of truth, and it lives outside the README. If you are evaluating this project, you must fetch the full docs from the repository to get actual commands and configuration keys.
Limitations and Failure Modes
One limitation is the explicit acknowledgment that the team cannot guarantee timely responses to all issues. The README states that due to limited team energy, they cannot promise to solve every problem users encounter, and they direct professional service requests to a corporate email. This is a red flag for production adoption if you expect community support. Another limitation is that performance data is tied to specific hardware, software versions, and workloads, and the README warns that results may fluctuate across tests. That means you cannot trust third-party benchmarks; you must reproduce them on your own stack. The framework's focus on Chinese GPUs means that if you are on an NVIDIA-only deployment, you are not the primary audience, and the maturity of NVIDIA support is unclear from the material. Finally, the project is young; the first release was in March 2025, and the latest is v0.6.0 in July 2026. While rapid releases indicate active development, they also imply that APIs and features may change, and long-term stability is unproven.
Alternatives and How They Differ
The README itself lists projects that influenced Chitu, including vLLM, SGLang, TensorRT-LLM, and llama.cpp. vLLM is the most direct alternative for serving LLMs on NVIDIA GPUs, with a Python-based architecture and a focus on high throughput via PagedAttention. SGLang offers a similar serving layer but adds structured generation and a more modular design. TensorRT-LLM is NVIDIA's proprietary optimization stack, which may deliver higher performance on NVIDIA hardware but is closed-source and tied to NVIDIA. llama.cpp is a lightweight, C++-based inference engine that runs on CPUs and GPUs, but it lacks the cluster and MoE focus that Chitu claims. The key difference is that Chitu explicitly targets Chinese accelerators and MoE models like DeepSeek, while vLLM and SGLang are more general but with less native support for Ascend or Moore Threads. If you are on NVIDIA and need broad community support, vLLM is the safer bet. If you must run on Ascend 910B, Chitu appears to be one of the few frameworks with explicit support, based on the release history.
Maintenance, Upgrade Cost, and Licensing
The project is under active development with a release cadence of roughly one minor version per month: v0.5.6 in May 2026, v0.5.7 in June, and v0.6.0 in July. That pace means upgrades are frequent, and each release may introduce breaking changes, especially since the version number is still below 1.0. The introduction of chitu.run in v0.6.0 suggests a shift toward a more user-friendly deployment model, but it also means that earlier deployment methods may become deprecated. The license is Apache-2.0, which is permissive and allows commercial use, modification, and redistribution, though the repository also contains third-party submodules with their own licenses in the third_party/ directory. The README mentions that code snippets from other projects are marked with SPDX identifiers, so you must review those if you plan to redistribute the code. The maintenance cost is non-trivial: you will need to track releases, test new versions against your hardware, and possibly adapt your deployment scripts as the framework evolves. The project also relies on external contributions and the team's limited capacity, so bug fixes may not come quickly for issues that affect only your specific setup.
Editorial conclusion
Chitu is for teams that need to deploy large MoE models like DeepSeek-R1 or Qwen3 on domestic Chinese accelerators (Ascend, Moore Threads, Hygon, MetaX) and want a framework that explicitly supports those targets. It is not for those seeking a general-purpose, vendor-neutral inference engine or who require extensive community support. Before adopting, verify that your specific model and hardware combination appears in the SUPPORTED_MODELS.md and PERFORMANCE.md docs, and test with the chitu.run launcher for your exact cluster topology, as the project acknowledges performance varies with hardware and load.
Community notes