Chinese-Mixtral activates two of eight experts per token

Eight experts per layer, two chosen per token

The architecture is the reason a 46.7B parameter model behaves like a 13B one. Each feed-forward layer holds 8 different experts, which are fully connected layers, and gating values select the best 2 of them to activate.

The selection is per token, not per sequence. Every token in the input independently picks its experts, rather than the whole sequence sharing one set, which is the structural difference the project highlights against mainstream models in the LLaMA line.

Those two facts produce the headline numbers: about 46.7B parameters in total, of which about 13B are active during inference. The same shape appears in the model comparison table for both checkpoints, 8x7B with roughly 13B activated and 8 experts with 2 activated.

The Mixtral paper's structural diagram is reproduced in the documentation alongside this description, and the technical report for the Chinese adaptation is a separate paper by Cui and Yao, Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral.

Continued pretraining first, instruction tuning second

Two checkpoints come out of one base, and the difference between them is more than a name.

Chinese-Mixtral is the base model, trained by continued pretraining on large-scale unlabeled Chinese data starting from the original Mixtral-8x7B-v0.1. Its training type is causal language modelling, its corpus is unlabeled general text, and it needs no input template. Its use case is text continuation: given an opening, produce the rest.

Chinese-Mixtral-Instruct is built on top of that base model through instruction tuning on labelled instruction data. It expects the Mixtral-Instruct template to be applied to inputs, and its use case is instruction understanding, which the table lists as question answering, writing, chat and interaction. If you want a chat model, this is the one to take.

Both used QLoRA plus full embedding and lm-head training, both kept the original 32000 vocabulary, and both are described as gaining notable improvements in mathematical reasoning and code generation.

32K of native context, with 128K measured

Context handling is where this family differs from the author's earlier work. The documentation contrasts Chinese-Mixtral with Chinese-LLaMA-Alpaca and Chinese-LLaMA-Alpaca-2 on this point: those projects extended the context, whereas Mixtral supports 32K natively.

Beyond that, the project reports a measured figure of up to 128K. The distinction between the native and measured numbers is stated rather than blurred, and the practical argument given is that one model can cover tasks of different lengths instead of you keeping several checkpoints for different jobs.

Both checkpoints carry the same context row in the comparison table, 32K with up to 128K measured, so the choice between base and Instruct has no effect on how much text you can feed the model.

Three downloads, from 2.4 GB to 87 GB

The same weights are published in three forms, and the choice is about bandwidth rather than capability.

The full model is 87 GB, downloads ready to use, and needs no merge step. It is the option for anyone with the bandwidth. The LoRA version is 2.4 GB, cannot be used on its own, and has to be merged with the original Mixtral-8x7B-v0.1 to become the full model, with the merge procedure documented in the project wiki. That path is aimed at users with limited bandwidth who already hold the original Mixtral. The GGUF version is the quantized form for inference, compatible with llama.cpp and similar tools.

A news entry from March 27, 2024 records that 1-bit, 2-bit and 3-bit quantized GGUF builds were added on the Hugging Face hub, which is what makes the low end practical: the project states that quantized inference through llama.cpp needs as little as 16G of memory or video memory. Downloads are hosted on Baidu, Hugging Face and ModelScope.

Pinned to the 2024 stack, with serving paths listed

The training requirements are pinned tightly. requirements.txt lists peft 0.7.1, transformers 4.37.2, bitsandbytes 0.42.0 and sentencepiece 0.1.99, plus torch 2.0.1 or newer, with datasets and safetensors unpinned. That is a coherent set for the QLoRA work the project describes, and it is also a stack from early 2024.

Serving is a different matter, because the project names its supported ecosystem rather than pinning it: transformers, llama.cpp, text-generation-webui, LangChain, privateGPT and vLLM. A deployment mode imitating the OpenAI API was added on March 26, 2024 in the v1.2 release, so an existing client can point at it without speaking a private protocol.

The repository also open-sources the pretraining and instruction-tuning scripts themselves, with the stated purpose that users can train or fine-tune further. The examples directory holds evaluation write-ups, including a multi-turn conversation example and comparisons against Alpaca, one of them scored by GPT-4.

A 2024 release line with a 2026 commit

The version history is short and dated. v1.0 on January 29, 2024 released the base model and the Instruct model. v1.1 on March 5, 2024 open-sourced the training and fine-tuning code and published the technical report. v1.2 on March 26, 2024 added the OpenAI-style deployment mode. The quantized GGUF builds followed the next day.

After that the project moved sideways rather than forward. The most recent news entry is from April 30, 2024 and announces a different repository, Chinese-LLaMA-Alpaca-3, with Llama-3 based Chinese 8B models. The last push to the main branch of this repository is dated April 19, 2026.

Documentation is bilingual, with a Chinese README and an English one, and there is a wiki, an issue tracker, a discussions board and a leaderboard arena hosted on the author's own domain. The broader set of related projects by the same author and collaborators spans Chinese LLaMA and Alpaca generations, LERT, PERT, MacBERT, Chinese ELECTRA, Chinese XLNet and Chinese BERT, plus the TextBrewer distillation and TextPruner pruning tools.

Editorial conclusion

Chinese-Mixtral fits a team that wants a sparse MoE model that handles Chinese natively, needs either text continuation or chat from the same family, and wants the training and fine-tuning scripts rather than weights alone. It does not fit anyone who needs a current toolchain, since the pinned stack is transformers 4.37.2 with peft 0.7.1 and bitsandbytes 0.42.0. Before you commit disk and time, decide which of the three forms you actually need, because the 2.4 GB LoRA is unusable on its own without the original Mixtral, and check that your inference path, transformers, llama.cpp or vLLM, is one the project lists.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. ymcui/Chinese-Mixtral on GitHub