Chinese-LLaMA-Alpaca-3: adapting Meta Llama 3 to Chinese without touching the tokenizer

What Chinese-LLaMA-Alpaca-3 adds on top of Meta Llama 3

Meta's Llama 3 ships with weak Chinese coverage relative to its English performance. The third phase of the Chinese-LLaMA-Alpaca series addresses that gap by releasing Llama-3-Chinese-8B, a base model, and Llama-3-Chinese-8B-Instruct, an instruction-tuned chat model, both derived from the 8B Llama 3 checkpoints. The README describes the method as incremental pretraining on large-scale Chinese data followed by fine-tuning on curated instruction data, and states that the resulting models improve Chinese semantic and instruction understanding over the second-generation Chinese Llama models.

The audience is narrow and identifiable. If you are building a Chinese-language assistant and want to stay inside the Llama 3 ecosystem (transformers, llama.cpp, vLLM, Ollama, text-generation-webui are all named as supported), this project gives you weights that plug into that toolchain rather than a bespoke inference stack. It is not aimed at teams training frontier models from scratch, and it is not a general multilingual project. Everything here is oriented around Chinese.

The repository is not only weights. It also publishes pretraining and instruction fine-tuning scripts and three instruction datasets: alpaca_zh_51k, stem_zh_instruction, and ruozhiba_gpt4 in both 4o and 4T variants. That combination matters if you intend to continue training rather than just serve the released checkpoints.

The tokenizer decision: keep Llama 3's 128K vocabulary

The most consequential design choice in this project is a negative one. Earlier Chinese Llama adaptations typically expanded the vocabulary with Chinese tokens and then resized the embedding matrix. This project does not. The README states plainly that the authors did not perform additional vocabulary expansion.

The justification is empirical and specific. Llama 3 grew the vocabulary from 32K to 128K and switched to BPE. The project reports that initial experiments found Llama 3's tokenizer encodes Chinese at roughly 95 percent of the efficiency of the expanded Chinese LLaMA-2 vocabulary, measured on Wikipedia data. Combined with conclusions drawn from the authors' earlier Chinese Mixtral work, cited as Cui and Yao, 2024, that was enough to skip expansion.

Skipping expansion has concrete downstream benefits. The embedding matrix stays the same size as Meta's, so you do not need to handle resized embeddings in conversion scripts, quantization tools, or serving frameworks. Any tool that already understands a stock Llama 3 checkpoint should load these weights without special-casing. The cost is a modest encoding inefficiency on Chinese text, which translates into slightly more tokens per Chinese sentence and therefore slightly more compute per unit of text. Whether 95 percent is acceptable depends on your workload. For most chat and Q&A applications it is a rounding error; for high-volume Chinese document processing where token count drives cost, it is a real line item.

Three instruct models, three different training recipes

The project released three instruct versions within about a month, and they are not iterations of the same pipeline. Understanding the difference matters because the README recommends v3 by default but the three were built differently.

Instruct-v1, released 2024/4/30, starts from the original Meta-Llama-3-8B base model and runs two stages: 120GB of Chinese pretraining followed by fine-tuning on roughly 5 million instruction examples. Instruct-v2, released 2024/5/8, skips the pretraining stage entirely and fine-tunes directly on Meta-Llama-3-8B-Instruct with the same 5 million instruction examples. Instruct-v3, released 2024/5/30, is produced by merging inst-v1, inst-v2 and inst-meta, then applying a small amount of additional instruction tuning on roughly 5,000 examples.

That third recipe is the one worth pausing on. Model merging plus a light fine-tune pass is a cheap way to combine the strengths of a Chinese-pretrained model and a Meta-instruct model, and it avoids the cost of another full pretraining run. It also means v3's behaviour is harder to reason about than v1's: you cannot point to a single training corpus and predict what the model learned. The README claims v3 shows significant improvement over v1 and v2 on downstream tasks. That claim is the project's own; treat it as a hypothesis to test on your data, not a settled result.

Both base and instruct models use LoRA plus full embedding and LM-head training, and both support an 8K context window, up from 4K in the second generation.

Getting the weights running: transformers, llama.cpp, vLLM

The README points to a wiki for detailed instructions and lists supported runtimes: transformers, llama.cpp, text-generation-webui, vLLM, and Ollama. Model downloads are hosted on Hugging Face under the hfl organisation, ModelScope under ChineseAlpacaGroup, and wisemodel. There is also a hosted demo Space at hfl-rc/llama-3-chinese-8b-instruct-demo, which is the fastest way to see output quality before downloading anything.

The single most important operational detail is the prompt template. The README states that Llama-3-Instruct uses a new instruction template that is not compatible with Llama-2-chat, and that users must follow the official template. The base model requires no template; the instruct model does. If you load an instruct checkpoint into a serving framework that defaults to a Llama 2 chat format, you will get degraded output and may misattribute it to the model. Check what your framework applies before you evaluate.

For local deployment the README describes tutorials for quantizing and running the models on a personal computer using CPU or GPU. Those tutorials live in the wiki rather than the README, so the exact quantization commands are not reproduced in the repository front page. The training and fine-tuning scripts are in the repository itself, and the released instruction datasets (alpaca_zh_51k, stem_zh_instruction, ruozhiba_gpt4) are the intended inputs for reproducing or extending the instruct tuning. I have not run any of these scripts, so I cannot speak to their resource requirements or whether they work unmodified on current library versions.

Where this project stops being the right tool

The context window is 8K and that is the native limit. The README notes that users can extend it with PI, NTK, or YaRN, but that is work you do yourself, and long-context extension methods often degrade short-context quality in ways that need their own evaluation. If your application routinely handles long Chinese documents, you are looking at a project that gives you a starting point, not a finished long-context model.

The project is also a snapshot, not a maintained line. The last release, v3.0, is dated 2024/5/30. The repository shows a later push timestamp, but no model release after v3.0 is listed. Anyone planning to depend on this for a production Chinese model should read the release list as evidence that development has moved on, and should check whether the sibling projects in the series (Chinese-Mixtral, Chinese-LLaMA-Alpaca-2) are more actively maintained for their use case.

There is a subtler limitation in the evaluation story. The README excerpt includes a comparison table with a Chinese-ability row showing figures like 49.3 / 51.5 for v1 and a truncated value for v2. I cannot see the full table or the benchmark definitions from the supplied material, so I cannot tell you what those numbers measure or whether the comparison is fair across the three training recipes. Anyone making an adoption decision on the strength of those numbers should locate the full table and the task definitions in the wiki first.

How this differs from vocabulary-expansion Chinese LLM forks

The obvious alternative approach is the one the Chinese-LLaMA-Alpaca series itself used in its first two phases: extend the tokenizer with Chinese tokens, resize the embedding matrix, and continue pretraining. That path buys better Chinese encoding efficiency, at the cost of a checkpoint that no longer matches stock Llama 3 dimensions. Every downstream tool then needs to know about the resized vocabulary, and merging LoRA adapters or converting to GGUF becomes a project-specific task.

This project deliberately takes the other branch. Its own second-phase models used the expanded-vocabulary approach; phase three abandoned it after measuring that Llama 3's native tokenizer was close enough. That is a meaningful reversal, and it is documented with a specific measurement rather than a preference. If you are choosing between the two, the trade is roughly: expanded vocabulary gives you fewer tokens per Chinese sentence and a bespoke artifact; native vocabulary gives you a stock-shaped artifact and slightly higher token counts. For a team that wants to move between vLLM, llama.cpp and Ollama without maintaining conversion patches, the second option is usually the cheaper one to own.

Licensing and the cost of staying current

The repository code is Apache-2.0, which is permissive and unsurprising. The weights are a different matter. The models are developed from Meta Llama 3, and the README links directly to Meta's Llama 3 repository as the upstream. Meta's Llama 3 community licence carries its own conditions, which are not the same as Apache-2.0. This is not legal advice, and I have not read the full licence text for this release; the practical point is that you cannot assume the Apache-2.0 badge on the repository governs the checkpoints you download. Check the licence field on the specific Hugging Face or ModelScope model page.

On maintenance cost, the honest read is that this is a frozen artifact. There is no indication of an upgrade path beyond v3.0, and the training recipes differ enough between v1, v2 and v3 that upgrading between them is not a drop-in swap: v2 changes the base model, v3 changes the training method entirely. If you fine-tune on top of one version, moving to the next means redoing that work. Budget for picking one version and staying on it, and verify that choice against your own evaluation set before you invest in fine-tuning.

Editorial conclusion

Adopt it if you need a Chinese-capable 8B model that runs inside the standard Llama 3 toolchain and you are willing to accept Meta's Llama 3 community licence on top of this repository's Apache-2.0 code. Do not adopt it if you need a Chinese model with an expanded tokenizer, a context window beyond 8K out of the box, or a project that still receives model releases. Before committing, verify three things yourself: the exact licence terms attached to the specific Hugging Face or ModelScope checkpoint you download, the instruction template your serving stack applies, and whether the v3 instruct weights beat v1 or v2 on your own Chinese evaluation set rather than on the project's reported numbers.

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. ymcui/Chinese-LLaMA-Alpaca-3 on GitHub