Chinese-BERT-wwm: whole word masking checkpoints from HFL
Pre-Training with Whole Word Masking for Chinese BERT（中文BERT-wwm系列模型）
At a glance
- What is it?
- ymcui/Chinese-BERT-wwm is a set of Chinese pretrained BERT and RoBERTa checkpoints trained with whole word masking instead of per-character masking. The models are the product; the repository is the download page and the evaluation record.
- Who is it for?
- Adopt Chinese-BERT-wwm if you need a Chinese encoder whose masking unit matches word boundaries and you can fine-tune it on your own labelled data; the HF checkpoints load through standard Transformers classes. Do not adopt it if you need a masked language model out of the box, since the open release omits the MLM head weights, or if you want a model that keeps receiving pretraining updates.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 150 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What problem Chinese-BERT-wwm solves
Google's original BERT-base Chinese tokenizes text at the character level, so a Chinese word like 语言 is fed to the model as two independent units. During pretraining, the masking procedure then hides characters one at a time. The model learns to recover a character from its neighbours, which is a different task from recovering a word. Chinese-BERT-wwm changes the masking unit: the repository describes applying whole word masking to Chinese, using Chinese Wikipedia (simplified and traditional) for training and LTP as the segmentation tool, so that all characters belonging to one word are masked together. The README's own example shows the difference. The original masked input leaves pro [MASK] ##lity in place, while the whole word version masks the full span [MASK] [MASK] [MASK]. The audience is anyone fine-tuning a Chinese encoder on a downstream task: classification, sequence labelling, question answering, or as a base for further pretraining. The repository is a checkpoint distribution, not a framework. It points at the paper Pre-Training with Whole Word Masking for Chinese BERT, published in IEEE/ACM TASLP, for the method, and it ships the weights on Hugging Face and Baidu Netdisk.
How whole word masking changes the training signal
The mechanism sits entirely in the sample generation stage. WordPiece splits a word into subwords; in the original scheme those subwords are masked independently, so the model can see part of a word and infer the rest from a short local context. Under whole word masking, if any subword of a complete word is masked, every subword of that word is masked as well. The README is explicit that mask here is the broad sense: replacing with the [MASK] token, keeping the original token, or substituting a random token. It is not limited to the literal [MASK] label, and the README links issue #4 for the detailed explanation and samples. The consequence is that the model cannot lean on a surviving fragment of the same word, so it has to use the surrounding sentence. The repository ships several checkpoints that differ in architecture and corpus rather than in this masking rule. BERT-wwm is the 12-layer, 768-hidden, 12-head, 110M parameter base model trained on Chinese Wikipedia. BERT-wwm-ext and the RoBERTa-wwm-ext models use EXT data, described as Chinese Wikipedia plus other encyclopedic, news and QA data, about 5.4B words. RoBERTa-wwm-ext-large is the 24-layer, 1024-hidden, 16-head, 330M parameter variant. RBT3, RBT4, RBT6 and RBTL3 are small models built from 3, 4, 6 and 3 Transformer layers respectively.
Downloading and loading Chinese-BERT-wwm with Transformers
The README directs users to Hugging Face for the PyTorch weights and to Baidu Netdisk for the TensorFlow archives, noting that base model files are around 400M and that users in mainland China should prefer the Netdisk mirrors. The Hugging Face organization page is at https://huggingface.co/hfl; the README says to open the model you want, switch to the Files and versions tab, and download the corresponding files. The repository states that all models support TensorFlow 2 as of 2021/1/27 and can be called or downloaded through the transformers library. Loading a checkpoint is therefore a standard call, with the model name taken from the Hugging Face listing rather than invented:
A first fine-tuning run and what to expect
For a downstream task you swap AutoModel for the task head and train on your own labels, exactly as with any BERT checkpoint. The repository does not ship a training script; the README covers loading, model comparison, baseline results and usage advice, and the top-level tree contains only data/, pics/, the two READMEs, LICENSE and GitHub metadata. So the loop below is a shape, not a copied recipe, and the only project-specific parts are the checkpoint name and the tokenizer that comes with it. What you should see is a loss that falls within the first few hundred steps on a few thousand labelled examples; if it does not, the usual cause is a tokenizer or label-mapping mismatch, not the pretrained weights.
The MLM head is missing, and other limits
The most consequential limitation is stated plainly in the download section: the open release does not include the MLM task weights, and if you need the masked language model task you must pretrain further with additional data, as with any other downstream task. That rules out the obvious first experiment of probing the model's fill-in-the-blank behaviour with a standard MaskedLM head, and it makes the checkpoints unsuitable as drop-in replacements in pipelines that rely on masked prediction. A second constraint is the update cadence. The last push to this repository was on 2026-04-19, and the news list shows the model family itself was assembled between 2019 and 2021, with later entries pointing at other HFL projects. Treat the checkpoints as fixed artifacts rather than something that tracks newer pretraining corpora or architectures. Third, the repository is not a toolkit: no training script, no evaluation harness, no conversion utility beyond a pointer to the Transformers conversion scripts. If you need to convert between TensorFlow and PyTorch yourself, you are doing that work outside this project. Finally, the baseline tables in the README are the authors' reported numbers on named Chinese tasks; they are a reference point for whether a checkpoint is in the right range, not a promise about your data.
How the checkpoints compare with each other and with MacBERT
The first real choice is inside the family. BERT-wwm is the smallest corpus option, trained on Chinese Wikipedia only. BERT-wwm-ext and RoBERTa-wwm-ext move to the 5.4B-word EXT corpus, and the RoBERTa variants change the pretraining recipe further while keeping whole word masking. RoBERTa-wwm-ext-large is the 330M parameter option when accuracy matters more than latency. RBT3, RBT4, RBT6 and RBTL3 exist for the opposite case: 3, 4 and 6 layer models distilled from the larger ones, which the README groups under small-parameter models with their own effect table. The second choice is whether to stay in this family at all. MacBERT, from the same lab and linked from the README's project list, changes the masking strategy again: instead of leaving a [MASK] token in the input, it substitutes a semantically similar word, which removes the pretrain-finetune mismatch that [MASK] introduces. That is a different fix for a related problem. If your bottleneck is the [MASK] token never appearing in real text, MacBERT addresses it directly; if your bottleneck is that character-level masking ignores word boundaries, whole word masking is the targeted change. The README also links Chinese-ELECTRA, Chinese-XLNet, LERT and PERT, so the family is one option among several Chinese encoders from the same group.
Licence, maintenance and upgrade cost
The repository is licensed Apache-2.0, which permits commercial use and modification subject to the usual conditions around notices and attribution; this is a description of the licence identifier, not legal advice, and the LICENSE file in the repository root is the authoritative text. Because the project distributes weights rather than code you compile, the upgrade cost is mostly a matter of re-downloading and re-evaluating. Swapping BERT-wwm-ext for RoBERTa-wwm-ext changes the tokenizer vocabulary and the hidden representation, so any fine-tuned head you trained on one checkpoint does not transfer to another; you retrain. The TensorFlow archives on Baidu Netdisk and the PyTorch weights on Hugging Face are separate artifacts, and the README's conversion path for PyTorch is either the Transformers conversion script or a direct download from the Hugging Face listing. The repository has no release tags retrieved, so pinning means pinning the Hugging Face revision rather than a version number from GitHub.
Editorial conclusion
Adopt Chinese-BERT-wwm if you need a Chinese encoder whose masking unit matches word boundaries and you can fine-tune it on your own labelled data; the HF checkpoints load through standard Transformers classes. Do not adopt it if you need a masked language model out of the box, since the open release omits the MLM head weights, or if you want a model that keeps receiving pretraining updates. Before committing, verify the parameter count of the checkpoint you pick, confirm the tokenizer matches your text, and check whether your task is already covered in the repository's baseline tables.
Frequently asked questions
How do I download Chinese-BERT-wwm?
The README lists two routes: Hugging Face links for each checkpoint, and Baidu Netdisk archives in TensorFlow format with a password for each model. It notes that users in mainland China should prefer the Netdisk mirrors and that base model files are about 400M.
Can I use Chinese-BERT-wwm for masked language modeling?
No, not out of the box. The README states that the open release does not include the MLM task weights, and that if you need the MLM task you must pretrain further with additional data, just as for any other downstream task.
What is the difference between Chinese-BERT-wwm and Chinese-BERT-wwm-ext?
Both use whole word masking and the same base architecture, but BERT-wwm is trained on Chinese Wikipedia while BERT-wwm-ext uses EXT data, which the README describes as Chinese Wikipedia plus other encyclopedic, news and QA data totalling about 5.4B words.
Community notes