Chinese-BERT-wwm: whole word masking for Chinese BERT, shipped as downloadable checkpoints rather than code

Whole word masking changes which characters get masked together

The one idea in this project is a change to how training samples are built. Whole Word Masking was released by Google on 2019-05-31 as an upgrade to BERT. With WordPiece, a complete word is cut into sub-pieces, and the pieces were masked independently. Under whole word masking, if any piece of a word is masked, the other pieces of the same word are masked too.

The README's example shows the difference. The sentence about predicting the next word with a language model and the English word probability is segmented into words first. In the original masking, only fragments such as the second character of a compound and the middle of probability get masked, leaving pro [MASK] ##lity. In the whole word version, all three pieces of probability become [MASK], and both characters of a two character word are masked together.

There is a caveat the README flags in bold. Mask here is meant broadly: replaced with [MASK], kept as the original word, or swapped for a random word. The table only uses the [MASK] case to keep the example readable.

Chinese needs a segmenter, and this project uses HIT's LTP for it

Google's own BERT-base, Chinese splits Chinese text by single characters. It does not consider traditional Chinese word segmentation, so there is no word boundary to keep together. This project applies whole word masking to Chinese by first segmenting text with LTP from the Harbin Institute of Technology, then masking every character that belongs to the same word.

The training text is Chinese Wikipedia, covering both simplified and traditional characters, for the original BERT-wwm model. The consequence is that the quality of masking depends on the segmenter's word boundaries. Words LTP splits differently from your domain's vocabulary, such as product names or medical terms, will be masked on LTP's terms, not yours.

The project is built on Google's official BERT repository, so the checkpoints follow the original architecture. Whole word masking changes only pre-training sample generation, which is why the models can be loaded as ordinary BERT models with existing downstream code.

The checkpoint table covers eight models in two sizes

Two sizes appear in the README. BERT-base is 12 layers, 768 hidden, 12 heads and 110M parameters. BERT-large is 24 layers, 1024 hidden, 16 heads and 330M parameters. Because the directory mostly holds base models, the word base is left out of the short names, and other sizes are labeled, such as large.

The download table lists BERT-wwm, BERT-wwm-ext, RoBERTa-wwm-ext, RoBERTa-wwm-ext-large, RBT3, RBT4, RBT6 and RBTL3. BERT-wwm is trained on Chinese Wikipedia. Every other row is trained on the EXT data: Chinese Wikipedia plus other encyclopedias, news and question answering, 5.4B words in total. The BERT-wwm-ext model arrived on 2019/7/30, RoBERTa-wwm-ext on 2019/9/10 and RoBERTa-wwm-ext-large on 2019/10/14.

RBT3 and RBTL3 were added on 2020/1/20 as 3-layer versions of RoBERTa-wwm-ext-base and large. They are the options if inference cost matters more than accuracy. The base model files are about 400M each, which is worth knowing before you script a download.

The released weights have no masked language model head to use

A short bold note in the download section carries the most practical warning in the project: the open-source version does not include weights for the MLM task. If you want to do masked language modeling, you must run further pre-training on extra data, in the same way you would for any other downstream task.

The consequence is concrete. You cannot load a checkpoint, mask a character and ask the model to fill it in, expecting the pre-training head to answer. The checkpoints are meant as encoders for fine-tuning on tasks such as classification or sequence labeling.

This also matters for people who search for example scripts. The repository contains a README, an English README, a data folder and a pics folder, and no training or inference code. Whatever example you run, it comes from the Transformers library or PaddleHub, not from this repository, and the pre-training code is the Google BERT repository the project builds on.

Weights come from Hugging Face or Baidu Netdisk, and there are no GitHub releases

The repository has no GitHub releases, so the download links in the README are the only distribution channel. Every model has a Hugging Face link under the hfl organization, for example hfl/chinese-bert-wwm and hfl/chinese-roberta-wwm-ext-large, and a Baidu Netdisk link to a TensorFlow version that needs an extraction password. The password for BERT-wwm, Chinese is qfh8.

A PyTorch version is not offered on Baidu. For PyTorch you either convert the TensorFlow weights with the conversion script from Transformers, or download the PyTorch weights from huggingface.co/hfl by opening the model, choosing the Files and versions tab and downloading the files.

The README suggests Baidu Netdisk inside mainland China and the Google download point for users abroad. Models were connected to Huggingface-Transformers on 2019/12/19 and to PaddleHub on 2020/3/23, and 2021/1/27 brought TensorFlow 2 support through the transformers library. Third party hosting is a risk: a Netdisk link can change or require a login, so Hugging Face is the stable choice when you want something scripted.

The newest news item is from 2023 and the last push was on 2026-04-19

This repository is not a living codebase. The most recent entry in the news section is dated 2023/3/28, when the team open-sourced Chinese LLaMA and Alpaca models in a separate repository. The title banner at the top now points readers to Chinese-LLaMA-Alpaca-2. The last push to the master branch was on 2026-04-19, and there are no releases to track, so for a version pin you rely on the Hugging Face model pages or your own copy of the files.

The license is Apache-2.0, and the homepage field points to the IEEE paper, Pre-Training with Whole Word Masking for Chinese BERT, published in IEEE/ACM Transactions on Audio, Speech, and Language Processing by Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin and Ziqing Yang. That paper is the citation to use in your own work.

The practical effect is that you should treat the checkpoints as fixed artifacts. Issues and questions may go unanswered, and the FAQ in the README is the main support material you can count on.

Later models from the same lab sit beside this one rather than replacing it

The README links a family of related projects from the same HIT and iFLYTEK Joint Lab: LERT, which brings in linguistic information and was announced on 2022/10/29, PERT, released on 2022/3/30, MacBERT, Chinese ELECTRA and Chinese XLNet. It also points to the TextBrewer distillation toolkit and the TextPruner pruning toolkit, and to the MiniRBT small model from 2022/11/15 and the VLE image and text model from 2023/3/9.

This changes how you choose. If you want the reference whole word masking baseline cited in the paper, use this repository. If you want a smaller model, the README's own answer is the 3-layer RBT3 and RBTL3 here, or MiniRBT. Which of the later models beats these checkpoints on your task is not something the README settles, so run your own evaluation on your data.

In 2020 the README also recorded a GLUE leaderboard first place for the lab and an EMNLP Findings acceptance for the earlier paper Revisiting Pre-Trained Models for Chinese Natural Language Processing. These are historical claims tied to those dates, not current rankings.

Editorial conclusion

Use Chinese-BERT-wwm as a baseline Chinese encoder when you want TensorFlow or Hugging Face weights with a published paper behind them and are ready to fine-tune. Do not expect a fill-mask model out of the box, do not expect new work here, and check the Hugging Face pages directly because the repository holds a README, not weights.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. ymcui/Chinese-BERT-wwm on GitHub