Chinese-XLNet: pretrained Chinese XLNet weights from HFL, and when to pick them over BERT

What Chinese-XLNet actually ships, and who the checkpoints are for

The repository is a distribution point for two Chinese pretrained XLNet models, built on CMU and Google's official XLNet implementation. XLNet-base is 12 layers, 768 hidden units, 12 heads and 117M parameters. XLNet-mid is 24 layers, 768 hidden units, 12 heads and 209M parameters. Both were trained on Chinese Wikipedia plus general data covering encyclopedic, news and question-answering text, a corpus the README puts at 5.4B words and describes as identical to the one used for BERT-wwm-ext.

The intended reader is someone who already has a Chinese NLP task and wants a pretrained starting point that is not BERT. The README frames the project as enriching Chinese NLP resources and offering a diversified choice of pretrained models, which is an honest description of scope: this is a model release, not a framework. If you are looking for a training pipeline you can point at your own corpus, the repository does not present itself that way. The src/ directory and the preprocessing notes are oriented toward reproducing the released weights, not toward general-purpose pretraining.

The mid model is the interesting one. It is roughly 1.8 times the parameter count of base, and the baseline tables in the README show that most of the headline gains come from mid rather than base. Choosing between them is a memory and latency decision, not a quality decision, and the README does not offer guidance on either. That omission is worth noting before you commit to 209M parameters.

How the XLNet objective differs from BERT, and why that shows up unevenly

XLNet is a generalized autoregressive pretrainer. BERT masks tokens and predicts them independently, which is why BERT fine-tuning code needs a mask token at inference time that never appeared during downstream training. XLNet instead maximizes the expected likelihood over all permutations of the factorization order, so it sees every token position as a prediction target without a fixed mask pattern. The practical consequence for the reader is that XLNet is not a drop-in architectural twin of BERT, and the repository's numbers reflect that.

Look at CMRC 2018, the simplified Chinese reading comprehension set. XLNet-mid reaches 29.1 EM and 55.8 F1 on the challenge set, against 24.0 and 47.3 for BERT-wwm-ext. On the test set, though, BERT-wwm-ext posts 71.4 EM and 87.7 F1 while XLNet-mid gets 69.3 and 89.2. So XLNet-mid wins on F1 and loses on exact match. That pattern repeats on DRCD, the traditional Chinese set: XLNet-mid leads at 85.5 EM and 93.6 F1 on test, ahead of BERT-wwm-ext at 83.6 and 90.4.

The honest reading is that XLNet-mid is a strong span extractor that is somewhat less inclined to produce the exact gold string. If your metric is F1 or your downstream use tolerates partial spans, that is fine. If your product requires exact answer strings, the EM gap on CMRC 2018 test is a real consideration, and the README does not explain it.

On ChnSentiCorp sentiment classification, the README lists XLNet-mid at 95.8 development accuracy and 95.4 test accuracy, with BERT-wwm at 95.4 development and 95.0 test. The XLNet-base row in that table is left blank. That is a gap in the published results, not a result, and you should not read the blank as a failure.

Installing Chinese-XLNet and running a first forward pass

There is no pip package for the weights themselves. The README's quick-load path goes through Hugging Face Transformers, which the project cites at version 2.2.2, and the model identifiers are hfl/chinese-xlnet-mid and hfl/chinese-xlnet-base. The README gives this example, with MODEL_NAME standing in for the identifier from its table:

tokenizer = AutoTokenizer.from_pretrained("MODEL_NAME")
model = AutoModel.from_pretrained("MODEL_NAME")

The table above maps XLNet-mid to hfl/chinese-xlnet-mid and XLNet-base to hfl/chinese-xlnet-base. The first call downloads the tokenizer vocabulary, the second the weights. If the download completes, you have a working encoder.

If you prefer the TensorFlow checkpoint, the README points to Baidu Netdisk links, and it notes that the mid model archive is about 800M. The README lists the extracted contents of the zip as follows:

chinese_xlnet_mid_L-24_H-768_A-12.zip
    |- xlnet_model.ckpt
    |- xlnet_model.meta
    |- xlnet_model.index
    |- xlnet_config.json
    |- spiece.model

Those five files are what the README lists after extraction: weights, meta, index, config and the SentencePiece vocabulary. The README also states that as of 2021-01-27 all models support TensorFlow 2 and should be called or downloaded through the transformers library. For PyTorch weights outside Hugging Face, the README instructs you to run the conversion script that Transformers provides, or download the bin and json files directly from the model page on huggingface.co. It does not provide a conversion script of its own.

One practical note the README gives: users inside mainland China are advised to use the Baidu Netdisk download, users outside to use the Google-hosted option. That is about transfer speed, not about which artifact is correct.

The fine-tuning gap is the repository's biggest limitation

The README has a section titled downstream fine-tuning details in its table of contents, and the baseline results are reported for CMRC 2018, DRCD and ChnSentiCorp. What it does not present is a complete fine-tuning script you can copy and run. The preprocessing material it does show is about building pretraining data, not about adapting the released checkpoint to a task.

This matters because XLNet fine-tuning is not identical to BERT fine-tuning. The input format, the handling of the segment structure and the optimizer settings differ, and the official XLNet repository is the place those details live. The README links to that repository at the top as the base the project builds on. If you arrive at Chinese-XLNet expecting a run_classifier.py equivalent, you will need to look at the upstream XLNet code and adapt it yourself.

The second limitation is the baseline coverage. Three datasets is a narrow slice of Chinese NLP. The README says so directly, noting that time and effort were limited and that the authors did not cover more task categories, inviting users to try for themselves. That is a fair statement, but it means you cannot assume the CMRC and DRCD ordering generalizes to named entity recognition, parsing or generation. XLNet's permutation objective is a different inductive bias, and the direction of the difference is task-dependent.

The third is reproducibility of the reported numbers. The README states that for each model the authors ran 10 times with different random seeds and report the maximum and the average, with the average in parentheses. So a single run of yours landing below the parenthesized figure is not evidence of a bug. It is the expected spread.

Chinese-XLNet versus Chinese BERT-wwm-ext: the actual difference

The most useful comparison is not XLNet against BERT in the abstract but Chinese-XLNet against Chinese-BERT-wwm, the sibling project from the same lab. The README's baseline tables include BERT, BERT-wwm and BERT-wwm-ext rows alongside the XLNet rows, and it attributes the BERT numbers to the Chinese-BERT-wwm project.

The architectural difference is the pretraining objective: masked language modeling with whole-word masking for BERT-wwm, permutation-based autoregressive modeling for XLNet. The corpus difference is smaller than you might expect, since the README states the general data used here is the same as the one behind BERT-wwm-ext. So the comparison largely isolates the objective.

On DRCD test F1, XLNet-mid scores 93.6 against 90.4 for BERT-wwm-ext. On CMRC 2018 test F1, 89.2 against 87.7. On CMRC 2018 test EM, 69.3 against 71.4. On ChnSentiCorp test accuracy, 95.4 against 95.0. The pattern is a consistent F1 edge for XLNet-mid and a mixed picture on exact match and classification, where the margins are small enough that seed variance could matter.

There is also a cost difference the tables do not show. XLNet-mid is 209M parameters against roughly 102M for BERT-base-class models, and XLNet's relative position encoding and permutation training make it slower to fine-tune in practice. The README reports no training time or throughput figures, so you have to measure that on your own hardware. If your constraint is inference latency on CPU, the 117M XLNet-base is the more comparable option to BERT, but the README leaves its ChnSentiCorp row empty and its CMRC 2018 numbers trail XLNet-mid on every column.

Licence, maintenance and the cost of upgrading

The repository is Apache-2.0. That is a permissive licence, and it is the same licence family as many of the sibling HFL projects. It does not, by itself, settle the terms of the upstream XLNet implementation or of the pretrained corpus, and the README does not discuss those. If you are shipping a product, check the upstream XLNet repository's terms and your own legal counsel's reading rather than treating the badge as the whole answer.

On maintenance: the last push to this repository was on 2026-04-19. It is not archived. The news section is dominated by announcements of other HFL projects, including Chinese-LLaMA-Alpaca in 2023, LERT in 2022 and PERT in 2022, which tells you where the lab's attention has gone. The Chinese-XLNet weights themselves are a finished artifact; there is no evidence in the README of an upgrade path, a versioned checkpoint series, or a deprecation notice.

The practical upgrade cost is therefore low but also static. The weights will not change under you, which is good for reproducibility. The risk sits in the dependency chain instead: the README's quick-load example cites Transformers 2.2.2, a version from 2019. Newer Transformers releases have changed tokenizer and model loading behavior across major versions, and the README does not document which current versions are known to work. Pinning your Transformers version is the conservative move, and testing the load path is the first thing to do after any dependency bump. There are no retrieved releases for this repository, so there is no changelog to consult.

Editorial conclusion

Adopt Chinese-XLNet if you need a Chinese pretrained checkpoint whose masked-language-model assumptions differ from BERT and you can load it through transformers with hfl/chinese-xlnet-mid or hfl/chinese-xlnet-base; the repository's own tables show XLNet-mid leading on CMRC 2018 challenge and DRCD F1, so reading comprehension and span extraction are the strongest cases. Do not adopt it if you want a training or fine-tuning framework: the repository publishes weights and preprocessing notes, and the README's fine-tuning section is where you should look before assuming a script exists. Verify first that the checkpoint name you intend to call matches the table entry, and that your inference stack handles the 209M-parameter mid model rather than the 117M base.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. ymcui/Chinese-XLNet on GitHub