Chinese-ELECTRA: four encoder sizes, two corpus lines, and one naming trap

The generator is discarded after pretraining, yet its config still ships

ELECTRA splits pretraining in two. A small masked language model, the generator, predicts the original token at masked positions and rewrites a portion of the input. The discriminator then judges, token by token, whether each one was replaced, which is the replaced token detection objective that takes the place of BERT's masked language model. Next sentence prediction is not used at all.

After pretraining ends, only the discriminator is carried forward as the base model for downstream fine-tuning. The released archives still reflect both halves, so an unpacked checkpoint contains electra_small.meta, electra_small.index and electra_small.data-00000-of-00001 alongside vocab.txt, discriminator.json and generator.json. The config pair is treated as recoverable: if either JSON is absent from an archive, the file points you at the repository's own config/ directory instead of asking you to reconstruct it.

Two corpus lines publish the same short names at the same archive size

The download tables are organised by corpus, not by model, and that is where the naming trap sits. One line is the 180G edition, released in October 2020 with high quality CommonCrawl data added. The other is the original edition on 20G of data. Each line publishes large, base, small-ex and small, and each publishes its own hub repository, so ELECTRA-small from one line and ELECTRA-180g-small from the other are different weights that a short-name search will happily mix up.

The sizes make it worse. Both small archives are listed at 46M, both small-ex archives at 92M, and both base archives at 383M, because the architecture is identical and only the training corpus changed. Architecture is not a way to tell the two lines apart.

A third table covers the legal domain, trained on large scale legal documents and announced in December 2020. It offers large, base and small, with no small-ex entry, so the four-size grid that holds for the general models does not hold there.

Every weight comes from two places, and one of them needs an extraction code

Each row in each table carries a hub link and a cloud drive link side by side, and the choice between them is geographic rather than technical. The note attached to the tables says users inside mainland China should take the drive link and users outside should take the hub link. The drive links are not plain downloads: each one is paired with a short extraction code, five characters each, different for every file, from 2v5r for the 180G large archive to 1r4r for the original small archive.

The same two channels existed from the start, with one imbalance recorded in the project's news list. When the large and small-ex models first shipped in May 2020, only Google Drive download addresses were available, and the entry says so plainly. The model hub route arrived separately: PyTorch users were pointed at Transformers in April 2020, and the models were added to PaddleHub the following month. Anyone starting today has both, but the drive path still carries the friction it had in 2020.

The archives unpack into a TensorFlow checkpoint, not portable weights

The usage note takes the smallest model as the example and shows what the zip becomes:

chinese_electra_small_L-12_H-256_A-4.zip
    |- electra_small.data-00000-of-00001    # 模型权重
    |- electra_small.meta                   # 模型meta信息
    |- electra_small.index                  # 模型index信息
    |- vocab.txt                            # 词表
    |- discriminator.json                   # 配置文件：discriminator（若没有可从本repo中的config目录获取）
    |- generator.json                       # 配置文件：generator（若没有可从本repo中的config目录获取）

The data, meta and index trio is a TensorFlow checkpoint layout, which is why the file describing the archive names a TensorFlow version of the model specifically. There is no framework-neutral single file in the package, and no safetensors equivalent appears anywhere in the listing. A PyTorch pipeline therefore depends on the hub copy rather than the archive, and the two are distributed by different means, so it is worth confirming that the conversion you pick matches the corpus line you meant.

Hyperparameters are spelled out for two sizes and break off inside the third

The training detail section gives corpus totals first, then a per-size list. ELECTRA-large: 24 layers, hidden size 1024, 16 attention heads, learning rate 1e-4, batch 96, maximum length 512, 2M steps. ELECTRA-base: 12 layers, hidden size 768, 12 heads, learning rate 2e-4, batch 256, maximum length 512, 1M steps. The batch size and step count move in opposite directions across the two, so the small models are not simply trained longer at a smaller batch.

The third entry begins with the small-ex name and then ends. No layers, no learning rate, no step count for small-ex, and no entry at all for small. The section's own rule covers the gap: parameters that are not mentioned keep their defaults. That is a workable convention, but it means the two sizes with the most attention in the download tables are the two whose training recipe you have to reconstruct yourself.

The corpus is described by token count, with the source list left broad

The pretraining data is large scale Chinese Wikipedia plus general text, totalling 5.4B tokens, and the project states that this matches the token count of its own RoBERTa-wwm-ext series. The vocabulary is inherited rather than built: the original Google BERT WordPiece vocabulary with 21,128 tokens, unchanged.

The 180G edition is the one place where the source gets more specific. Its October 2020 entry says high quality CommonCrawl data was added, and the corpus is identified by weight, 180G, rather than by a list of datasets. Combined with the general text of the original line, that leaves the exact mixture unspecified for both. The consequences are practical rather than academic: the 5.4B token figure describes a run whose composition you cannot reconstruct from the page, and the vocabulary being Google's means any tokenizer change is a decision you make downstream, not one this project has already made for you.

An English corpus builder sits in a Chinese project's tree

The root of the repository is the upstream ELECTRA training code, forked rather than rewritten:

.github/
.gitignore
LICENSE
README.md
README_EN.md
build_openwebtext_pretraining_dataset.py
build_pretraining_dataset.py
cmrc2018_drcd_evaluate.py
config/
configure_finetuning.py
configure_pretraining.py
finetune/
model/
pics/
pretrain/
run_finetuning.py
run_pretraining.py
util/

Two files in that list came from the English model and serve no Chinese purpose. build_openwebtext_pretraining_dataset.py builds a dataset from OpenWebText, an English corpus, and there is no Chinese equivalent beside it; build_pretraining_dataset.py is the one that matters here. cmrc2018_drcd_evaluate.py scores against a Chinese reading comprehension benchmark, which is the clearest sign of which evaluation the fork is aimed at. Everything else follows the upstream layout, with pretrain/ and finetune/ split by task and util/ holding shared helpers.

The two readme files at the root are the Chinese original and its English translation, cross-linked on their first line.

The contents guide promises baseline scores and usage notes that carry no text

A table near the top indexes seven sections: an introduction to the ELECTRA principle, model downloads, quick loading, baseline system results covering reading comprehension and text classification, usage instructions, an FAQ, and citation. Two of them are written. The introduction explains the generator, the discriminator, replaced token detection and the absence of next sentence prediction. The download section carries the tables, the usage note and the training detail.

The other five are names without bodies. The prose ends inside the training detail list, at the small-ex entry. Quick loading through Transformers and PaddleHub, the baseline results for reading comprehension and text classification, the detailed usage instructions, the FAQ and the citation block are all indexed and all unwritten.

That matters most for the accuracy claims. The news list records a first place finish on the GLUE leaderboard for the lab in August 2020 and an EMNLP Findings acceptance for the paper Revisiting Pre-Trained Models for Chinese Natural Language Processing in September 2020, and the legal-domain entry links to results for the judicial models. What is not here are the numbers behind any of it, so the leaderboard is the only place to check a score. There are no GitHub releases, 0 open issues, and the last push is dated 19 April 2026.

Editorial conclusion

Chinese-ELECTRA is a sensible choice when you want a small Chinese encoder, a WordPiece vocabulary already fixed at 21,128 tokens, or a legal-domain variant, and when you would rather load weights from the model hub than train your own. It is a weaker fit if you need a current checkpoint, since the last push is dated 19 April 2026 and the newest item in the project's own news list is from March 2023.

Four things are worth confirming before you build on it. First, decide which corpus line you actually want, because the original 20G line and the 180g line publish identically named small and small-ex models at the same 46M archive size from different hub repositories, and picking by short name is how people end up fine-tuning the wrong weights. Second, check the format you are handed: the archives unpack into a TensorFlow checkpoint trio plus a config pair, so a PyTorch workflow depends on the hub conversion being present and current. Third, read the hyperparameter table for your size, because it is spelled out for large and base and simply stops inside the small-ex entry, with everything unmentioned left at defaults. Fourth, keep in mind that what you download is a discriminator trained with replaced token detection; no next sentence prediction head exists anywhere in the line.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. ymcui/Chinese-ELECTRA on GitHub