BertWithPretrained: A Reading Companion for BERT, Not a Production Library

What BertWithPretrained Is For, and Who It Is Written For

The README states the goal plainly: this is an implementation of the BERT model and its downstream tasks on PyTorch, and it also includes an explanation of the model and the principles behind each underlying task. That second clause is the point. The repository is not trying to replace a model hub or a training framework. It exists so that someone can open model/BasicBert/MyTransformer.py, read the self-attention code, then open Tasks/TaskForSQuADQuestionAnswering.py and see how the same encoder is wrapped for span extraction. The intended reader is a student or an engineer who already knows Transformers and wants to see BERT assembled from parts. The README even sets a prerequisite: three sibling projects covering translation, classification and couplet generation are named as background reading before starting. That is an unusual thing for a README to demand, and it tells you the author assumes a specific learning path rather than a general audience. The seven checked items in the Implementations list define the scope: the base model, single-sentence classification in Chinese, MNLI, SWAG, SQuAD v1.1, NSP plus MLM pretraining, and Chinese NER.

The Module Split: BasicBert, DownstreamTasks, Tasks, and utils

The layout separates four concerns. model/BasicBert holds the encoder itself: MyTransformer.py for self-attention, BertEmbedding.py for input embeddings, BertConfig.py to load config.json, and Bert.py to assemble them. model/DownstreamTasks holds one file per head: BertForSentenceClassification.py, BertForMultipleChoice.py, BertForQuestionAnswering.py, BertForNSPAndMLM.py, BertForTokenClassification.py. The Tasks directory holds the runnable scripts, one per dataset, and utils holds the data plumbing: data_helpers.py for preprocessing and dataset construction, log_helper.py for logging, and creat_pretraining_data.py for building pretraining examples. The data flow is therefore conventional: a Task script loads a dataset through data_helpers, instantiates the matching head from DownstreamTasks, and the head wraps the encoder from BasicBert. Two pretrained checkpoints are vendored in the repository, bert_base_chinese and bert_base_uncased_english, each with its configuration files, so the English tasks and the Chinese tasks draw from different starting weights. Note that the data/ tree mixes languages deliberately: SingleSentenceClassification is a 15-class Chinese Toutiao dataset, PairSentenceClassification is MNLI, MultipeChoice is SWAG, SQuAD is SQuAD-V1.1, and WikiText and SongCi are the English and Chinese pretraining corpora respectively. The directory name MultipeChoice is misspelled in the README, which is the kind of detail that matters when you are scripting a path.

Running a Task: Commands, Environment Pins, and the Download Step

The README splits usage into two steps. Step 1 is to download each dataset and the corresponding BERT pretrained model where the directory is empty, placing them in the matching data subdirectory; the README points to a README.md inside each data directory for specifics. Step 2 is to change into the Tasks directory and run the script. The four documented invocations are `python TaskForSingleSentenceClassification.py`, `python TaskForPairSentenceClassification.py`, `python TaskForMultipleChoice.py`, and, per the implementation list, `python TaskForSQuADQuestionAnswering.py`, `python TaskForPretraining.py` and `python TaskForChineseNER.py`. There is no CLI argument parsing described, no config flag mentioned, and no entry point beyond editing and running the script, so per-task settings live in the script itself. The environment is pinned hard: Python 3.6 with torch 1.5.0, torchtext 0.6.0, torchvision 0.6.0, transformers 4.5.1, numpy 1.19.5, pandas 1.1.5, scikit-learn 0.24.0 and tqdm 4.61.0. Those pins, not the model code, are the practical obstacle. Python 3.6 reached end of life in December 2021, and torch 1.5.0 predates several CUDA generations, so reproducing the documented runs on current hardware may require containers rather than a fresh virtualenv. The README does not describe a requirements.txt or conda environment file, so the version block is the specification.

What the Logged Training Runs Actually Show

The README embeds console output for three tasks, and those logs are the only performance evidence in the material. For Chinese single-sentence classification the log shows 4186 batches per epoch, an epoch time of about 1123 seconds on the first epoch and 1130 seconds on the tenth, training loss falling from 2.862 at batch 0 to 0.100 at the end of epoch 9, and validation accuracy of 0.884 and 0.888 across two evaluations. For MNLI the log shows 17181 batches per epoch, roughly 2610 and 2542 seconds per epoch, and validation accuracy of 0.827 and 0.830. For SWAG the log excerpt ends mid-timestamp at batch 4590 of 4597 in epoch 0, so no final validation number is visible in the supplied text. Read these numbers as one author's runs on unspecified hardware, not as reproducible benchmarks: the README does not state the GPU, the batch size, the learning rate, or the random seed. The epoch times are the more useful signal. At roughly 19 minutes per epoch for classification and 43 minutes for MNLI, ten epochs is a multi-hour job, and the README's own examples run to epoch 9. If you plan to modify the encoder and retrain, budget accordingly.

Where This Repository Will Not Serve You

Three limits are visible in the material. First, licensing. The supplied repository metadata lists the license as unknown, and no license file appears in the README or the described structure. Without a license, the default position is that the author retains all rights, which makes redistribution or incorporation into a commercial codebase legally unclear. That is a factual gap, not legal advice, and it is the single highest-risk item here. Second, there is no packaging. You cannot `pip install` this; the usage model is cloning the repository, downloading datasets into data/, and running scripts from Tasks/. Any integration means copying files. Third, the task scripts are monolithic by design. Because the README documents no configuration interface, adapting a task to your own dataset means editing data_helpers.py and the corresponding Task file. That is fine for study and awkward for reuse. There is also a scope limit worth naming: the question answering task targets SQuAD-V1.1, and the README does not mention SQuAD 2.0 or unanswerable questions, so the QA head should be assumed to always predict a span. If your inputs can legitimately have no answer, this implementation does not address that case.

How It Differs From the transformers Library

The obvious alternative is Hugging Face transformers, which is itself pinned here at 4.5.1, meaning the repository depends on the library it duplicates. The difference in approach is the point of the project rather than an accident. transformers exposes BERT through a configuration-driven factory: you name a checkpoint, the library resolves the architecture, downloads or loads weights, and hands you a model with a standard forward signature. The internals are abstracted behind that interface. BertWithPretrained inverts this. The encoder is written out in MyTransformer.py and BertEmbedding.py, each downstream head is a small readable class, and the training loop lives in the Task script where you can see the loss computation and the optimizer step. If your goal is to fine-tune a model on a deadline, transformers is the shorter path and gives you a maintained dependency. If your goal is to understand why the SQuAD head produces start and end logits over the sequence, or how the NSP and MLM losses are combined in BertForNSPAndMLM.py, the explicit version is the one that teaches you something. The trade is maintenance: transformers tracks upstream PyTorch and CUDA, while this repository is pinned to a 2021 stack.

Maintenance Posture and What to Check Before You Commit

The repository is not archived, and the last push recorded in the metadata is 2026-09-08, but no releases have been retrieved, so there is no versioned artifact to depend on. The dependency pins are the real maintenance cost: moving to a current PyTorch means auditing MyTransformer.py and the embedding code for API changes, and moving to a current transformers version means checking whether the 4.5.1 usage in the Task scripts still applies. None of that is described in the README, which offers no upgrade notes. The bundled checkpoints are the bert-base-chinese and bert-base-uncased weights, so the model sizes and vocabularies are fixed by those two directories. Before you spend time on this, confirm three concrete things: that a LICENSE file exists in the repository root, that your CUDA and driver combination still supports torch 1.5.0 or that you can containerize Python 3.6, and that the data subdirectories named in the README (SingleSentenceClassification, PairSentenceClassification, MultipeChoice, SQuAD, WikiText, SongCi, ChineseNER) contain the files the per-directory READMEs describe, since Step 1 of the usage instructions makes dataset acquisition your responsibility.

Editorial conclusion

Adopt BertWithPretrained if you want to read a self-contained PyTorch implementation of BERT and its downstream heads, or to run one of the seven listed tasks on the bundled data as a learning exercise. Do not adopt it as a dependency for a production pipeline: the pinned stack is Python 3.6 with torch 1.5.0 and transformers 4.5.1, there is no packaging, no test harness beyond the test directory, and no license file in the supplied material, so the terms of reuse are unstated. Before committing, verify three things: whether a LICENSE file exists in the repository, whether your GPU and CUDA version still support torch 1.5.0, and whether the per-task data directories actually contain the files the README expects, since the README instructs you to download each dataset yourself.

Official sources

1. Issues
2. mlwithme/BertWithPretrained on GitHub
3. README