PhoBERT: Vietnamese BERT that expects pre-segmented input

The problem PhoBERT addresses: Vietnamese needs a monolingual encoder, not a multilingual one

Vietnamese is a low-resource language for pre-trained encoders, and the practical choice before PhoBERT was between multilingual models such as mBERT or XLM-R, which spread capacity across many languages, and small monolingual models trained on limited data. The README positions PhoBERT as the first public large-scale monolingual language model pre-trained for Vietnamese, in base and large versions, and reports that it obtains state-of-the-art results on four downstream tasks: part-of-speech tagging, dependency parsing, named-entity recognition and natural language inference. The intended audience is a team that has Vietnamese text and wants a starting checkpoint for a tagging, parsing, NER or NLI model rather than training an encoder from scratch. That is a narrow but well-defined audience: you need enough Vietnamese labelled data to fine-tune, and you need to accept the pre-processing contract described below.

RoBERTa pre-training, RDRSegmenter pre-processing: the two halves of the design

The architecture follows RoBERTa, which the README describes as an optimisation of the BERT pre-training procedure. The base checkpoint has 135M parameters, the large one 370M, and both cap at a maximum length of 256 tokens. The base models were pre-trained on 20GB of Wikipedia and news text; phobert-base-v2 adds 120GB of text from OSCAR-2301. The second half of the design is less visible from the model card and more important in practice. PhoBERT does not operate on raw Vietnamese. Pre-training data was pre-processed with the RDRSegmenter from VnCoreNLP, including Vietnamese tone normalization and word and sentence segmentation. The README states plainly that the input text must already be word-segmented, and its example sentence is written as 'Chúng_tôi là những nghiên_cứu_viên .' with underscores joining the components of multi-syllable words. The tokenizer was therefore fitted to a segmented word vocabulary, not to raw syllables. If you skip segmentation, the tokenizer still returns IDs, but they correspond to a different distribution than the one the encoder was trained on.

Getting it running with transformers and py_vncorenlp

The transformers path is short. Install the library with pip install transformers and the tokenizer package with pip3 install tokenizers, then load the model and tokenizer with AutoModel.from_pretrained and AutoTokenizer.from_pretrained against vinai/phobert-base. The README's example wraps the encoded sentence in torch.tensor, runs it under torch.no_grad(), and notes that model outputs are tuples. A TensorFlow 2.0+ variant is shown as a comment using TFAutoModel. The segmentation step comes from a separate package: pip install py_vncorenlp, then py_vncorenlp.download_model(save_dir='/absolute/path/to/vncorenlp') to fetch the VnCoreNLP components, followed by py_vncorenlp.VnCoreNLP(annotators=["wseg"], save_dir=...) to load the word segmenter. Calling rdrsegmenter.word_segment(text) returns a list of segmented sentences, as in the README output where 'Ông Nguyễn Khắc Chúc' becomes 'Ông Nguyễn_Khắc_Chúc'. The README recommends using this same segmenter for downstream applications on raw text, precisely because it matches the pre-training pre-processing. A fairseq path exists too, delegated to a separate README_fairseq.md in the repository, which this review has not read.

The fast tokenizer is still in discussion, and that shapes your serving stack

The README states that a slow tokenizer for PhoBERT was merged into the main transformers branch, while a fast tokenizer is still under discussion, referencing a pull request comment. Until that lands, users who want the fast tokenizer are told to install a fork: clone the datquocnguyen/transformers repository on the branch fast_tokenizers_BARTpho_PhoBERT_BERTweet and install it with pip3 install -e . Running a fork of transformers in production is a real maintenance decision, because you inherit the fork's divergence from upstream and must track two release streams. The alternative is the slow tokenizer, which is simpler but slower at scale. Either way, the segmentation step remains outside the tokenizer, so a serving pipeline needs at least two stages: VnCoreNLP word segmentation, then tokenization and the encoder. That is more moving parts than a model that accepts raw strings, and it is the clearest structural cost of adopting PhoBERT today.

Where PhoBERT is the wrong tool: long documents and raw-text pipelines

The 256-token maximum length is the hard boundary. Vietnamese news articles and legal or administrative documents routinely exceed it, and the README offers no sliding-window or long-context recipe, so handling longer inputs is left to the integrator. The second failure mode is the segmentation dependency itself. Because segmentation is a separate model with its own download step, it introduces a network fetch at setup time and a versioned asset you must pin; the README's example hardcodes an absolute save_dir, which is a hint that this is expected to be a managed local artefact rather than something resolved automatically. If your environment cannot run the segmenter, or you need to score raw user text with a single model call, PhoBERT is the wrong choice. The README itself points readers toward BamiBERT for cases where these limits matter, describing it as operating directly on raw input with an extended context length of up to 2,048 tokens.

Licence matters here more than usual: MIT on two checkpoints, AGPL v3 on the newest

The repository is MIT, but the model table shows that the licence differs per checkpoint. vinai/phobert-base and vinai/phobert-large are MIT. vinai/phobert-base-v2, the checkpoint trained on the larger 20GB plus 120GB corpus, is GNU Affero GPL v3, with a separate LICENSE_for_PhoBERT_v2 file in the repository. That is not a detail to discover late. AGPL v3 carries network-service obligations that MIT does not, so a team that swaps vinai/phobert-base for vinai/phobert-base-v2 to gain the extra pre-training data changes its licence position at the same time. The two checkpoints have the same parameter count and the same 256-token limit, so the difference is the corpus and the licence, nothing else. Anyone planning to ship PhoBERT inside a product should confirm which checkpoint the code actually loads. This is a description of what the repository states, not legal advice; get your own counsel on AGPL obligations.

Maintenance cost and the realistic alternative

There are no retrieved releases for this repository, so versioning runs through the master branch and the Hugging Face model IDs. The upgrade surface is therefore threefold: the transformers version (or the fork, if you need the fast tokenizer), py_vncorenlp and its downloaded VnCoreNLP components, and the checkpoint itself. Each can move independently. The stated alternative is BamiBERT, referenced in the README as a BERT-based Vietnamese model trained from scratch on a 129GB corpus for 20 epochs, with up to 2,048 tokens and no external word segmentation. The difference in approach is not a matter of tuning: BamiBERT removes the segmentation stage and raises the context ceiling, which is exactly the two constraints that define PhoBERT's integration cost. The README reports that across eight Vietnamese benchmarks BamiBERT achieves the best performance on 11 of 15 metrics and second-best on three others among base-sized Vietnamese encoders. Those numbers come from the project's own README, not from independent measurement here. If your workload is raw text and long documents, the comparison is worth running on your own data before you build the segmentation pipeline.

Editorial conclusion

Adopt PhoBERT if your Vietnamese pipeline can run VnCoreNLP word segmentation before tokenization and you are fine with a 256-token ceiling; the MIT-licensed vinai/phobert-base and vinai/phobert-large checkpoints are the safer defaults for commercial work. Do not adopt it if you need a fast tokenizer, raw-text inference, or sequences beyond 256 tokens: the README states the fast tokenizer is still in discussion, and the current alternative it points to is BamiBERT, which operates on raw input with up to 2,048 tokens. Before committing, verify which checkpoint you are pulling, because vinai/phobert-base-v2 is AGPL v3 while the older checkpoints are MIT, and confirm that your segmentation step matches the RDRSegmenter used during pre-training.

Official sources

1. Issues
2. License: MIT
3. README
4. VinAIResearch/PhoBERT on GitHub