MNBVC: A 60 TB Chinese Corpus That Refuses to Be Indexed

What MNBVC actually is, and who it is for

MNBVC stands for Massive Never-ending BT Vast Chinese corpus. The README describes it as a Chinese corpus spanning news, essays, novels, books, magazines, papers, subtitles, forum posts, wikis, classical poems, lyrics, product descriptions, jokes, embarrassing stories and chat logs, all collected from the internet. The stated target is 253 TB, and the README reports the current total at 60,944 GB, or 24.1 percent of that goal. The last push to the repository was on 2026-09-13.

The intended user is not an application developer looking for a clean dataset to fine-tune on. It is a pretraining-scale consumer: someone assembling tens of terabytes of raw Chinese text who is willing to do the cleaning themselves. The README is explicit that archives receive only coarse processing, such as HTML and XML to txt, and CSV and TSV to JSON. Everything past that stage is left to the consumer, or to the separate cleaning tools the project links to.

There is also an explicit request in the README that media not report on the project, framed as a wish to avoid attention that could end the collection effort. That posture shapes the whole distribution model, and it is the first thing to understand before evaluating anything else.

How the corpus is packaged and delivered

Distribution is the unusual part. The README lists two channels: a P2P sync tool called Weili (verysync), and Baidu Netdisk. The P2P route gives two keys, one per partition. The README warns that part 1 alone needs more than 10 TB of disk space, and points to a wiki page called the new plan for smaller chunks. It also advises disabling TCP hole punching and UDP transfer in the client settings, noting that leaving them on may congest the router even if transfer is faster.

The archives are password protected, and the README states the password is 253874. Inside, the README says, each subfolder carries a links.txt listing the source URLs for that folder, plus a PNG screenshot of the source web page. The README also states that numeric strings of eight digits or longer are stripped as a de-identification step.

The formats inside are txt, json, jsonl and parquet, with the README saying the collection will eventually converge on jsonl and parquet. That last point matters: if you build a pipeline against the current mixture, expect to revisit it as the archives are updated, because the README says each archive is refreshed as cleaning progresses.

Why there is no index, and what that costs you

The README states plainly that the project does not provide an index or classification of the contents of the archives. The stated reason is copyright: the team says it cannot audit the provenance of every source, and withholding an index is a way to keep the downloads available. The README asks users not to discuss what specific archives contain.

This is a real design decision with a real cost. You cannot query the corpus for a topic, a date range or a source before downloading. You cannot estimate how much of the 60,944 GB is forum posts versus exam papers. You cannot sample a slice to check text quality without pulling whole archives. For a research team used to Hugging Face datasets where you can inspect a few rows in the browser, this inverts the workflow: you commit storage first and discover contents second.

The README does point to ModelScope and Hugging Face under the dataset name liwu/MNBVC for cleaned and classified data, which is the closest thing to a browsable layer. But the README describes that as data that will be placed there progressively, not as a complete mirror of the archives.

Getting the corpus: P2P keys and the password

There is no install step for MNBVC itself, because MNBVC is data plus a set of companion tools, not a runnable package. The README gives no pip install, no CLI and no library entry point for the corpus. What it gives is a sync key and a password.

The practical first step is installing the Weili client from the vendor site the README links, then pasting the part 1 key. The README provides the key as a string and a direct link. The key for part 1 is reproduced here exactly as the README gives it:

B4MVPVJTK3DOOAOPVLJ3E7TA7RWW4J2ZEAXJRMRSRHSBPDB7OAFHUQ

Part 2 has its own key, and the README presents the two as separate partitions, so you add them as separate sync tasks rather than one:

B4FQSD525XQQDY6XNO7JZ6BM2EIKAUTVPLLVX6N52HIWBZ7G72R7EQ

Once an archive finishes downloading, the README states the password is 253874. Unpacking is a normal archive extraction with that password; the README does not name the archive tool, so use whatever handles the container format you receive. After extraction, the README says you should find links.txt and a PNG screenshot at the root of each subfolder, which is the provenance record. If those are missing, the archive did not come through intact.

The alternative channel is Baidu Netdisk, and the README splits it into two documents, dupan/README.md and dupan/README2.md, which are the only two entries under dupan/ in the repository tree alongside LICENSE, README.md, .gitignore and original_intention.jpeg.

The cleaning tools are separate repositories, not part of MNBVC

The README links to a long list of companion projects maintained under the -mnbvc suffix, and this is where the actual engineering lives. For encoding detection there is charset_mnbvc. For batch txt-to-jsonl conversion with duplicate-paragraph filtering there is deduplication_mnbvc. For sampling files by keyword while preserving directory structure there is scan_copy_files_mnbvc. For format validation there is DataCheck_MNBVC. For PDF work there are pdf_meta_data_mnbvc, mmdp_mnbvc, pdf2txt_mnbvc and docling_parse_mnbvc, plus mm_template_mnbvc for converting text files to parquet.

There is also a set of source-specific cleaners: tianya-mnbvc for the Tianya forum, reddit-mnbvc, WikiHowQAExtractor-mnbvc, Math_mnbvc, stackexchange_mnbvc, parallel_corpus_mnbvc, MNBVC-judgment for court documents, and several others. Crawler code for GitHub, Notabug and Bitbucket repositories is published too.

The important caveat is that these are independent repositories with their own maintenance and their own READMEs. The MNBVC README does not describe their interfaces, their dependencies or their compatibility with each other. A team adopting MNBVC is adopting a collection of tools, not a pipeline, and the integration work is on the consumer.

Where MNBVC is the wrong choice

If you need a legally clean corpus, MNBVC is the wrong choice. The README states the team has no ability to audit the copyright of its sources, and the no-index policy is a direct consequence of that. There is no per-document licence metadata, only a links.txt of source URLs and a screenshot. For a commercial product with a compliance review, that is not an auditable chain.

If you need reproducibility, MNBVC is also awkward. The archives are updated as cleaning progresses, the README says so directly, and there are no releases in the repository. You cannot pin a version of the corpus the way you would pin a dataset revision. A training run described in a paper today may not be reproducible from the archives six months from now.

If your Chinese data need is modest, the storage math does not work. Part 1 alone is described as needing more than 10 TB. For a team that wants a few hundred gigabytes of Chinese text, the ModelScope or Hugging Face mirrors under liwu/MNBVC are the sensible entry point, and the archives are not.

Finally, if you need a maintained API, look elsewhere. The repository is not archived and its last push was on 2026-09-13, but what is pushed is documentation, keys and links, not a versioned library.

How MNBVC differs from Common Crawl and Hugging Face datasets

The closest comparison is Common Crawl. Both are web-scale text collections with no copyright review and no topic index. The difference is in the unit of delivery. Common Crawl publishes monthly WARC files over HTTP with a documented file naming scheme and a public index server that lets you query by URL, so you can fetch a targeted subset. MNBVC publishes password-protected archives over a P2P sync client, and the README states it deliberately provides no index. Common Crawl is also not Chinese-specific; MNBVC's stated scope is Chinese text including minority subcultures and leetspeak-style writing, which is a narrower and more deliberate filter.

The other comparison is a curated Hugging Face dataset. Those typically give you a dataset card, a licence field, a viewer and a versioned revision. MNBVC's ModelScope and Hugging Face mirrors move in that direction, but the README presents them as an ongoing deposit of cleaned, classified data rather than the primary artifact. The primary artifact remains the archive set, and its contract is: here is a password, here is a key, do not ask what is inside.

That contract is coherent given the project's stated goal of surviving long enough to reach 253 TB. It is also the single biggest reason an engineering team would choose a smaller, indexed dataset instead.

Editorial conclusion

MNBVC is for teams that already have petabyte-class storage and can accept a corpus with no index, no copyright review and no per-file provenance beyond a links.txt and a screenshot. It is not for anyone who needs to know what is inside before downloading, or who needs a licence audit trail. Before you commit disk space, verify the current archive password (the README states 253874), confirm which of the two P2P keys matches the partition size you can hold, and check that the per-subfolder links.txt and PNG screenshot actually exist in the archives you pull.

Frequently asked questions

Official sources

1. esbatmop/MNBVC on GitHub
2. Issues
3. License: MIT
4. README