natasha/corus: Loaders for Russian Text Corpora in Python

What corus actually does for a Russian NLP pipeline

The package solves a narrow, repetitive problem: every Russian corpus dump has its own format, and every project that uses two or three of them ends up writing the same throwaway parsing code. corus collects the download links in one Reference table and pairs each dataset with a function such as load_lenta, load_librusec, load_mokoron, load_wiki, load_gramru, load_corpora and load_simlex, importable as from corus import load_lenta.

The README frames the scope directly: links to publicly available Russian corpora plus code for loading and parsing, with more than 20 datasets and over 350Gb of text listed. The audience is anyone building Russian-language NLP who needs raw text rather than a model. The repository is mostly a Jupyter Notebook (docs.ipynb) plus a Python package, which tells you the intended workflow is exploratory: open a notebook, pull a dump, iterate over records.

What corus does not do is equally clear. It does not host the corpora, does not download them, and does not ship a CLI. The install_requires list in setup.py is empty, so the package has no runtime dependencies of its own; parsing is done with the standard library.

The loader contract and the data flow it assumes

The mechanism is a generator per dataset. You give the loader a path to a file you downloaded yourself, and it yields records. The README example uses the Lenta.ru dump, and the returned object is a named record with url, title, text, topic and tags fields. Iteration is lazy, so a 1.66 Gb compressed file does not have to be materialised in memory as a list of strings.

That design has a consequence worth naming: the loader is coupled to a specific dump version. The Reference table lists load_lenta for Lenta.ru v1.0 and load_lenta2 for v1.1+, with different row counts and different archive formats (csv.gz versus csv.bz2). If you download the wrong version for the function you called, you get a parse error or silently wrong fields, not a helpful message.

Some datasets expose two functions rather than one. Rossiya Segodnya has both load_ria_raw and load_ria, which suggests a choice between the raw structure and a normalised view. The Reference table does not explain the difference in the README text, so you have to read docs.ipynb or the source to know which one fits.

Installing corus and loading your first corpus

corus is on PyPI and supports Python 3.5+ and PyPy 3 according to the README. Installation is a single command:

pip install corus

There is nothing else to configure at install time, since setup.py declares no install dependencies. The first real task is downloading a dump, because corus will not do it for you. The README gives this exact command for Lenta.ru v1.0:

wget https://github.com/yutkin/Lenta.Ru-News-Dataset/releases/download/v1.0/lenta-ru-news.csv.gz

Once the file is on disk, load it and pull the first record. The README shows this interaction, and the object you should see is a LentaRecord with the fields url, title, text, topic and tags:

from corus import load_lenta

path = 'lenta-ru-news.csv.gz'
records = load_lenta(path)
next(records)

For a full pass over the corpus, iterate instead of calling next. The README's loop is the shape most training pipelines start from:

records = load_lenta(path)
for record in records:
    text = record.text

If you need a different dataset, the Reference table gives the wget line and the importable function name side by side, so the same three steps apply.

Where corus stops being the right tool

The largest limitation is that corus is a parser, not a data source. Nothing in the README automates retrieval, and several datasets require manual steps that a script cannot paper over. The Mokoron Twitter corpus points to a Dropbox link for db.sql and says to download it manually. GramEval2020 needs a wget, an unzip, three mv commands to rename dataTrain and dataOpenTest into train and dev, a cleanup step, and a second wget for the private test file. Reproducing that in CI is possible but is your code, not corus's.

Second, the documentation is in Russian. The README links to the Corus page on natasha.github.io and a Datafest 2020 talk section, and states plainly that materials are in Russian. If your team cannot read Russian, the README and docs.ipynb are the only English-language surface, and they are terse.

Third, there are no releases retrieved for this project, and versioning lives in setup.py, currently 0.10.0. There is no changelog in the repository listing, so an upgrade means diffing the source or reading commit history. The last push to the default branch was on 2026-04-21, so the project has moved recently, but the absence of tagged releases makes pinning a deliberate act: pin corus==0.10.0 in your requirements rather than tracking master.

Finally, the package is Russian-specific by construction. Every loader in the Reference table targets a Russian-language dump. If your corpora are English or multilingual, corus has nothing to offer and you would be better served by a general dataset library.

How corus differs from Hugging Face datasets

The closest comparison is Hugging Face datasets, and the difference is architectural rather than cosmetic. datasets owns the download step: a dataset identifier resolves to a hosted script and the data is fetched, cached and versioned by the library. corus does the opposite. It hands you a wget command and a parser, and the file lives wherever you put it.

That split matters for two reasons. On the plus side, corus adds no dependency tree and no network behaviour to your environment; install_requires is empty, so installing corus cannot break an existing stack. On the minus side, you own storage, checksums, re-downloads and disk cleanup for files that run to tens of gigabytes, such as the 144.92 Gb Lib.rus.ec dump or the 12.94 Gb Russian Wikipedia dump.

If your work is Russian text and your infrastructure already has a data lake or a mounted volume, corus is the smaller tool. If you want one API that fetches and caches datasets across languages, datasets covers more ground and takes on more responsibility. The two are not mutually exclusive: you can download a dump by hand and still load it with corus.

Licence, maintenance and what an upgrade costs

corus is MIT licensed, both in the LICENSE file and in the setup.py metadata, which permits commercial use and modification provided the copyright notice and permission notice are retained. That covers the code only. Each corpus in the Reference table carries its own terms: Lenta.ru comes from a third-party GitHub repository, Lib.rus.ec is a dump prepared for the RUSSE workshop, Wikipedia is a Wikimedia dump, and the Mokoron corpus is distributed through Dropbox. The README does not state a licence per dataset, so checking the upstream source before redistribution or commercial training is your responsibility, not something the package resolves for you.

Maintenance cost is low but not zero. The package itself changes rarely and has no dependencies to chase. The real recurring cost is upstream: dump URLs move, archive formats change between versions (csv.gz to csv.bz2 for Lenta.ru), and a loader written against v1.0 will not read v1.1+. Budget for a periodic check that the wget lines in the Reference table still return files and that the loaders still parse them. The Makefile shows the project's own verification path, with lint running flake8 over corus and exec-docs re-executing docs.ipynb via nbconvert.

Editorial conclusion

Adopt corus if your pipeline already works with Russian text and you want load_lenta, load_wiki or load_corpora to turn downloaded dumps into records without writing a parser per dataset. Do not adopt it if you need the package to fetch data for you, if your corpora are not Russian, or if you need documentation in English beyond the README. Before committing, check that the loader you need exists in the Reference table, confirm the dump URL in that table still resolves, and run the loader on a truncated file to see what a record looks like.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. natasha/corus on GitHub
4. README