juand-r/entity-recognition-datasets: a catalogue of NER corpora and the conversion code behind it

What juand-r/entity-recognition-datasets is for

Picking a named entity recognition corpus is mostly a licensing and format problem, not a modelling problem. A team wants news text with person and organisation tags, or clinical notes with medication and dosage tags, and discovers that half the well-known corpora sit behind data use agreements, that the rest arrive in four different annotation formats, and that the licence of one dataset is a stack of six different licences. This repository exists to make that first hour less painful. It is a table of English NER corpora with columns for domain, licence, reference and where to get the data, followed by links to corpora in other languages. The audience is an engineer or researcher starting an entity extraction project who needs to know what is legally and practically available before writing any training code. It is not a library, not a model, and not a benchmark. Nothing in the repository trains anything.

The table, the data directory and the CoNLL 2003 target format

The mechanism is deliberately boring. README.rst holds one large table for English datasets. Each row names the dataset, its domain (News, Twitter, Medical, Anatomical, Queries, Malware, Defense, Finance, Robotics, Wikipedia, Historical, Music, Astrophysics, Government), its licence, a citation, and either a checkmark meaning the corpus is included in the repository or an external link. Corpora that cannot be redistributed, such as CoNLL 2003 under its data use agreement, MUC-6 and OntoNotes 5 under LDC licences, or the i2b2 medical sets, are represented by entries in the data directory that explain where to obtain them. That directory also holds code to convert datasets to the CoNLL 2003 format when the original distribution uses something else. The choice of CoNLL 2003 as the single target is the quiet architectural decision here: it is the format most sequence taggers expect, so normalising everything to it means one loader instead of one loader per corpus. The trade-off is that CoNLL 2003 is a flat, token-per-line, BIO-style column format, and several of the listed corpora are not flat. The repository lists NNE, a dataset for nested named entity recognition, and nested annotations do not survive a lossless trip into a single tag column. The table also carries datasets with non-commercial licences, including WikiNEuRal, MultiNERD and WorldWide under CC BY-SA-NC 4.0, which a commercial project cannot use regardless of how convenient the conversion is.

Installing the requirements and running a conversion

There is no package to install. You clone the repository, and the only dependency declared at the top level is pinned in requirements.txt:

pip install -r requirements.txt

That file pins scikit-learn==0.18.2, an old release. On a current Python interpreter that pin will either fail to build or pull in a scikit-learn that conflicts with whatever else is in your environment, so the practical move is to install it in a throwaway virtual environment or to read the conversion scripts and port the few functions you need. The repository layout is the real entry point: data/ holds the per-dataset directories, src/ holds code, and the README tells you that the data directory contains both the pointers for restricted corpora and the conversion code. For a dataset that is included directly, such as wikigold or WNUT17, you copy the files out of its subdirectory. For one that is not, you follow the link in the table, obtain the corpus under its own terms, then run the conversion script for that dataset and check that the output is token-per-line CoNLL 2003 before feeding it to a tagger. If a dataset has no script, the README does not document a fallback, and you are writing the converter yourself.

Where the catalogue stops being useful

The README opens with a note that the author is no longer actively adding datasets and that there are likely more NER datasets that have appeared since 2020. Treat that as the repository's real boundary rather than a disclaimer. A catalogue of training data ages badly: corpora, licences and hosting locations all move, and a link that worked in 2019 may now redirect or 404. The dated entries in the table show the shape of the problem, with most references clustered between 2003 and 2017 and a handful from 2021 to 2025 appended later. The second limitation is that the table is a summary, and summaries lose the details that decide adoption. re3d, for instance, is annotated as containing several datasets under different licences, including CC-BY-SA 3.0, CC BY-NC 3.0, CC BY 3.0 AU, public domain, and the UK Open Government Licence v3.0, with one entry pointing to a separate legal notice. GUM 3.1.0 is listed as comprising three datasets under three different Creative Commons licences, with the annotations under CC-BY 4.0. You cannot decide whether you may use those from the table alone. The README itself directs you to the corresponding subdirectory for more detailed licence information, and that is the step people skip.

Alternatives and how they differ in approach

The closest alternative is Hugging Face Datasets. The difference is architectural: Hugging Face hosts the data and exposes it through a loading API, so a corpus is one function call away and versioning is handled by the hub, while this repository is a set of links plus local conversion scripts. Hugging Face is the better choice when the corpus you want is already there and you care about reproducible loading in a training script. This repository is the better choice when the corpus is not on the hub, when it is gated behind a data use agreement that requires you to request it yourself, or when you specifically need the licence picture across many corpora at once. A second alternative is a domain-specific collection, for example the clinical NER datasets that the table lists under i2b2 and CADEC. Those are not interchangeable with a general catalogue: they come with access procedures and annotation guidelines of their own, and the value this repository adds is telling you they exist and who to ask. A third is simply taking a modern multilingual corpus such as WikiNEuRal or MultiNERD directly from its own repository, which is what the table's links point to anyway; going through this catalogue adds the licence summary and nothing else.

Maintenance, licence and upgrade cost

The last push to the default branch was on 2026-07-02, so the repository is not abandoned, but the README's own note about no longer actively adding datasets means the table should not be read as current. There are no releases, so there is no version to pin and no changelog to consult; you consume the master branch as it stands. The repository's own licence is MIT, which covers the code and the README, not the corpora. That distinction is the one that matters legally: each dataset keeps its own terms, and the table lists DUA, LDC, CC-BY, CC-BY-SA, CC-BY-NC, CC-BY-SA-NC, CSIRO, UK Open Government Licence and public domain entries side by side. MIT on the repository gives you no rights over any of the data. Upgrade cost is low in the sense that nothing is installed and nothing breaks, and high in the sense that the conversion scripts were written against scikit-learn 0.18.2 and an older Python, so reusing them on a current stack means either isolating the environment or rewriting the few functions involved. This is not legal advice; read the licence file in the subdirectory of the dataset you intend to use.

Editorial conclusion

Adopt this repository if you need to choose an NER corpus and want licence and availability information in one table, or if you need the conversion scripts for datasets that ship in their own format. Do not adopt it expecting a maintained index: the README states that the author is no longer actively adding datasets and that entries are likely missing since 2020, so anything released after that year has to be found elsewhere. Before relying on it, open the data subdirectory for the specific corpus you plan to use and confirm its licence terms and the CoNLL 2003 conversion path, because the top-level table only summarises both.

Frequently asked questions

Official sources

1. Issues
2. juand-r/entity-recognition-datasets on GitHub
3. License: MIT
4. README