wiktextract: turning Wiktionary dumps into structured dictionaries

Template expansion is the whole reason this exists

Wiktextract is a utility and Python package for extracting data from Wiktionary, and the sentence that explains its approach is in the overview.

One thing that distinguishes this tool from any system the project is aware of is that it expands templates and Lua macros in Wiktionary. That enables much more accurate rendering and extraction of glosses, word senses, inflected forms, and pronunciations, and it also makes the system much easier to maintain. All this results in much higher extraction quality and accuracy.

This is the key insight for anyone evaluating it. A Wiktionary page is not a dictionary entry, it is a wiki page that constructs a dictionary entry using templates, and in the English edition a large share of the actual content lives in Lua modules rather than in the page text. A parser that reads the wikitext directly sees template invocations where the definition should be. One that expands them sees the definition.

The second benefit is maintenance, and it is the more surprising one. If a template is expanded rather than interpreted, then a template that changes affects the extraction uniformly, and the parser does not need a rule per template. That is why the tool is described as the most comprehensive available for extracting information from Wiktionary as of December 2020.

The cost is stated plainly in the readme as well. Extracting all data for all languages from the English Wiktionary may take from an hour to several days, depending on your computer. Expanding Lua modules is not cheap, but it enables superior extraction quality and maintainability, and the advice given is that you may want to look at the data downloads instead of running it yourself.

The dependencies make the mechanism visible. The package depends on `wikitextprocessor`, installed directly from the author's own git repository, which is the component that does the template and Lua expansion, plus `nltk` for natural language processing, `pydantic` for validation, and `levenshtein` for string distance.

One English edition is complete, the others cover the basics

The tool extracts information from various Wiktionary data dumps, most notably and completely the English edition, and the readme makes an important point about what an edition contains.

An edition of Wiktionary contains extensive dictionaries and inflectional information for many languages, not just the language it has been written in.

That is the observation that makes the English edition so valuable to process. The English Wiktionary describes words in thousands of languages, with declension and conjugation tables for most of them, and a smaller edition like the Finnish one does not. So one extraction from enwiktionary yields lexicons for many languages, which is why the tool can claim to extract information for all languages that have data in the edition.

The English edition module extracts a long list, and reading it shows the scope. Glosses, parts-of-speech, declension and conjugation information when available, translations for all languages when available, pronunciations including audio file links, qualifiers including usage notes, word forms, links between words including hypernyms, hyponyms, holonyms, meronyms, related words, derived terms, compounds and alternative forms. It also extracts links to Wikipedia pages, Wikidata identifiers, and other such data when available.

What makes it more than a dictionary is the per-sense annotation. For many classes of words, a word sense is annotated with specific information such as what word it is a form of, what is the RGB value of the color it represents, what is the numeric value of a number, what SI unit it represents. So a number entry carries its numeric value and a colour entry carries its RGB, as structured fields rather than as prose.

Other editions are less complete, and the readme is honest that this is partly because the edition itself does not necessarily have the same width of data, but the project tries to cover the basics. It also extracts translingual data and information about characters, meaning anything that has an entry in Wiktionary.

The package is version 1.99.7 with the development status marked as 3 - Alpha, and the project metadata records the licence as MIT, with the version numbering approaching 2.0.

JSON Lines is a memory decision, not a formatting preference

The output format is JSONL, JSON objects separated with newlines, and the readme explains the choice in a way that tells you what the tool is built for.

Some people have asked for the full data as a single JSON object, instead of the current one JSON object per line format. The decision was made to keep it as a JSON object per line, because loading all the data into Python requires about 120 GB of memory. It is much easier to process the data line-by-line, especially if you are only interested in a part of the information.

That is a real constraint and it is the right one. The English Wiktionary is the largest wikitext corpus in common use, and a JSON document holding all of it parsed into Python objects would not fit on a normal machine. Streaming one object at a time is what makes the output usable on a laptop, and it is also what makes it usable inside a pipeline where you want to filter as you go.

The readme shows the three snippets you need, and they are worth copying verbatim. Reading it line by line:

import json

with open("filename.json", encoding="utf-8") as f:
    for line in f:
        data = json.loads(line)
        ... # parse the data in this record

Collecting it into a list, which is the thing you should not do with the full dataset:

import json

lst = []
with open("filename.json", encoding="utf-8") as f:
    for line in f:
        data = json.loads(line)
        lst.append(data)

And pretty-printing a single record for a human:

print(json.dumps(data, indent=2, sort_keys=True, ensure_ascii=False))

The `ensure_ascii=False` is a detail that matters for a multilingual dataset, since it keeps the scripts and the non-Latin characters readable rather than escaping them into sequences.

Most people should download the data instead of producing it

The pre-extracted data section is, realistically, the part most users will use, and the readme is direct about that.

For most people, it may be easiest to just download pre-expanded data, at the kaikki.org raw data page. The raw wiktextract data, the extracted category tree, the extracted templates and modules, and a bulk download of audio files for pronunciations in both ogg and mp3 formats are all available.

The download granularity is unusually fine, and that is what makes it useful for research. There is a download link at the bottom of every page and a button to view the JSON produced for each page. You can download all data, data for a specific language, data for just a single word, or data for a list of related words, such as a particular part-of-speech or words relating to a particular topic or having a particular inflectional form. All downloads are in JSON Lines format, and the bigger downloads are also available in compressed form.

That last capability, downloading by inflectional form or by topic, is the one that turns the dataset into a research instrument rather than a word list. Asking for all words that are the plural of something, or all words in a semantic category, is a question you can now ask of a corpus of a hundred million words.

The site itself is described as being updated every few days, and some extracted Wiktionary editions data are available for browsing and downloading there.

For the schema, the arrangement is a little uneven. Non-en editions have JSON schema files at the project's GitHub Pages site. The English edition does not have a JSON schema, and the fields are instead listed in a mapping index on kaikki.org, and the project has `TypedDict` models in `type_utils.py` in the extractor for the English edition, which gives the same information to a type checker rather than to a JSON schema validator.

The command line, the schema, and where the tool does not reach

The `wiktwords` script makes extracting the information for use by other tools trivial without writing a single line of code, extracting the information specified by command options for languages specified on the command line and writing the extracted data to a file or standard output in JSONL format. It is the only entry point declared in the package manifest, as `wiktwords = "wiktextract.wiktwords:main"`.

The top-level directory listing shows the shape of the project. There is a `languages/` directory, which is where the per-edition extractors live, and a `src/` directory with the `wiktextract` package inside it, with a `data/*/*` package data pattern and a `py.typed` marker, so the package ships its own type information. There is a `tests/` directory, `tools/` with the `github_pages.py` script the Makefile uses to publish the schema pages, `usertools/`, `docs/`, a `Containerfile` for a containerised build, a `push-pypi` script for publishing, and an `inflmapduplicatekeys.sh` script whose name suggests it checks the inflection map for duplicate keys.

The `TODO` file at the root is a rough edge that says the project is candid about what is unfinished, and the `Development Status :: 3 - Alpha` classifier agrees.

The build and test setup is straightforward. The Makefile has a `test` target running `python -m unittest discover -b -s tests`, a `test_coverage` and `coverage_report` pair using coverage with branch coverage enabled and multiprocessing concurrency, an `rcheck` target running ruff over `src/wiktextract` and `tests`, and a `github_pages` target. The pyproject configures ruff and black with a line length of 80, and the coverage configuration omits `taxondata.py` with the comment that it is a huge file, which is a useful signal about the shape of the data: a table of taxonomic names is a large static dataset that does not need coverage measurement.

Where the tool does not reach is worth naming. It reads a dump file, and it does not talk to the live wiki. It parses, it expands, it extracts, and it emits JSONL. Anything that needs the wiki's edit history, its talk pages or its category graph as it exists right now is not what this does.

Editorial conclusion

wiktextract fits a researcher who needs a structured multilingual lexicon from Wiktionary and either has the patience to run a multi-hour extraction or is willing to use the pre-extracted data at kaikki.org, which is usually the faster route. It does not fit a project that needs the data tomorrow, since a full English edition extraction is documented as taking from an hour to several days. Before you run it, read the note about memory, because loading the whole English dataset at once needs roughly 120 GB, which is why the output is JSON Lines rather than one JSON object.

Frequently asked questions

Official sources

1. Issues
2. README
3. tatuylonen/wiktextract on GitHub