WikiExtractor: Pulling Plain Text Out of Wikipedia Dumps Without a Database
A tool for extracting plain text from Wikipedia dumps. Warning**: problems have been reported on Windows due to poor support for StringIO in the Python implementation on Windows.
At a glance
- What is it?
- WikiExtractor is a Python 3 script that turns raw Wikipedia XML dumps into clean, structured text files. It handles template expansion, supports parallel processing, and has a companion tool for Cirrus dumps, but Windows users should check the StringIO warning before relying on it.
- Who is it for?
- Adopt WikiExtractor if you need a dependency-free, scriptable way to turn Wikipedia XML dumps into plain text for NLP, search indexing, or corpus building, and you work on Linux or macOS. Skip it if you are on Windows, because the documented StringIO problems can break extraction, or if you need to preserve MediaWiki markup, since the tool strips it by design.
- Can I use it commercially?
- Yes, with strict conditions. AGPL-3.0 is a network copyleft licence: if people use a modified version over a network, for example as a hosted service, you must offer them its source code under the same licence.
- Is it still maintained?
- Yes. The repository last received commits 10 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 14, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What WikiExtractor Solves and Who Needs It
Wikipedia dumps are huge XML files, often several gigabytes, filled with MediaWiki markup, templates, and metadata. Anyone building a text corpus for natural language processing, search, or offline reading needs plain text, not raw wiki syntax. WikiExtractor is a Python script that reads a dump and outputs clean text in a structured format. It is aimed at researchers, developers, and data engineers who want a simple, scriptable pipeline without setting up a database or using a heavy framework. The tool requires Python 3 and no additional libraries, which makes it easy to drop into existing workflows. It is not a full text-processing suite; it is a focused extractor that does one job: turn wiki markup into readable text.
How It Handles Templates and Why That Matters
MediaWiki templates are a major obstacle to plain-text extraction. A typical article transcludes infoboxes, citations, and navigation boxes, all of which are defined elsewhere in the dump. WikiExtractor solves this by preprocessing the entire dump to extract template definitions, then expanding those templates during extraction. The README explains that this is done by 'preprocessing the whole dump and extracting template definitions.' This is a two-pass approach: first pass collects template bodies, second pass expands them in each article. The cost is time, but the benefit is that the output text is closer to what a reader sees, without raw template syntax. For repeated extractions, the tool can cache parsed templates to a file with the --templates option, which avoids reprocessing the dump. This is a practical design for corpus building, where you might extract the same dump multiple times with different settings.
Getting It Running: Commands and Options
You can invoke the script directly with 'python -m wikiextractor.WikiExtractor <dump file>' or install it via pip with 'pip install wikiextractor'. The installer also adds two commands: 'wikiextractor' and 'extractPage'. The main command accepts a dump file and several options. The output directory is set with '-o OUTPUT', and the maximum size per output file with '-b n[KMG]', defaulting to 1M. The '--json' flag changes the output format to JSON, which is useful for programmatic consumption. The '--no-templates' option skips template expansion entirely, which the README says 'significantly speeds up the extractor.' For parallel processing, the '--processes' option controls how many processes handle articles concurrently. The 'extractPage' command takes a dump and an article ID with '--id' or a template number with '--template', giving you a quick way to pull a single page without processing the whole dump.
The Output Format and How to Use It
The default output is split into multiple files of similar size, each containing documents wrapped in '<doc id="" url="" title="">' tags. The README shows this format and notes that the full specification is in the project wiki. With '--json', each file contains JSON documents instead, which is easier to parse with standard tools. The '--compress' option, available in the Cirrus extractor, compresses output files with bzip2, saving disk space. The '--namespaces' option lets you filter by namespace, so you can extract only article pages and skip talk pages or user pages. This output structure is designed for downstream processing: you can feed the files directly into a text analysis pipeline, or concatenate them for a single corpus. The split-by-size approach is practical for handling large dumps, but it means you need to manage multiple files, which some users might find awkward.
The Cirrus Extractor: A Different Input Format
The repository includes a second script, 'cirrus-extractor.py', which works on Cirrus dumps rather than the standard XML dumps. Cirrus dumps are JSON-based and contain text with already expanded templates, as the README states. This means you skip the template expansion step entirely, which can save a significant amount of processing time. The Cirrus extractor has similar output options, including '-o', '-b', '-c' for bzip compression, and '-ns' for namespaces. It also adds a 'language' and 'revision' attribute to the doc tags, reflecting the richer metadata in Cirrus dumps. This is a useful alternative if you need pre-expanded text and can handle the larger dump size. However, the README does not provide a direct comparison of speed or output quality between the two extractors, so you would need to test both on your data.
Known Limitations and When It Is the Wrong Tool
The most obvious limitation is the Windows warning. The README explicitly says 'problems have been reported on Windows due to poor support for StringIO in the Python implementation on Windows.' This is a serious caveat for any team working on Windows. The tool is not a full MediaWiki parser; it strips markup and templates, which is fine for text extraction but not for tasks that require preserving wiki structure, such as editing or round-tripping. The '--no-templates' option speeds things up but leaves template syntax in the output, which may be unacceptable for clean text. Another limitation is that the output format is custom, so you must write a parser to consume it, even though the format is simple. If you need to process Wikipedia in real time or query individual articles, this batch-oriented tool is the wrong choice; a database-backed approach like the MediaWiki API would be better.
Alternatives and How They Differ
A common alternative is the 'wikitextprocessor' library, which parses MediaWiki wikitext into a structured tree and can extract text with more fidelity. Unlike WikiExtractor, which is a command-line script that produces text files, wikitextprocessor is a Python library you embed in your own code, giving you finer control over what to extract and how to handle templates. Another alternative is the 'mwiki' tool, which is written in C and focuses on speed, but it requires compilation and has a different output model. The key difference is approach: WikiExtractor is a standalone extractor with a fixed pipeline, while libraries offer programmatic access and customization. For a one-off extraction, WikiExtractor is simpler; for complex processing or integration, a library might be more flexible. The choice depends on whether you need a quick script or a component in a larger system.
Maintenance, Licensing, and Upgrade Considerations
The project is under the AGPL-3.0 license, which has implications for redistribution and network use, though this is not legal advice. The repository shows recent activity, with a v3.1.0 release in August 2026 that specifically addresses Windows compatibility, and a v3.0.8 release that updates Python regexes. This suggests active maintenance, but the Windows fix is recent, so you should verify it works on your setup. The tool has been around since at least 2015, according to the citation in the README, which indicates stability. Upgrading from older versions may change output format or option behavior, so check the release notes before upgrading. The dependency-free design reduces upgrade risk, but the custom output format means you must maintain your own parser if it changes. The cache file for templates is another point to consider: it can become stale if the dump changes, so you must regenerate it.
Editorial conclusion
Adopt WikiExtractor if you need a dependency-free, scriptable way to turn Wikipedia XML dumps into plain text for NLP, search indexing, or corpus building, and you work on Linux or macOS. Skip it if you are on Windows, because the documented StringIO problems can break extraction, or if you need to preserve MediaWiki markup, since the tool strips it by design. Before committing, verify that your dump format matches the expected XML structure, test with a small dump and the --no-templates flag to see if template expansion is worth the time cost, and check the AGPL-3.0 license if you plan to distribute modified versions. The tool is mature and actively maintained, but its value depends on your tolerance for its output format and its Windows limitations.
Community notes