wtf_wikipedia turns wikitext into JSON without an API key

A parser for wikitext, not a client for the API

wtf_wikipedia is described in its own package description as a parser that turns wikiscript into json. That distinction matters. The library does not treat Wikipedia as a data source with a friendly schema. It treats wikitext, the markup language Wikipedia articles are written in, as a parsing problem, and it solves that problem so you can get at the contents of an article from either raw markup or a fetched page.

The project is candid about the difficulty. The README points at academic work on parsing wikitext and says it is very, very hard, and it lists the reasons why: a partial implementation of inline css, nested elements that do not honour the scope of other elements, a language with no errors, and deep recursion of similar syntax.

That framing explains the shape of the API. You do not ask for fields by a schema. You parse the document and then interrogate it, asking for an infobox key, for sentences, for links, for images, for a section by title. The author also gives a reason for doing this at all, that we put our information where we cannot take it out, and that this kind of tool helps with it.

The package is MIT licensed, the homepage is huh.wiki, and the default branch is dev.

Fetching a page and pulling a single infobox value

The entry point is a fetch helper that takes a title and returns a parsed document:

import wtf from 'wtf_wikipedia'

let doc = await wtf.fetch('Toronto Raptors')
let coach = doc.infobox().get('coach')
coach.text() //'Darko Rajaković'

That example shows the shape of the whole library in three lines. Fetch a page, reach into its infobox, name a key, and get back a value object whose text method returns a plain string.

A second argument selects the language edition, as in the Glastonbury example in the README where the fetch is called with 'en' and the first sentence of the document comes back as readable prose. The value returned by text is not the raw markup with tags stripped mechanically. It is the result of interpreting the markup, which is why template expansion and link resolution are part of the job.

There is also a browser entry point, since the package ships a client build and the README shows it loaded from unpkg with a callback:

<script src="https://unpkg.com/wtf_wikipedia"></script>
<script>
  wtf.fetch('Radiohead', { 'Api-User-Agent': 'Name your script here' }, function (err, doc) {
    let members = doc.infobox().get('current members')
    members.links().map((l) => l.page())
    //['Thom Yorke', 'Jonny Greenwood', 'Colin Greenwood'...]
  })
</script>

The Api-User-Agent option in that snippet is a courtesy to the API rather than a parser feature, and it is the one piece of the fetch path that is not really about wikitext.

Text output is where template handling shows up

The text method is the part of the library that has to make hard decisions, because a Wikipedia sentence is usually a template invocation with prose arguments around it. The README gives this example:

let str = `[[Greater_Boston|Boston]]'s [[Fenway_Park|baseball field]] has a {{convert|37|ft}} wall. <ref>Field of our Fathers: By Richard Johnson</ref>`
wtf(str).text()
// "Boston's baseball field has a 37ft wall."

Three different mechanisms are visible in one line of input. A piped wiki link becomes the label text. A convert template becomes a plain number with its unit. A reference tag vanishes, along with its citation content, because a footnote is not part of the article prose.

That is the behaviour you want from a parser, and it is also where the risk of silent divergence from the rendered page sits. A template the parser does not recognise may survive into the output as literal markup or as a best guess, and you will not notice unless you compare the two.

The documentation also notes that wikitext has no errors, so a malformed input does not throw. The parser will always return something, and the something may be empty or partially interpreted.

JSON output comes in three preset sizes

The json method returns the data from a page, and the README describes the default output as really verbose. Three presets give you a choice of size:

doc.json('sm') // title, description, infoboxes, categories
doc.json('md') // sm + images, links, templates, references, coordinates, plaintext
doc.json('lg') // md + all section data

The progression is cumulative. The small preset is enough for a search result card or a catalogue entry. The medium preset adds what most people actually want from an article: the images, the links, the templates that were used, the references, the coordinates and the plain text version. The large preset adds the full section data.

You can also cherry pick instead of taking a preset, using the same accessor style as the rest of the API:

// get just the links:
doc.links().map((link) => link.json())
//[{ page: 'Theatrical superstitions', text: 'supersitions' }]

// just the images:
doc.images()[0].json()
// { file: 'Image:Duveneck Whistling Boy.jpg', url: 'https://commons.wiki...' }

// json for a particular section:
doc.section('see also').links()[0].json()
// { page: 'Slide Whistle' }

Note that section access is by title, so a section rename on Wikipedia changes what your code finds.

A CLI, TypeScript types and an ESM only package

The package manifest shows a project with real engineering around it. The current version in package.json is 10.5.0, the package type is module, and the exports map separates browser and Node builds, with .mjs and .cjs bundles plus matching .d.ts and .d.cts type declarations. The engines field requires Node 18.0.0 or newer.

There is a command line entry point registered as wtf_wikipedia, pointing at cli.js, and a man page is shipped at builds/man/wtf_wikipedia.1. So the library is not only for embedding in a script, it can be driven from a shell.

The build tooling is rollup with a config at the repository root, and the scripts section shows the usual set of tasks: build runs a version script and then rollup, build:all also runs the workspace packages, lint is eslint, and the unit tests run under tape with the --strict flag piped through tap-dancer. There is a codecov configuration and a tsconfig for type checking.

The repository layout is a pnpm workspace with pnpm-lock.yaml and pnpm-workspace.yaml, and the declared packageManager is pnpm 12.5.1. There is a plugins directory, which is where the extension points live, and an AGENTS.md alongside contributing.md and a changelog.md.

Bulk extraction is somebody else's tool

The README is explicit about the limits of the scope. Parsing the whole of English Wikipedia is not what this library is for; the recommendation is to use it in conjunction with dumpster-dive, a separate project by the same author that handles the dump files.

The install line for that route is a global npm install:

npm install -g dumpster-dive

The distinction is one of input. wtf_wikipedia parses a page of markup, one article at a time, whether that markup comes from the API through fetch or from a string you pass in. A dump is a set of multi gigabyte XML files, and handling those needs streaming, index management and its own progress reporting. Mixing the two concerns would make the library heavier for the common case.

The same split shows up in the API surface. wtf_wikipedia has no caching layer, no rate limiting and no retry logic for the fetch helper, because it treats fetching as an aside to parsing. If your use case needs thousands of pages, you will be writing that layer yourself or adopting the dump tool.

Version numbers and release rhythm

The repository is not archived and the last push was on 2026-10-16, so work is landing on the default branch, which is dev rather than a stable branch. That is worth noting, because the branch layout means the published package and the repository head can differ in shape.

The published releases tell a slower story. Version 10.4.2 was tagged on 2026-05-27, 10.4.1 on 2025-12-12 and 10.4.0 on 2025-02-02. The manifest already reads 10.5.0, so the version on the working tree is ahead of the newest tag, which is the normal shape for a project that bumps its version on the way to a release.

The project is MIT licensed, and the manifest marks the package with provenance publishing enabled, which ties published artefacts to the build. Nothing in the licence is unusual for a JavaScript library, though any specific compliance question belongs with your own counsel.

The documentation lives partly in the repository under docs/ and partly on Observable, where the .text() and .json() docs are hosted. The examples in this article come from that README, and the fuller API reference is split across those two places.

Editorial conclusion

wtf_wikipedia is a good fit for a Node project that needs one article at a time and wants typed output rather than a scraped HTML page, since the .json() presets and the TypeScript declarations cover the common cases without an API key. It is the wrong tool for bulk extraction, where the README itself points at dumpster-dive, and it is not a Wikipedia client, so it does not solve fetching, caching or rate limiting for you. Before depending on it, install wtf_wikipedia, parse one page you already know the contents of, and compare the .text() output with the rendered article, because template handling is where a parser like this is most likely to differ from what a reader sees.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. spencermountain/wtf_wikipedia on GitHub