# DBpedia's extraction framework: a Scala pipeline you are not meant to run casually

> The extraction-framework repository is the Scala code that turns Media Wiki page source into RDF statements about each page, and the documentation is blunt about the cost: the core team's own run takes four to seven days. It ships two ways to execute, a Maven-built Docker image that carries a hardcoded path from the maintainers' machines, and a Spark path that now covers more than half of the extractors.

**dbpedia/extraction-framework** — The software used to extract structured data from Wikipedia

- Repository: https://github.com/dbpedia/extraction-framework
- Stars: 947 · Forks: 306
- Language: Scala
- License: not declared
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/dbpedia-extraction-framework

## Four components turn a page node into a graph of statements

The architecture is a pipeline, and the documentation names each stage. Source provides an abstraction over a source of Media Wiki pages. WikiParser specifies a parser that transforms a Media Wiki page source into an Abstract Syntax Tree. An Extractor is defined as a mapping from a page node to a graph of statements about that page. Destination provides an abstraction over a destination of RDF statements. Read in that order, the framework does not look up facts in a database. It walks page source, parses it, applies extractors, and writes RDF to a sink.

The repository is split into modules that mirror that flow, with a core/ directory for the core components, a dump/ directory holding the dump extraction application, plus live/, server/, mappings/, scripts/ and wiktionary/ directories and a documentation/ tree. A separate utility area covers ontology handling, with classes for representing an ontology and methods for reading and writing them, located in the org.dbpedia.extraction.ontology namespace. The ontology.owl and ontology.xml files sit at the top level, so the schema the extractors target is versioned inside the same repository.

## More than half of the extractors now run on Apache Spark

Scaling was addressed by porting the framework to Apache Spark, and the README is specific about coverage: more than half of the extractors can be executed using Spark. The catch is stated immediately afterwards. The Spark extraction process is a slightly different process that requires different execution, and the configuration is described in a separate section of the extraction process guide rather than in the main quickstart. So Spark is not a drop-in runtime for the single machine path, it is a second execution model with its own setup.

There is also a policy attached to it. The documentation asks that if possible, new extractors should be implemented using Apache Spark, and points contributors at the New Extractors guide for the process. That is a meaningful statement about the direction of the codebase, and it also creates a burden for newcomers. Writing an extractor in the Spark style means learning a distributed execution model and its configuration surface before you have contributed a single statement. For a project that only intends to modify a handful of existing extractors, the standalone route remains the more direct path.

## The MARVIN bot is the closest thing to a one command run

The documentation calls running the framework a relatively complex task and offers the MARVIN release bot as the easy way. The bot automates the whole pipeline, from downloading the ontology, mappings and Wikipedia dumps through extraction and post-processing, so a run matches what the DBpedia core team does. The commands it expects are short:

```bash
git clone https://git.informatik.uni-leipzig.de/dbpedia-assoc/marvin-config
cd marvin-config
./setup-or-reset-dief.sh
# test run Romanian extraction, very small
./marvin_extraction_run.sh test
# around 4-7 days
./marvin_extraction_run.sh generic
```

The two run modes are the point. The test mode does a very small Romanian extraction, which is the sensible way to see whether the pipeline completes before committing to a full pass. The generic mode is annotated as taking around four to seven days. That figure is the single most important number in the README for anyone planning capacity, and it applies to the easy path with a bot handling the downloads and the post-processing.

## Standalone execution is the recommended route for working on the code

If your goal is improving the framework rather than reproducing a release, the documentation pushes you the other way. Standalone execution, as described in the QuickStart guide, means running the extraction framework alone, and the recommendation is emphatic: it is highly recommended, since during this process you will learn a lot about the extraction framework. The trade-off is that you take on the steps the bot would have done for you.

That makes the choice between the two documented paths a choice about who handles the inputs. MARVIN downloads the ontology, the mappings and the Wikipedia dumps, then post-processes the result. Standalone leaves those tasks with you and gives you a pipeline you can debug. The documentation also points to a Debugging Guide for that case, which is a fair signal of where the friction sits. A team that wants a repeat production run should look at the bot; a team that wants to understand or change extraction behaviour should read the quickstart instead.

## The Docker image carries a path from the maintainers' machine

The Dockerfile in the repository root builds the project in a Maven image with JDK 8 and rewrites paths that point at the core team's workspace. Here is the whole file:

```dockerfile
FROM maven:3-jdk-8

WORKDIR /dbpedia

RUN bash -c 'mkdir -p /dbpedia/extraction-framework-files/{basedir,logdir}'

ADD . /dbpedia/extraction-framework

VOLUME /dbpedia/extraction-framework-files

RUN grep -rl "/home/marvin/workspace/active/basedir" ./* | xargs sed -i 's\/home/marvin/workspace/active/\/dbpedia/extraction-framework-files/\g'

RUN cd extraction-framework/ && mvn clean install
```

The search and replace step over the whole tree is a hardcoded assumption: the build expects a workspace layout belonging to a developer named Marvin, and the image rewrites those references into a mounted volume with basedir and logdir subdirectories. A volume is declared at that path so run state survives the container. The final build line runs a full Maven install of the extracted project. Useful as a reference for reproducing a build, but the path rewriting shows that the image was built for one specific workstation rather than as a general distribution.

## Scala 2.8, a 2016 release tag, and a September 2026 push

The framework is written in Scala 2.8, according to the README, which is worth weighing next to the other dates here. The published releases are named by the data vintage rather than by a semantic version: DBpedia_2015-10 tagged on 2016-07-25, DBpedia_2015-04 from 2015-07-17, and DBpedia_2014 from 2015-06-12. The repository itself is not archived, and the last push was on 2026-09-28, so the code is still being changed even though the newest listed release is more than a decade old.

That combination is a real signal about the upgrade path. Activity in the repository does not produce a new release tag, so a consumer pinning to a release gets a fixed artefact while the master branch moves on. An upgrade therefore means either building from source, which the Maven build in the Dockerfile supports, or waiting for a tagged release that is named after Wikipedia data rather than a numbered version. Neither option is unusual for research infrastructure, but it should be a deliberate decision rather than an assumption of a normal package release cycle.

## The licence is stated as GNU GPL, and the docs point to the community for the rest

The README states that the framework is available from the DBpedia GitHub repository under the GNU GPL License. The top-level repository listing does not include a separate LICENSE file, so the exact GPL version and any accompanying text are not pinned in the files you can see from the repository root. If you intend to redistribute code derived from the framework, read the repository's own licence files directly before you build on it, and treat the README line as a pointer rather than the terms themselves.

The surrounding guidance is community-oriented. There is a home page at dbpedia.org, developer documentation at dev.dbpedia.org/Extraction, and a Slack workspace where the dev-team channel is described as the main point for development updates and discussion. The README also notes that the change log may reveal more recent developments, and that newer configuration options live in the project wiki, which is a useful signal that the wiki, not the README, holds current configuration detail. A Developer's Certificate of Origin requirement appears in the contribution guidelines, and translations and further contribution rules sit alongside it.

## Conclusion

Adopt this framework when your own pipeline needs the same Media Wiki to RDF conversion, or when you intend to add an extractor to DBpedia itself, and accept a multi-day batch job and a Scala 2.8 codebase. Do not adopt it for quick lookups on Wikipedia content, since the repository holds no prebuilt dataset, only the code that produces one. Before starting, read documentation/quickstart.md for the standalone path, check the GNU GPL terms the README states against the terms your own distribution carries, and confirm that your cluster matches the Apache Spark setup described in documentation/extraction-process.md before planning a generic run.

## FAQ

### How long does a full DBpedia extraction run take?

The MARVIN bot command for a generic run is annotated as taking around four to seven days, and that is the documented duration for the automated path. The test mode runs a much smaller Romanian extraction.

### What does the DBpedia extraction framework actually produce?

It produces RDF statements. A page source is parsed into an AST by the WikiParser, an Extractor maps a page node to a graph of statements about that page, and the Destination package handles where those RDF statements are written.

### Does the DBpedia extraction framework support Apache Spark?

Yes, and the README says more than half of the extractors can be executed using Spark. The Spark path is a slightly different process that requires different execution, documented separately in the extraction process guide.

### Which Scala version does the DBpedia extraction framework use?

The README states that the framework is written in Scala 2.8. The repository builds with Maven, and the bundled Dockerfile uses the maven:3-jdk-8 base image.

## Sources

- [dbpedia/extraction-framework on GitHub](https://github.com/dbpedia/extraction-framework)
- [Issues](https://github.com/dbpedia/extraction-framework/issues)
- [README](https://github.com/dbpedia/extraction-framework/blob/master/README.md)
- [Releases](https://github.com/dbpedia/extraction-framework/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/dbpedia-extraction-framework