InterPlanetary Wayback: putting WARC archives on IPFS and replaying them in a browser

What the project actually does with a WARC file

InterPlanetary Wayback facilitates permanence and collaboration in web archives by disseminating the contents of WARC files into the IPFS network. IPFS is a peer-to-peer content-addressable file system that inherently allows deduplication and facilitates opt-in replication, and that pair of properties is what the whole design leans on.

The mechanism is splitting. ipwb splits the header and payload of WARC response records before disseminating into IPFS, which lets the deduplication work at the level of individual headers and bodies rather than whole records. Two archived pages that share a stylesheet share one IPFS object for that stylesheet, and the same is true of a common response header set.

The second half of the loop is the index. ipwb builds a CDXJ index with references to the IPFS hashes that are returned, and combines the header and payload from IPFS at the time of replay. So the archive is not stored in a place; the archive is a set of content-addressed objects on a peer-to-peer network, and the index is the map that says which objects make up which capture.

The implementation is described as primarily consisting of two scripts. ipwb/indexer.py is the archival indexing script, which takes the path to a WARC input, extracts the HTTP headers, the HTTP payload, and the relevant parts of the WARC-response record header, creates byte string representations, pushes them into IPFS using a locally running IPFS daemon, and creates a CDXJ file with this metadata for replay.py. ipwb/replay.py is the replay script that resolves requests for archival content contained in IPFS for replay in the browser.

The primary language is Python, the licence is MIT, and the last push was on 2026-07-24 with the project not archived.

Installing, and the IPFS daemon that has to be there first

ipwb requires Python 3.9 or newer, and it can also be used with Docker. The conventional installation is a single pip command:

$ pip install ipwb

or from source, which is how you get the development version with changes not yet released:

$ git clone https://github.com/oduwsdl/ipwb
$ cd ipwb
$ pip install ./

The dependency list in the repository tells you what the project is actually built on. warcio handles reading and writing WARC files, ipfshttpclient is the HTTP client for the IPFS API, Flask serves the replay interface, pycryptodome handles the cryptographic work, requests and beautifulsoup4 are the HTTP and HTML tooling, and surt is the library for generating canonicalised URL surt form.

The setup requirement that catches people out is that the IPFS daemon must be installed and running before you start ipwb. The daemon is named kubo, previously go-ipfs, and the documentation points at downloading kubo and taking your node online. Once installed, you start it with:

$ ipfs daemon

And there is a port conflict note worth reading, because it is the kind of thing that wastes an afternoon. If you encounter a conflict with the default API port of 5001 when starting the daemon, running the following prior to launching the daemon will change the API port to one of your choosing, here shown to be 5002:

$ ipfs config Addresses.API /ip4/127.0.0.1/tcp/5002

The Docker route handles the daemon for you. The Dockerfile in the repository is a multi-stage build: a base stage from a python image that installs a custom entrypoint script, creates folders for WARC, CDXJ and IPFS stores, downloads and installs IPFS with IPFS_VERSION defaulting to v0.31.0 and runs ipfs init, sets up the locale, and installs the requirements from requirements.txt. A node stage runs the standard JavaScript linter over the bundled assets, and there is a separate test stage. The IPFS_PATH is set to /data/ipfs so the node's data lives with the rest of the data.

Indexing a WARC, and reading the output as CDXJ

Indexing is a single command, run in a separate terminal session from the daemon, or in the same one if the daemon is in the background:

$ ipwb index (path to warc or warc.gz)

The example in the README uses a sample from the repository:

$ ipwb index samples/warcs/salam-home.warc

What the indexer does step by step is worth spelling out, because it explains the later replay behaviour. The indexer partitions the WARC into WARC Records and extracts the WARC Response headers, the HTTP response headers, and the HTTP response bodies, which are the payloads. Relevant information is extracted from the WARC Response headers, temporary byte strings are created for the HTTP response headers and payload, and these two byte strings are pushed into IPFS.

So two objects per capture, not one. The HTTP header set and the body are separate IPFS objects, which is what makes the deduplication worthwhile when a capture shares headers or assets with another.

The output goes to STDOUT by default, which means the index is a stream you can redirect into a file:

$ ipwb index (path to warc or warc.gz) >> myIndex.cdxj

That is a deliberate design choice rather than an omission. A CDXJ index is line-oriented, so writing it to stdout composes with shell pipelines and with other archive tooling, and the same format is what other web archive systems already read. The repository ships sample indexes and sample WARC files under samples/, so you can test the replay path without archiving anything of your own.

Replaying, and the Service Worker that prevents live leakage

Replay takes the path to the CDXJ index as a parameter:

$ ipwb replay <path/to/cdxj>

And ipwb also supports using an IPFS hash or any HTTP location as the source of the CDXJ:

$ ipwb replay http://myDomain/files/myIndex.cdxj
$ ipwb replay QmYwAPJzv5CZsnANOTaREALhashYgPpHdWEz79ojWnPbdG

Once started, the replay system's web interface can be accessed through a web browser, at http://localhost:2016/ by default.

The interesting engineering is in an aspect of archival replay systems that most implementations get wrong. An archival replay has to rewrite resource references so that they are dereferenced properly from the archive from around the same datetime as the root memento, and not from the live site, where the resource might have changed or gone missing. Many archival replay systems perform server-side rewriting, but it has limitations when URIs are generated using JavaScript, because the server never sees those URIs. ipwb uses a Service Worker for rerouting requests on the client side when they are dereferenced, to avoid any server-side rewiring.

That is the right choice for a JavaScript-heavy modern web, and it has a cost that the README is explicit about. The replay system utilizes a Service Worker for URL rerouting and rewriting to prevent live leakage, which is the term for a replayed page quietly fetching a live resource that the archive did not capture. For security reasons many web browsers have mandated HTTPS for the Service Worker API, with the only exception being the domain localhost. Caddy Server and Traefik are given as reverse-proxy options, and they come with built-in HTTPS support and manage TLS certificates transparently and automatically from Let's Encrypt.

So there is a real deployment constraint: to serve a replay on a domain name other than localhost, you need a reverse proxy with HTTPS in front of it, and the Service Worker will not register without it.

The banner, and the separate library it comes from

The second distinctive feature is the archival banner. The purpose of an archival banner is to highlight that a replayed page is a memento and not a live page, to provide metadata about the memento and the archive, and to facilitate additional interactivity. Many archival banners used in different web archival replay systems are obtrusive in nature and have issues like style leakage.

The fix is a Custom HTML Element, reconstructive-banner, implemented as part of the Reconstructive library and used in ipwb. The style leakage problem is the interesting one: a banner that injects its own CSS into the page can be overridden by the archived page's own styles, or can override them, so a reader cannot tell what is archive and what is original. Building the banner as a self-contained custom element keeps its styling under its own control.

Reconstructive is a separate, reusable and extendable library, and the README says it was written so that any archival replay system can use it, not just ipwb. The Reconstructive assets are bundled inside the ipwb package, since the package data in setup.py includes assets, assets/favicons and templates, and the JavaScript lint configuration in the repository explicitly ignores ipwb/assets/reconstructive*.js, which tells you those files are vendored from the other project and are not linted here.

The project also carries an evaluation.sh script and a docs/ directory with the logo and the process diagram, and the release script and the entrypoint script at the root are what the Docker image uses. The release naming is unusual, with versions like 0.2024.10.24.1853 and 0.2024.05.24.1842, which read as a date and a build counter rather than a semantic version.

Editorial conclusion

Adopt ipwb if you are running a web archive and want its WARC contents distributed over IPFS so that captures are deduplicated, replicated opt-in by the network, and replayable without a central server holding the archive. Do not adopt it as a general-purpose Wayback machine, because the project describes its own replay script as rudimentary, the whole path depends on a running IPFS daemon and a Service Worker that needs HTTPS on any domain other than localhost, and the newest tagged release dates from 2024. Before you commit, index a small sample WARC, look at the CDXJ the indexer writes, and confirm that the header and payload split produces the deduplication you are expecting on real captures, since that is the property the whole design rests on.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. oduwsdl/ipwb on GitHub
4. README
5. Releases