Zimit: turning any website into an offline ZIM archive

What Zimit does and what a ZIM file is for

Zimit is a scraper allowing you to create a ZIM file from any website, and the description puts it as making a ZIM file from any web site and surfing offline.

The ZIM format is the container that makes the result useful. It is the format Kiwix readers use, and the point of producing one rather than a directory of HTML files is that a ZIM is a single file that is searchable, self-contained and can be served by a reader with no network and no original site behind it. That matters for archiving, for distributing a snapshot of documentation, and for anything that has to keep working after the original site changes or disappears.

Zimit is part of the openZIM organisation, and it adheres to openZIM's contribution guidelines. The README also says it has implemented openZIM's Python bootstrap, conventions and policies v2.0.0, which is the sort of thing that tells you the project follows a shared house style rather than inventing its own.

The technical background section describes the approach in three sentences. Zimit runs a fully automated browser-based crawl of a website property and produces a ZIM of the crawled content, and it runs in a Docker container. The system runs a website crawl with the Browsertrix Crawler, which produces WARC files, and then converts the crawled WARC files to a single ZIM using warc2zim. The zimit.py file is the entrypoint for the system.

That two-stage design is worth pausing on, because it is the reason the tool is as capable as it is. The crawl is done by a real browser engine, so JavaScript-rendered pages, forms and dynamic content behave the way they do in a browser rather than the way they do for a naive HTTP fetch.

The container, the output volume and the keep flag

Zimit is intended to be run in Docker, and the image is published at the openZIM GitHub container registry. The Dockerfile shows the design: it starts from the Browsertrix Crawler image at version 1.14.1, adds the deadsnakes PPA for a newer Python on Ubuntu, and installs python3.14-venv so the Zimit code can live in its own virtual environment and not conflict with the crawler.

It also disables the Chrome upgrade during the build, and sets up the /output directory as a placeholder for the default output location.

The example commands show the three entry points:

docker run ghcr.io/openzim/zimit zimit --help
docker run ghcr.io/openzim/zimit warc2zim --help
docker run  -v /output:/output ghcr.io/openzim/zimit zimit --seeds URL --name myzimfile

The middle of that list is worth noting. The same image exposes both the zimit entrypoint and warc2zim directly, so you can run the conversion step on WARC files you captured some other way. That is a deliberate consequence of building on the existing components rather than reimplementing them.

After the crawl is done, warc2zim writes the ZIM to the /output directory, which should be mounted as a volume so the ZIM is not lost when the container stops. That is the single most common way to lose the result of a long crawl, and the README states it plainly.

The --keep flag is the debugging aid. With it, the crawled WARCs and a few other artifacts are also kept in a temp directory inside /output. The README describes it as in case of failure, WARC files and other temporary files, stored as a subfolder of the output directory, are always kept, otherwise they are automatically deleted, and the flag makes them survive even in case of success. Since the WARC is the intermediate representation, keeping it is what lets you re-run the conversion without re-crawling.

To rebuild the image locally, the README gives:

docker build -t ghcr.io/openzim/zimit .

The parameters and what each one controls

The image accepts a set of Zimit parameters and, as the README puts it, as well as any of the Browsertrix crawler and warc2zim ones. So the tool is a front end over two existing CLIs rather than a closed set.

Three are required. --seeds URL is the url to start crawling from, and multiple URLs can be separated by a comma, though it is usually not needed since these are just the seeds of the crawl. The first seed URL is used as the ZIM homepage, which is a small but important detail: it decides what a reader sees when they open the archive. --name is the name of the ZIM file, and --output is the output directory, defaulting to /output.

The rest are limits and filters. --pageLimit U limits capture to at most U URLs, which is the parameter you reach for when you want a bounded archive rather than a whole site. --scopeExcludeRx takes a regex and skips matching URLs from crawling, and it can be specified multiple times, with the example being a pattern that excludes URLs containing a query parameter, a signup landing path or another parameter. --workers N sets the number of crawl workers to run in parallel.

--waitUntil is a Puppeteer setting for how long to wait for page load. The default is load, but for static sites --waitUntil domcontentloaded may be used to speed up the crawl, to avoid waiting for ads to load for example. That is a genuinely useful optimisation for documentation sites and it is the kind of setting a general crawler cannot offer.

The example for scopeExcludeRx is instructive about how these filters interact. A pattern like a query parameter, a signup path or another parameter will exclude URLs containing any of them, which is how you stop a crawl from wandering into a search results page or a session-creation endpoint and looping there.

Ad blocking by default, and how to turn it off

The README carries a note about the image's behaviour that is worth reading before you archive anything.

The image automatically filters out a large number of ads by using the three blocklists from anudeepND. If you do not want this filtering, you can disable the image's entrypoint in your container with docker run --entrypoint="" ghcr.io/openzim/zimit ... .

The mechanism is visible in the Dockerfile. Three lists are downloaded at build time, adservers.txt, CoinMiner.txt and facebook.txt, concatenated into /etc/blocklist.txt, and an entrypoint script appends that file to /etc/hosts before executing the container command. The comment in the Dockerfile explains the ordering: the list is intentionally run post-install so it is not cached in earlier layers, because the URL stays the same but the content is updated.

That is a neat piece of build hygiene. A hosts-file blocklist baked into an image layer would freeze the list at build time, so a rebuild would be needed to pick up new entries. Running it at container start means the list is fetched fresh each time, and the image stays small because the lists are not stored in the layer.

The trade-off is that the crawl resolves hostnames against a modified hosts file, so any domain on those lists is unreachable from inside the container. For most archives that is exactly what you want. For a site whose own assets or API live on a domain that appears in a shared blocklist, it is not, which is why the README gives you the escape hatch. Disabling the entrypoint is a blunt instrument, since it also removes the rest of the entrypoint's behaviour, but it is the documented way to get the unfiltered behaviour.

The limits, which mostly belong to the components underneath

The README is honest that a versatile tool cannot archive any website without limits, and it is careful about where the limits come from.

Most capabilities and known limitations are documented in the warc2zim README. There are also some limitations in the Browsertrix Crawler, which is used to fetch the website, and in wombat, which is used to properly replay dynamic web requests, but these are not, as the README puts it, yet clearly documented. That is a useful thing to know before you file a bug: the behaviour you are seeing may belong to a component three projects away.

The practical limitations follow from the browser-based approach. A site that requires a login produces an archive of the login page rather than of the content behind it. A site that personalises by cookie produces an archive of whatever the crawler saw. Infinite scroll is captured only as far as the crawl went. And because the crawler runs a real Chrome, the resource cost of an archive is far higher than a plain HTTP fetch, which is a good reason for the --pageLimit and --scopeExcludeRx parameters to exist.

The content licensing question sits outside the tool. Scraping a site and republishing the result is governed by the copyright in that content and by the terms of the site, and neither Zimit nor the ZIM format changes that. The README does not discuss it, and the tool does not make it a non-issue.

There is also a historical note about the 1.x line. While Zimit 1.x relied on a Service Worker to display the ZIM content, that is no longer the case since Zimit 2.x, which has no special requirements any more. That matters for anyone who remembers the older approach and for anyone wondering what they need to install on the reader side.

How Zimit compares to wget and to a hand-run Browsertrix crawl

Two comparisons make the choice clearer.

The first is against wget or httrack for making a local copy of a site. Those are HTTP-level tools. They fetch URLs and save what comes back, which is fast and light and produces a directory of files that mirror the server's structure. What they cannot do is execute JavaScript, so a site that renders its content in the browser gives you a set of empty shells. Zimit runs a real browser, which is exactly what the WARC format is designed to capture, so the archive contains what a visitor would have seen.

The second comparison is against running the Browsertrix Crawler yourself and converting the WARC with warc2zim. That is entirely possible, since both are separate projects with their own documentation, and Zimit's own image exposes warc2zim directly. The difference in approach is packaging and defaults. Zimit assembles the two into one container with a sane configuration, adds the ad blocklists, handles the output volume, and gives you a single set of parameters that spans both stages. If you already run Browsertrix in your pipeline, Zimit is redundant for you.

So the practical choice is whether you need browser rendering. For a static documentation site, wget is faster and the result is easier to serve. For anything with JavaScript, a ZIM is the only one of the two that produces something worth reading offline, and that is the case Zimit was built for.

Editorial conclusion

Adopt Zimit when you need a self-contained offline copy of a site, such as documentation, an archived news site or material that must remain readable without a network. Do not expect it to capture sites that require login or that are heavily personalised, since the limitations of the underlying crawler apply. Verify first that the URL you want is reachable without authentication, that the default page limit is not truncating the crawl, and that the blocklist filtering is acceptable to you, because the image disables it only when you override the entrypoint.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. openzim/zimit on GitHub
4. README
5. Releases