Marginalia Search: an index of the small, old and weird web, and a versioning scheme built around crawling

A search engine whose stated goal is to be small

The repository description is unusually blunt about what this project is: an internet search engine for text-oriented websites, indexing the small, old and weird web. The README expands on that into a mission statement. The aim is to develop new and alternative discovery methods for the internet, and the project describes itself as an experimental workshop as much as a public service, with the overarching goal of elevating the more human, non-commercial sides of the internet.

The side goal is a hardware claim, and an unusual one for a search engine: do this without requiring datacenters and enterprise hardware budgets, so the operation can run on affordable hardware with minimal operational overhead. The long term plan, as stated, is to refine the engine until it provides enough public value to be funded through grants, donations and commercial API licenses, with non-commercial share-alike always free. That is a funding model as much as a licence policy, and it explains why the search index is deliberately not trying to be everything to everyone.

The repository topics back this up: alt-search, indexer, internet-search, language-processing, no-cloud, search-engine, self-hostable, small-web, web-crawler and web-scale. Note that web-scale and no-cloud sit side by side in that list, which is a tension worth holding on to. The README asks for modest hardware while the topic list claims scale. The written goal is the modest one, so that is the one to plan against.

Versions are numbered after crawl operations, not code

The versioning scheme is the clearest statement of intent in the repository, and it explains why version numbers look odd. Marginalia uses a modified Calendar Versioning where the first two pairs of numbers are a year and month coinciding with the latest crawling operation, and the third number is a patch number.

version
yy.mm.VV
crawl

The example given is `23.03.02`, described as a release with crawl data from March 2023, released in May 2023, and the second patch for the 23.02 line. The consequence for operators is spelled out. Versions sharing a year and month are compatible with each other, or offer an upgrade path where the same dataset can be reused. Across different crawl sets, data format changes may be introduced, and you are generally expected to re-crawl from scratch, because crawler data has roughly the shelf life of the project's own major release cycles.

That last point is the one to plan around. The README says that after about two to three months the crawl data gets noticeably stale, with many dead links. So a Marginalia deployment is not a thing you install and leave; it is something whose quality decays on a clock the project sets for you.

The release history shows both halves of the scheme. v24.01.2 is dated 2024-02-18 and calls itself a second corrections release addressing technical debt and stability. v24.10.0 is dated 2024-10-14, with a preceding v24.10.0-rc. Those dates track a crawl anchored in late 2024 rather than the day a commit landed.

Setting up locally, and the model data you must fetch first

Setup is delegated rather than spelled out. The README points contributors to `run/readme.md` for a local test environment and to docs.marginalia.nu for the wider documentation, and it tells you one thing you must do before compiling: run `run/setup.sh`. That script downloads supplementary model data which the code needs to run, and which is also needed to run the tests.

./run/setup.sh
./gradlew build

The build itself is Gradle, which the repository layout makes visible through `build.gradle`, `settings.gradle`, `gradlew`, and a set of specialised gradle files including `docker.gradle`, `dist.gradle`, `protobuf.gradle` and `srcsets.gradle`. That spread of build files tells you the project produces several distinct artifacts rather than a single library, with container and distribution concerns split out.

If you plan to work on the code rather than just run it, the README has a separate pointer to `doc/ide-configuration.md`, which is a detail that suggests the build expects an IDE to be configured in a particular way.

One deliberate piece of advice for developers: crawling is discouraged, and sample data is available instead. Given the hardware numbers below and the shelf life of crawl data, using the sample set is also the sane way to make a change before attempting a real crawl.

Hardware: 32 GB of RAM, and terabytes of disks

The hardware section of the README gives two different answers, and both are honest about their scope. A production-like environment requires a lot of RAM, ideally enterprise SSDs for the index, and some additional terabytes of slower harddrives for storing crawl data. It can be made to run on smaller hardware by limiting the size of the index. The system will definitely run on a 32 GB machine, possibly smaller, but at that size it may not perform very well, as it relies on disk caching to be fast. A local developer deployment is possible with much smaller hardware and a much smaller index.

Set those side by side and the shape of the system becomes clear. The index wants fast storage, the crawl archive wants cheap bulk storage, and performance depends on disk caching rather than on memory alone. The 32 GB figure is a floor for a working system, not a production target, and the README qualifies it itself.

This is also where the comparison to conventional search infrastructure is most pointed. The project names datacenters and enterprise hardware budgets as the thing it is trying to avoid, while still asking for terabytes of disk in production. The way it reconciles that is operational overhead and hardware class rather than total capacity, and for anyone evaluating this, the question to ask is how often you intend to re-crawl, because that is what drives the archive size.

Licensing that GitHub cannot summarise

Here is a straightforward contradiction between what the repository records and what the README says. The repository metadata reports the license as NOASSERTION, which is how GitHub represents a licence it cannot classify as a single standard one. The README describes something more specific: the bulk of the project is available with AGPL 3.0 with exceptions, some parts are co-licensed under MIT, and third party code may have different licences. The tree supports that description, since it holds a `LICENSE.md` file and a `third-party/` directory.

Both facts can be true at once. AGPL 3.0 with exceptions, MIT co-licensing and vendored third-party code together genuinely do not reduce to one licence identifier, so NOASSERTION is the correct classification for the repository as a whole and the README sentence is the correct description of practice. What NOASSERTION does not do is tell you what terms apply to the file you are about to copy.

The README gives you the rule for finding out: see the appropriate readme.md or license.md in the relevant part of the tree, with `code/` documented in `code/readme.md`. So if the AGPL matters to you, which it will if you plan to run a modified instance as a service, read the per-directory terms rather than relying on the repository-level label.

What the last two releases actually changed

The release notes are more informative than the README about how the engine works, and they document an unusual amount of candour about past mistakes. The v24.10.0 notes say the query parsing and evaluation model was re-written from scratch, because the original model was very flaky and hard to maintain. The replacement performs query segmentation and introduces a graph-based model for bag-of-words query evaluation. The same release added full phrase matching, so that quoted search terms work better and result ranking takes word order in the query into account. It also added a screenshot capture service where screenshots are fetched and refreshed by page views on the site, plus a CSS overhaul with dark mode, and result pagination that the notes openly call fake, because each index node fetches the total number of requested results while only the best results across all nodes are selected in the query service.

The v24.01.2 notes cover a different register. Domain ranking was overhauled to use JGraphT in place of what the notes call the primordial home-cooked pagerank implementation, and several services start faster because loading the domain id blacklist is delayed. Sideloading gained the ability to handle reddit posts, and QueryStrategy and TemporalBias were exposed through the query-service API. The most revealing item is a fix: a series of actions in the operations gui could lose crawl data if a re-crawl happened before a spec-generated crawl was loaded, mitigated in several ways.

Read together, these notes describe an engine whose ranking is graph-based and whose operational tooling is where the sharp edges are. The v24.10.0 pagination description in particular is the kind of detail that tells you how much of a search interface here is presentation over a distributed query path.

White label search, with a caveat about how configurable it is

The system can be run as a copy of Marginalia Search or as a white-label search engine for your own data, whether that data is crawled or side-loaded. That is a genuinely different proposition from self-hosting an off-the-shelf engine, and it is why the repository is worth reading even if you have no interest in the small web.

The README attaches an important condition to that offer: at present the logic is not very configurable, and a lot of the judgements made are based on the Marginalia project's goals, with additional configurability being worked on. Read plainly, that means the ranking and parsing behaviour encode what this project considers a good page, and you are not going to retune that from a config file. The v24.10.0 release notes support the same picture, since a graph-based bag-of-words evaluation model with query segmentation is a design choice rather than a knob.

There is also a demo of the set-up and operation of the self-hostable barebones mode linked from the README, which is the fastest way to see whether the operational model suits you before committing disk.

Funding runs through the NGI0 Entrust Fund, established by NLnet with financial support from the European Commission's Next Generation Internet programme, under grant agreement No 101069594. The README links a donation page as well, and the licensing arrangement in the previous section, non-commercial share-alike always free, is consistent with a project that depends on grant money and small donations rather than on search revenue.

Editorial conclusion

Marginalia Search fits a specific appetite: finding text-oriented pages that commercial engines have little reason to surface, and running such an engine without a datacentre budget. The README supports that goal plainly, and the crawl-anchored versioning scheme is the most interesting engineering decision in the repository. Two things deserve checking before you commit. First, the newest tagged release is v24.10.0 from October 2024 even though the repository was pushed in September 2026, so you are working from a roughly two-year-old release with newer commits behind it. Second, the licence is messier than a single label suggests: the repository records NOASSERTION while the README describes AGPL 3.0 with exceptions and some MIT parts. Start from `run/readme.md`, run `run/setup.sh` before compiling, size your disk against the crawl data you want, and read the per-directory licence notes before you embed anything.

Frequently asked questions

Official sources

1. Issues
2. MarginaliaSearch/MarginaliaSearch on GitHub
3. Project website
4. README
5. Releases