# Mwmbl: a community-built web search index

> Mwmbl is an open source, non-profit search engine with a centralised index and volunteer-operated crawlers. Its unusual fixed-page hash design may reduce storage costs, but the project states that this approach has not been tested at large scale.

**mwmbl/mwmbl** — An open source, non-profit web search engine

- Repository: https://github.com/mwmbl/mwmbl
- Website: https://mwmbl.org
- Stars: 1,875 · Forks: 98
- Language: Python
- License: AGPL-3.0
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/mwmbl-mwmbl

## Mwmbl's centralised index and distributed crawler

Mwmbl addresses the cost and incentives of conventional web search, especially the README's concern that advertising can encourage pages and ad placements designed to generate impressions rather than answer queries directly. It is intended for people looking for an ad-free, non-tracking search option, as well as developers and volunteers who are willing to contribute computing resources, links or code. The hosted service is available at https://mwmbl.org, while operational and developer documentation is published separately at https://book.mwmbl.org.

The search data and contribution methods are not fully distributed. Mwmbl uses one centralised index, while crawling is distributed among servers operated by volunteers. This differs from the README's description of YaCy, where the index itself is distributed through a peer-to-peer network. Concentrating the searchable index gives Mwmbl a conventional retrieval target. Donating crawler capacity does not give a volunteer ownership of that index or a separate local copy of every indexed item.

The crawler is written primarily in Python, but the repository is not a small search script. Its dependencies include Django, PostgreSQL support through `psycopg2-binary`, Redis, NumPy, SciPy, scikit-learn and XGBoost. The repository also contains `mwmbl_rank`, a Rust extension built through `maturin`, while `front-end/` holds a separate JavaScript front end. A production image builds that front end, creates a Python 3.11 environment, compiles the native extension and runs the service.

## Fixed 4096-byte pages replace a conventional inverted index

The index is the main technical claim in the README. Mwmbl does not describe its storage as a conventional inverted index that maps every term to a list of items. Instead, it uses a fixed number of pages, each currently 4096 bytes, intended to match one memory page. Each page holds a compressed list of items.

For a term that should make an item eligible to rank, Mwmbl hashes the term to a number from zero through `N - 1`. The item is stored in the page selected by that hash. At query time, the engine hashes the terms in the user's query, loads the corresponding fixed-size pages, filters their items to those that actually contain the requested terms, and then ranks the matching items.

Because several query terms can hash to the same page, the search operation still needs an exact containment check after loading. Hashing narrows where the engine must read; it does not by itself prove that an item ranks for a term. Compression allows an item to carry more than one ranking term, which is intended to keep the index smaller than the usual inverted-index layout.

This is an explicit cost hypothesis, not a demonstrated operating advantage. The README says that compressing item lists should permit a smaller index, then immediately notes that the idea "has yet to be tested on a large scale." That admission matters for technical evaluation. The layout is interesting, but the documentation does not provide a scale test, collision analysis, storage comparison or retrieval measurement.

## Running Mwmbl's crawler with Docker Compose

The clearest first contribution does not require assembling the full Django development environment. The README directs volunteers with spare bandwidth and processing capacity to the Docker Compose crawler documentation. The repository defines a `mwmbl-crawler` service built from `Dockerfile.crawler`, plus a `redis` service used as its work queue.

Before starting it, the documented Compose configuration requires `MWMBL_API_KEY` and `MWMBL_CONTACT_INFO`. The crawler service is configured with `CRAWLER_WORKERS=4`, `CRAWL_THREADS=20`, `CRAWL_DELAY_SECONDS=0.0` and `REDIS_URL=redis://redis:6379`. Its state is stored in the named `crawler_data` volume at `/home/mwmbl/.mwmbl`. The service uses `restart: unless-stopped` and limits its JSON log to `50m` with three files.

The README excerpt does not provide the final invocation for this Compose setup, and the crawler documentation is external, so this article does not invent a command. Follow the linked `running-the-mwmbl-crawler-with-docker-compose` procedure after setting the required values. A Firefox extension is another contribution path. According to the README, it retrieves search results from commercial engines so they can be used to evaluate Mwmbl's own ranking, and it does not access personal data.

## Local development needs PostgreSQL, Redis and Python 3.11

For engine development, the Makefile requires `uv`, PostgreSQL and a Redis instance at the configured `REDIS_URL`. Its default is `redis://127.0.0.1:6379`, while tests use the separate database `redis://127.0.0.1:6379/15`. That separation matters because the test target flushes its Redis database. The PostgreSQL user must have `CREATEDB` permission so Django can create its test database. The project metadata restricts Python to `>=3.11,<3.12`.

Set `DATABASE_URL` before creating or using the development database. The Makefile deliberately supplies no default. Once `DATABASE_URL`, PostgreSQL, Redis and `uv` are available, this is the documented dependency installation sequence:

```bash
uv sync
```

This installs the locked Python environment. The repository uses a Rust build backend through `maturin`, so a complete native build may also need the Rust toolchain described in `Dockerfile`. Install the repository's Git hooks once:

```bash
make install-hooks
```

After that, the documented Makefile quick-start targets apply migrations, run the test suite and start the development server:

```bash
make migrate
make test
make run
```

These are separate operations rather than one bundled tutorial. A first successful local milestone is `make test`, followed by `make run`; whether it opens the intended pages also depends on the development configuration documented in the Mwmbl book.

## Formatting and static checks reproduce CI expectations

The repository pins its development tools and centralises the normal checks in `pyproject.toml` and the Makefile. Python code is formatted and linted with `ruff`, while `ty` performs type checking. The README says both tools are pinned in the `dev` dependency group so developers use the same versions.

After `uv sync`, the manual check commands are documented as:

```bash
make check
```

That target is described as the format check, lint and type check performed by CI. To apply formatting and safe automatic lint fixes, use:

```bash
make fix
```

The type-checking configuration has a significant caveat. `ty` is limited to error-level diagnostics, with a substantial warning backlog downgraded to warnings under `[tool.ty.rules]`. One documented cause is that Django's `Model.objects` is invisible to `ty`, and the project does not use a `django-stubs` plugin. Therefore, a clean `make check` is useful for reproducing the configured CI gate, but it is not evidence that all Django typing issues have been found.

There are no GitHub releases to define a versioned upgrade path for operators. The project version in `pyproject.toml` is `0.1.0`, but that package metadata should not be read as a release cadence. The repository is not archived and its last push was on 2026-10-16, which indicates recent repository activity. Dependency upgrades still require reading `uv.lock` and the changed configuration rather than waiting for a documented release train.

## Index scale, crawl identity and operational trade-offs

The clearest limitation is also the one the README states plainly: Mwmbl's index is currently much smaller than those of commercial search engines. A smaller index affects recall before ranking quality becomes relevant. If relevant pages have not been crawled, stored or curated, neither the hash lookup nor XGBoost-based ranking can surface them. The README's ambition to replace commercial engines is therefore a direction, not evidence of equivalent coverage.

The large-scale validity of the fixed-page design remains unconfirmed. Hashing distributes terms across `N` pages, but the documentation excerpt does not explain load balancing, hot pages, update amplification, deletion, recovery or how compressed item growth affects page boundaries. Those omissions do not prove that the implementation lacks mechanisms, because the full book and source are outside this description. They do mean an engineer cannot assess those points from the README.

Crawler deployment also creates a boundary between contribution and ownership. Volunteers supply crawling resources, and community members curate results, but the index remains centralised. Operators who require every index shard to remain under their own control should examine YaCy instead. Those who only want to query other indexes can consider SearXNG, which the README identifies as a meta search engine. Marginalia Search is also different in intent: it favours text-heavy sites, while Mwmbl aims at the broader goal of replacing a general commercial engine.

## Conclusion

Mwmbl fits contributors who want to support a community-operated search index and accept that it is much smaller than commercial engines. It is a poor fit for organisations requiring a documented large-scale retrieval path or predictable relevance. Evaluate the current coverage of mwmbl.org first, then check Python 3.11, PostgreSQL and Redis requirements before attempting a local build.

## FAQ

### What is Mwmbl used for?

Mwmbl is a non-profit, open source web search engine whose community runs crawlers and curates results. Its hosted service is at https://mwmbl.org.

### Does Mwmbl have ads or user tracking?

The README states "No ads, no tracking, no profit." It also says the Firefox crawler extension does not use or access personal data.

### How do I contribute computing resources to Mwmbl?

The README directs contributors with spare processing capacity and bandwidth to the Docker Compose crawler guide linked under `running-the-mwmbl-crawler-with-docker-compose`. It requires `MWMBL_API_KEY` and `MWMBL_CONTACT_INFO` configuration.

### Can I run Mwmbl locally for development?

Yes, but the Makefile requires `uv`, PostgreSQL and Redis, while `pyproject.toml` requires Python 3.11. The developer book is the documented source for the complete local setup, and `make test` plus `make run` are the relevant quick-start targets after dependencies are configured.

### How is Mwmbl different from YaCy and SearXNG?

Mwmbl keeps a centralised index while distributing crawling among volunteers. YaCy distributes its index across a peer-to-peer network, while SearXNG is a meta search engine rather than Mwmbl's community-curated standalone index.

## Sources

- [Issues](https://github.com/mwmbl/mwmbl/issues)
- [License: AGPL-3.0](https://github.com/mwmbl/mwmbl/blob/main/LICENSE)
- [mwmbl/mwmbl on GitHub](https://github.com/mwmbl/mwmbl)
- [Project website](https://mwmbl.org)
- [README](https://github.com/mwmbl/mwmbl/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/mwmbl-mwmbl