WaterCrawl: A Self-Hosted Crawl and Search Stack for LLM Pipelines

The gap WaterCrawl fills between a URL and a prompt

Fetching a page and getting clean text for a model are two different jobs. The README describes WaterCrawl as a web application that crawls pages and extracts relevant data, built on Python, Django, Scrapy and Celery. That combination is the point: Scrapy handles the fetching and parsing loop, Django exposes the control surface, and Celery moves long crawls off the request thread. The project is aimed at teams building retrieval or training pipelines who want the crawl to happen on their own hardware. The README's own framing is "Self-hosted & Open Source" with "Full control over your data". If your data cannot leave your network, that sentence is the whole argument. If it can, the calculus changes, because you are now maintaining a multi-service deployment to do something a hosted API also does. The repository's topics include html2markdown and llm-crawler, which tells you the intended output format is text a model can consume rather than raw HTML. The audience is therefore engineers wiring a knowledge base, not analysts clicking through a scraping UI.

Django, Scrapy and Celery: how the pieces divide the work

The README names the stack but not the call graph, so treat this as the shape the repository implies rather than a documented diagram. Django serves the application and the REST API described as having OpenAPI documentation. Scrapy does the crawling. Celery carries asynchronous jobs, which is why the feature list can promise "Asynchronous Processing" with progress reported over Server-Sent Events. SSE matters here: a crawl of a large site runs for minutes, and a streaming endpoint lets the client watch progress instead of polling a job table. The search feature is separate from the crawl, and the README lists search depths of basic, advanced and ultimate, plus country and language targeting. That suggests search is not a thin wrapper over one index but a configurable step with cost and latency implications that scale with depth. Results handling is also its own concern: the README advertises downloading and processing search results with customizable parameters. On the storage side, MinIO appears in the deployment instructions, so object storage is part of the runtime, not an optional add-on. That is the architectural fact most likely to surprise someone expecting a stateless container.

Getting it running with Docker Compose

The README's quick start is four commands. Clone the repository, then:

cd docker cp .env.example .env docker compose up -d

The application is then reachable at http://localhost. The README attaches an important warning to that default: if you deploy on any domain or IP other than localhost, you must update the MinIO configuration in .env. The three keys it names are MINIO_EXTERNAL_ENDPOINT, MINIO_BROWSER_REDIRECT_URL and MINIO_SERVER_URL, with the endpoint changed from localhost to your actual domain and the two URLs rewritten to match. The README states the consequence plainly: failing to update these settings results in broken file uploads and downloads. That is a deployment-time failure with a confusing symptom, since the app will start and the crawl may run while artifact retrieval quietly fails. The README also says to update .env with appropriate values before production and to set up the database and MinIO, pointing at DEPLOYMENT.md for detail. A separate CONTRIBUTING.md covers local development, which is a different path from the Docker one and is not reproduced in the README.

SDKs and integrations decide how much API code you write

WaterCrawl ships client libraries rather than expecting you to hand-roll HTTP calls. The README marks Python, Node.js, Go and PHP clients as available, with a Rust client listed as coming soon. Each is documented on docs.watercrawl.dev under a /clients path. For a TypeScript shop, the Node.js client is the relevant one; the repository's primary language is TypeScript, though the README describes the server as Python, Django, Scrapy and Celery, so the two facts describe different parts of the project. On the automation side, the README lists a Dify plugin published in the Dify marketplace, an N8N node published as @watercrawl/n8n-nodes-watercrawl on npm, and Dify Knowledge Base integration. Langflow is described as a pull request that is not merged yet, and Flowise as coming soon. Those two entries are worth reading carefully: one is a pending contribution, the other an intention. Neither is something to build a pipeline on today. If your orchestration layer is n8n or Dify, the integration already exists; if it is Flowise or Langflow, you are writing the glue yourself against the REST API.

Where WaterCrawl is the wrong tool

The clearest limitation is the deployment surface. This is not a library you import. Running it means running Django, Celery workers, a database, MinIO and whatever the Docker Compose file wires together, then keeping that alive. For a team that needs to fetch a few hundred pages once, that is a poor trade against a script that calls a hosted extraction API. The MinIO configuration requirement reinforces the point: the README's warning about MINIO_EXTERNAL_ENDPOINT exists because the stack generates URLs that a browser or client must resolve, which means the deployment needs a stable, reachable hostname. Running it behind a rotating preview URL or on a laptop that changes networks will produce the broken upload and download behaviour the README warns about. The search feature carries its own uncertainty. The README lists three depth levels but does not say what backs them or what they cost, so anyone comparing it to a dedicated search API is comparing against an undocumented implementation. Finally, the repository shows no retrieved releases, so there is no published changelog to read for upgrade behaviour. Treat version pinning as your own responsibility.

How it differs from a single-purpose extraction library

Crawl4AI is the natural comparison, and the difference is architectural rather than feature-by-feature. Crawl4AI is a Python library: you install it, call it in your process, and get markdown back. WaterCrawl is a service: you deploy it, then talk to it over a REST API with OpenAPI documentation, optionally through one of the four client SDKs. That choice buys you things a library cannot give you easily. A crawl can be dispatched, run asynchronously on Celery workers, and watched over SSE while your own process does something else. Multiple applications can share one crawler instead of each embedding its own. The cost is everything that comes with a service: a database, object storage, a deployment target, an upgrade path. There is also a search capability in WaterCrawl that a pure extraction library does not attempt. If your requirement is "turn this one URL into markdown inside my script", the library wins on simplicity. If your requirement is "several teams and agents need crawled and searched content from one controlled endpoint that never leaves our network", the service shape is the reason to pick WaterCrawl, and the operational work is the price.

Licence, maintenance and what to check before you commit

The repository's licence is reported as NOASSERTION, and the README describes it as the WaterCrawl License, "essentially MIT with a few additional restrictions". That phrasing is the README's, not a legal reading, and the restrictions themselves are in the LICENSE file rather than the README. If you plan to embed WaterCrawl in a product or resell access to it, read that file before you build anything on top. This is not a formality: the difference between MIT and "MIT plus restrictions" is exactly the kind of difference that surfaces late. On maintenance, the material gives you little to work with. There are no retrieved releases, so no version history or migration notes. The README points to DEPLOYMENT.md and CONTRIBUTING.md for the operational and development details, which means your upgrade cost is whatever those documents say at the time you deploy. The stack's breadth is the real maintenance line item: a Django application, Celery workers, a database and MinIO all move independently, and a Compose file that pins them today will need attention as base images age. Budget for that before you treat WaterCrawl as fire-and-forget infrastructure.

Editorial conclusion

Adopt WaterCrawl if you want crawl and search results to stay inside your own infrastructure and you are willing to run Django, Celery, Postgres and MinIO as a service. Do not adopt it if you want a single pip install that returns markdown from a URL in-process, or if you cannot give the stack a real domain. Before committing, verify three things: that the LICENSE file's additional restrictions are acceptable to your legal team, that you can set MINIO_EXTERNAL_ENDPOINT correctly for your host, and that the SSE progress endpoint fits how your job runner reports status.

Official sources

1. Issues
2. Project website
3. README
4. watercrawl/WaterCrawl on GitHub