webclaw: local-first web extraction for LLM pipelines, shipped as a Rust CLI, MCP server and REST API
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
At a glance
- What is it?
- webclaw converts a URL into markdown, plain text, JSON or an "llm" format, and exposes the same engine through a CLI, an MCP server and a self-hostable HTTP server. The catch is that the open-source core is AGPL-3.0 while a hosted API is sold alongside it, and the README leaves the local-versus-hosted boundary blurry.
- Who is it for?
- Adopt webclaw if you want page-to-markdown conversion running on your own machine or cluster and you are comfortable with AGPL-3.0: the CLI, MCP server and self-hostable server are all in this repository. Do not adopt it if you need a permissively licensed library to embed in a closed-source product, or if your targets are JavaScript-heavy or bot-protected and you cannot set WEBCLAW_API_KEY.
- Can I use it commercially?
- Yes, with strict conditions. AGPL-3.0 is a network copyleft licence: if people use a modified version over a network, for example as a hosted service, you must offer them its source code under the same licence.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The two failure modes webclaw is aimed at
The README frames the problem in one line: most scraping tools hand an agent either a blocked page, a login wall or an empty app shell, or raw HTML stuffed with navigation, scripts, styling and ads. Both outputs are useless to a model without further work. The first has no content at all; the second buries the content under boilerplate that consumes context window and dilutes retrieval.
webclaw's answer is a single pipeline that takes a URL and returns clean content. The README's own example is a scrape of example.com into markdown, and the CLI surface is `webclaw https://example.com --format markdown`. The audience is narrow and specific: people building RAG indexes, agent tools and data extraction jobs in or around Rust, who would rather run extraction on their own hardware than send every URL to a third-party API. The topics list confirms the positioning, naming firecrawl, crawl4ai, jina, scraperapi and scrapingbee as the tools it is measured against, plus self-hosted and ai-agents.
What actually happens between URL and markdown
The repository describes four surfaces over one engine: a CLI, an MCP server, a REST API and SDKs, with the extraction engine and a self-hostable server in the same repo. The tool table is the clearest statement of the internal split. `scrape`, `crawl`, `map`, `batch` and `diff` are marked as local. `extract` and `summarize` are marked "Yes, with local or configured LLM", which means those two are not self-contained: they need a model, and the README does not say which local runtimes are supported or what the configuration key is.
The crawl mechanism is stated plainly: `webclaw https://docs.rust-lang.org --crawl --depth 2 --max-pages 50` follows same-origin links and extracts discovered pages. Same-origin is the constraint worth noting. A documentation site split across docs.example.com and api.example.com will not be traversed as one crawl. `map` is the cheaper sibling: it discovers URLs without extracting every page, which is the right call when you want to decide what to fetch rather than pay for extraction on everything.
The topics list includes tls-fingerprinting, which suggests the fetch layer does more than a default HTTP client, but the README does not document how it is configured or which sites it changes the outcome for. Treat that as an unverified claim from the repository metadata.
Install paths and the commands that matter
There are five documented install routes. For agents, `npx create-webclaw` detects supported MCP clients (Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI) and writes their configs. For a package manager, `brew tap 0xMassi/webclaw` then `brew install webclaw`. Prebuilt macOS, Linux and Windows binaries sit on GitHub Releases. Docker is `docker run --rm ghcr.io/0xmassi/webclaw https://example.com`. From source, `cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli` and the same for `webclaw-mcp`.
The README is honest that source builds can fail on missing native tooling and lists per-distro prerequisites: `pkg-config libssl-dev cmake clang git build-essential` on Debian and Ubuntu, the equivalent set on Fedora and Arch, and `xcode-select --install` on macOS. That is a real cost. A Rust project that pulls in OpenSSL and CMake is not a two-minute install on a slim container image.
Beyond scraping, the flags worth knowing are `--only-main-content`, `--include "article, main, .content"`, `--exclude "nav, footer, .sidebar, .ad"`, `--brand` for logos, colors and fonts, and `--diff-with pricing-old.json` for comparing a page against a saved snapshot. The MCP config is a standard stdio entry pointing at `npx -y @webclaw/mcp`, and there is a separate skill package installed with `npx skills add 0xMassi/webclaw-skill`.
The local and hosted split is the part to read carefully
The README says most sites extract locally with no API key, and that setting `WEBCLAW_API_KEY` handles bot-protected and JavaScript-rendered pages. That sentence is doing a lot of work. It means the open-source engine is the local path, and the hosted service at webclaw.io is the fallback for pages the local path cannot get. The repo description calls webclaw "local-first", and this is what local-first means here: local by default, remote when local fails, with the failure mode being exactly the blocked page and empty app shell the README opened with.
That is a reasonable design, but it puts the hardest cases outside the open-source boundary. If your workload is mostly JavaScript-rendered or bot-protected, you are not really running webclaw; you are running a client for webclaw.io. The README does not state what the hosted path costs, what rate limits apply, or what data leaves your machine when `WEBCLAW_API_KEY` is set. Anyone evaluating this for content that cannot leave their network needs to answer that before writing code.
AGPL-3.0 is the constraint that decides the adoption question
The licence is AGPL-3.0. For a CLI you run on your laptop or a crawler you run inside your own infrastructure for internal indexing, that is usually workable. For embedding the extraction engine in a product you distribute, or exposing a modified version as a network service, AGPL-3.0 imposes source-availability obligations that permissive licences do not. This is not legal advice, and the exact scope depends on how you link and deploy, but the practical effect is that webclaw is a poor fit for a closed-source commercial library dependency.
The commercial shape reinforces this reading: the same organisation sells a hosted API, which is the conventional way an AGPL project funds itself. If your use case is "I want to call an extraction API and not think about licences", the hosted service is the intended answer, not the repository. If your use case is "I want to run extraction inside my own perimeter", the repository is the answer and the licence is the price.
Where the documentation stops
The README is long on command examples and short on the things that determine whether extraction actually works on your targets. There is no documented accuracy measure, no statement of which HTML parsing or readability approach is used, no description of how the "llm" output format differs from markdown, and no explanation of what `extract` returns for a given page or what schema it expects. The `extract` and `summarize` tools are marked as needing a local or configured LLM, but the README never names the configuration key or the supported providers.
The example output is a scrape of example.com, which is a static page with almost no structure. It demonstrates the interface, not the extraction quality. The workflow examples directory is referenced (html-to-markdown-rag, firecrawl-compatible-api, mcp-web-scraping, proxy-backed-crawling, cloudflare-diagnostics) but the README does not summarise what any of them show. The cloudflare-diagnostics and proxy-backed-crawling examples are the interesting ones, since they imply the project has opinions about anti-bot handling, and the README does not say what those opinions are.
Compared with crawl4ai and Firecrawl
The topics list names crawl4ai and Firecrawl as alternatives, and the difference in approach is visible from the material. crawl4ai is a Python project; webclaw is Rust, and the README's install instructions are Cargo, Homebrew, prebuilt binaries and Docker. If your stack is Python, crawl4ai drops into the same process as your RAG code; webclaw means either shelling out to the CLI, running the server and calling it over HTTP, or going through MCP. That is a real integration cost, and the README offers no Python SDK in the material supplied.
Firecrawl is primarily a hosted API with a self-hostable component, and webclaw's README includes a "Firecrawl-compatible API" example, which suggests it can stand in for that shape. The difference is where the default runs. webclaw's default is local extraction with the hosted service as fallback for hard pages, and the tools table marks five of eight tools as fully local. The licence difference is the sharper one: AGPL-3.0 here versus whatever the alternative offers, and that alone will rule webclaw in or out for some teams before any technical comparison happens.
Maintenance, releases and what to verify first
The release cadence shown is three versions in the weeks around August 2026 (v0.6.20, v0.6.21, v0.6.22), with the last push to main in September 2026. Frequent point releases at the 0.6.x level suggest active development and no stability commitment yet. Nothing in the material describes a plugin or extension interface, so extending the extraction engine means forking it, and under AGPL-3.0 that fork is yours to maintain against upstream changes.
Upgrade cost is mostly the usual Rust binary problem: prebuilt binaries and Homebrew make upgrades cheap, source builds mean recompiling with the OpenSSL and CMake toolchain present. If you deploy the self-hosted server, you own its operations, and the README says nothing about its configuration, persistence or resource profile. Before adopting, verify three things on your own hardware: that `npx create-webclaw` writes an MCP config you are happy with, that extraction quality holds on your actual target pages rather than example.com, and that the pages you care about resolve locally without `WEBCLAW_API_KEY`. That third check decides whether you are adopting an open-source tool or a client for a paid API.
Editorial conclusion
Adopt webclaw if you want page-to-markdown conversion running on your own machine or cluster and you are comfortable with AGPL-3.0: the CLI, MCP server and self-hostable server are all in this repository. Do not adopt it if you need a permissively licensed library to embed in a closed-source product, or if your targets are JavaScript-heavy or bot-protected and you cannot set WEBCLAW_API_KEY. Before committing, run `npx create-webclaw` on one machine and read the config it writes, then check the extraction quality on three of your own URLs with `--only-main-content` and with `--include`/`--exclude` selectors, because the README's example output is a single static page and says nothing about how the engine behaves on an app shell.
Community notes