fastCRW: a Rust scrape, crawl and search engine with a Firecrawl-compatible API

The gap fastCRW is aimed at: one binary instead of a scrape stack

Most teams that need web content for an LLM end up assembling several things: a fetcher, an HTML-to-markdown converter, a link frontier for crawling, a search provider, and a schema extractor. fastCRW collapses those into one Rust binary that exposes scrape, crawl, map, search and extract as operations. The README frames the result as turning URLs into markdown or structured JSON through one engine.

The intended user is fairly narrow. It is someone running AI agents or retrieval pipelines who wants the fetching layer to be a process they control rather than a vendor account. The MCP server path makes that explicit: `npx -y crw-mcp@latest install` registers a CRW skill and MCP server into detected AI tools, and `crw setup` can perform the same step. The README names Claude Code, Cursor, Codex, Gemini CLI, OpenCode and Windsurf as tools that are picked up automatically when already configured, and states that nothing else is touched and the key stays in `~/.config/crw/config.toml` rather than being copied into each tool.

If you only need to fetch a handful of pages once, this is more machinery than the job requires. The value shows up when the same fetch layer has to serve a CLI, a Python or Node client, and an agent over MCP.

Five operations, one API surface, and a compatibility promise

The README lists the operations plainly. Scrape turns one URL into markdown, HTML, links, screenshots or schema JSON. Crawl follows a bounded site crawl and collects its pages. Map discovers URLs without scraping every page. Search queries the web and can optionally scrape selected results. Extract produces structured fields from one or many URLs.

The distribution model matters more than the operation list. fastCRW advertises a drop-in Firecrawl-compatible API with /scrape, /crawl and /search, and the README links a migration guide. That is a deliberate adoption strategy: if your code already speaks Firecrawl's request shapes, switching the base URL is the migration. The compatibility claim is scoped to those endpoints in the material available, so treat any endpoint outside them as unverified until you check the API reference.

There are two ways to consume it. The CLI is direct: `crw https://example.com` scrapes and works right after install, while `crw search "rust async runtime"` requires `crw setup` first. The SDKs are separate packages: `pip install crw` for Python and `npm install crw-sdk` for Node, both constructing a client and calling `scrape` with a `formats` array. The Python example passes `formats=["markdown"]` and reads `page["markdown"]`; the Node example passes `{ formats: ["markdown"] }` and reads `page.markdown`.

Install paths and the config file that holds your key

The documented install is a single shell command: `curl -fsSL https://fastcrw.com/install | sh`. The README states it runs locally and free with no account. The same command accepts a key inline, `curl -fsSL https://fastcrw.com/install | CRW_API_KEY=crw_live_... sh`, which installs the binary, connects the key, and registers the MCP server with detected AI coding tools. Setting `CRW_NO_AGENTS=1` skips that registration step, and `crw setup` can be run on its own to choose interactively.

Two details in that flow are worth noting because they are easy to miss. First, piping a remote script into `sh` executes whatever the server returns at that moment; the README does not document a checksum or signature step, so if that matters to you, prefer the crates.io package `crw-server` or build from source. Second, the key is written to `~/.config/crw/config.toml` once and referenced from there rather than duplicated into each tool's configuration, which is a sensible choice for rotation but means that file is now a credential store you have to protect.

Building from source requires Rust 1.85 or newer. The README gives the sequence: `git clone https://github.com/us/crw`, `cd crw`, then `make check-fast`. The install script covers macOS and Linux on Intel and ARM; Windows is not mentioned in the material provided.

Local and managed are not the same product

The README's own comparison table is the most useful thing in it. Managed API is positioned for zero infrastructure and managed scaling, started by creating an API key and running `crw setup`. Local or self-hosted is positioned for data control, private networks or custom infrastructure, started by installing and running `crw <URL>`. The operations row is the honest part: managed covers managed proxies, billing and hosted capabilities, while self-hosted leaves you to choose renderers, search, auth, proxies and capacity.

That sentence carries the real constraint. If your target pages need JavaScript rendering or a search backend, the README does not claim those come bundled with the local install. It points to `/v1/capabilities` and a response-shapes page, and states outright that capabilities and response shapes can differ by deployment. So a local install and a managed key are not interchangeable at the feature level even though the endpoint names match. Any evaluation that tests only one of the two tells you nothing about the other.

The practical consequence: decide which deployment you are actually shipping before you benchmark anything, and probe `/v1/capabilities` on that deployment rather than assuming parity.

The benchmark claims, and what they do not cover

fastCRW's headline numbers come from two separate evaluations, and the README is unusually specific about which is which. The first runs on Firecrawl's own public 1,000-URL dataset, where the project claims it recovered more truth than Crawl4AI and Firecrawl, matched the fastest median latency, and idled at roughly 14 MB RAM. The repository description states 2.3x faster than Tavily and 1.5x faster than Firecrawl on 1K-URL benchmarks. The second evaluation is answer accuracy rather than scrape recall: 90.0% correct on 600 AA-Omniscience questions, with a comparison against Firecrawl at 73, Exa at 70, You.com at 69, Parallel at 68 and Tavily at 64.

These are the project's own published results, with methodology and reproduction steps linked in BENCHMARKS.md and on the site. I have not run them, and the numbers should be read as claims rather than measurements until you reproduce them on your own URLs. The memory figures also disagree between sources: the description says 6 MB RAM for the single binary, while the README's benchmark section says roughly 14 MB idle. Those may measure different things (binary footprint versus resident idle), but the material does not explain the difference, so do not quote either number without checking which one applies.

The bigger caveat is coverage. Scrape recall on someone else's dataset and answer accuracy on a question set are both proxies. Neither tells you how the crawler behaves on a site with heavy client-side rendering, aggressive bot detection, or a robots.txt that forbids most of what you wanted.

Licence split: AGPL engine, MIT SDKs, separate embedding terms

The repository states that the engine and MCP server are AGPL-3.0, while the Python and TypeScript SDKs are MIT. That split is deliberate and it shapes who can use fastCRW comfortably. Calling the AGPL engine over a network from your own service is the scenario the licence is designed to reach: if you modify the engine and let users interact with it remotely, the AGPL's network clause is the part your legal reviewer will want to read.

For internal tooling, the obligation is usually manageable. For a product where the scraper is a component you modify and expose to customers, it is a different conversation. The SDKs being MIT means client-side integration code carries no copyleft, but the client is not the part that does the scraping.

There is a third category in the README: embedding has its own licence, with a contact address rather than a named licence. That is a commercial-licensing path, not an open one, and the material does not describe its terms. I am not giving legal advice here; the point is that fastCRW is not uniformly permissive, and the boundary between the MIT SDKs and the AGPL engine is exactly where most adopters will need a decision.

Where it is the wrong tool, and what to use instead

The clearest failure mode is licence mismatch. If your product cannot carry AGPL obligations on the scraping component, fastCRW's engine is disqualified before any benchmark matters, and the MIT SDKs do not change that because they are clients. The embedding licence exists as an escape hatch, but it is a commercial conversation with unknown terms in the material provided.

The second case is capability mismatch between deployments. If your pipeline depends on managed JS rendering or managed search and you intend to self-host, the README does not promise those locally. You would be choosing a deployment that lacks the features you tested against.

The third case is scope. If you need continuous, scheduled crawling of a large site with change detection, or a search index you own, fastCRW is a fetch layer, not a data platform.

For a concrete alternative, consider Crawl4AI. The README benchmarks fastCRW against it on Firecrawl's 1,000-URL dataset, and the difference in approach is architectural rather than numeric: Crawl4AI is a Python library you import and drive inside your own process, so you configure crawling in Python and inherit Python's deployment and dependency story. fastCRW is a compiled binary with an HTTP surface and a CLI, so you deploy it as a service and call it over the network or from a shell. If your pipeline is already Python and you want the crawler in-process with your own async code, the library model is the better fit. If you want a small process that several languages and an MCP client can share, the binary model is the better fit. The trade is deployment shape, not raw speed.

Maintenance cost and what to verify before you commit

The release cadence visible in the material is fast: v0.34.0 on 2026-09-08, v0.33.0 on 2026-09-02, v0.32.0 on 2026-08-24. Three minor releases in roughly two weeks, all still on a 0.x version number. That is a project moving quickly, and it means two things for an adopter. Expect to upgrade more often than you would with a 1.x dependency, and expect response shapes to move, which is why the README routes you to a dedicated response-shapes page rather than promising stability.

Self-hosting shifts the operational work to you. The README's own table says you choose renderers, search, auth, proxies and capacity. Each of those is a component you now own, and each has its own failure mode. The upside is the same list: none of those choices are made for you.

Before adopting, verify three things against your own target sites rather than the published datasets. Run `/v1/capabilities` on the deployment you actually intend to use, local or managed. Reproduce the benchmark on a sample of your own URLs, since scrape recall on Firecrawl's dataset says nothing about a page behind a login wall. And confirm with whoever owns licensing that the AGPL-3.0 engine fits your distribution model, because that decision is independent of every performance number in the README.

Editorial conclusion

Adopt fastCRW if you want a single self-hosted binary that answers /scrape, /crawl and /search in Firecrawl's shape, or if you want an MCP server registered into Claude Code, Cursor, Codex, Gemini CLI, OpenCode or Windsurf without writing that glue yourself. Do not adopt it if you need a permissive licence on the engine, since the engine and MCP server are AGPL-3.0 while only the Python and TypeScript SDKs are MIT, or if you need a capability the local deployment does not implement. Before committing, run /v1/capabilities against your specific deployment, because the README states that capabilities and response shapes can differ between the managed API and a self-hosted install.

Official sources

1. License: AGPL-3.0
2. Project website
3. README
4. Releases
5. us/crw on GitHub