spider-rs/spider: A Rust Crawler That Streams Pages and Defers Headless Chrome

The problem Spider targets: page streams for pipelines, not one-off scrapes

Most scraping code is written for a single page and then bent into a crawler. Spider starts from the other end. Its README describes a concurrency-first engine that streams pages as they arrive, renders JavaScript only on pages that need it, and scales from a single script to a distributed fleet without changing your code. The stated use cases are feeding the open web into vector stores for LLM and RAG pipelines, monitoring sites for SEO and price changes, exporting pages as Markdown, JSON, or WARC, and driving headless Chrome for AI browsing agents. That list is a fair summary of who this is for: engineers building ingestion jobs in Rust, not analysts who want a spreadsheet of prices. The repository ships a library, a CLI (spider_cli), a Node package, a Python package, and an MCP server, so the same engine is reachable from several runtimes, but the Rust crate is the centre of gravity.

HTTP first, Chrome on demand: the mechanism the README actually specifies

The design choice that matters is the two-path crawl. Spider runs HTTP-first and launches headless Chrome only when a page needs JavaScript, according to the README. To get that behaviour you enable features = ["chrome"] and call crawl_smart() rather than crawl(). Both paths stream, so pages come back as they are fetched instead of batching at the end. The streaming surface is a subscription channel: Website::new returns a value you call subscribe(16) on, which yields a receiver; you spawn a task that loops on rx.recv().await and prints page.status_code and page.get_url(), then call crawl().await and unsubscribe(). The 16 is the channel buffer size, and it is the only number in the local example that is not a default. Everything else about the crawl is expressed through builder methods on Website. The README states that proxies, retries, rate limiting, and stealth are built in, and that the same API drives one async task or a distributed worker fleet without the concurrency model changing. That last claim is the architectural bet: the crawl loop is identical whether it runs in your process or behind the hosted service, which is why the README can promise moving to managed infrastructure with one config change.

Getting it running: cargo add, builder methods, and the two test commands

Installation is one line per target: cargo add spider for the library, cargo install spider_cli for the command line, npm i @spider-rs/spider-rs for Node, pip install spider_rs for Python, and cargo install spider_mcp for the MCP server. The README's local example uses use spider::{tokio, website::Website}; note that tokio is re-exported, so you do not add it separately in that snippet. Configuration is a builder chain on Website::new. The README shows with_limit(50) for concurrent requests, with_depth(10) for how deep to follow links, with_delay(500) for a pause between requests in milliseconds, with_respect_robots_txt(true), with_subdomains(true), with_user_agent(Some("MyBot/1.0")), with_stealth(true), and a final build().unwrap(). The README claims every option has a sensible default and you should set only what you need, though it does not enumerate the defaults, so treat build() as the place where unset behaviour is decided and read the Configuration docs before assuming a safe crawl. For the hosted path, the dependency becomes spider = { version = "2", features = ["spider_cloud"] }, and you construct SpiderCloudConfig::new("sk-...").with_mode(SpiderCloudMode::Smart), then attach it with with_spider_cloud_config(cloud). Smart mode routes through proxies first and escalates to the unblocker only on pages that fight back, so you pay for bypass only where it is needed. On the contribution side, the README documents cargo test -p spider for unit tests and RUN_LIVE_TESTS=1 cargo test for live network tests, which tells you the default suite does not hit the network.

Where the local engine stops and Spider Cloud starts

The README is unusually candid about the boundary. It says the hardest part of crawling at scale is not the code but the proxies, headless browsers, and constant anti-bot churn, and that Spider Cloud runs all of that behind the same API. Read that as a product statement and as a limitation at once. The library gives you a crawler; it does not give you a proxy pool, and stealth mode is a request-shaping option, not a guarantee against a determined anti-bot system. The cloud path has its own cost model that the README only gestures at: Smart mode escalates to the unblocker on pages that fight back, so billing is tied to how often your targets resist. That is a reasonable design, but it means crawl cost is a function of the sites you choose, not of your own traffic volume, and you cannot estimate it from the repository alone. There is also a coupling to note: the same engine runs Spider Cloud, so the hosted service and the open-source crate move together. That is good for parity and less good if you want to pin behaviour independently of a vendor's release cadence. The MIT licence covers the crate; the hosted service is a separate commercial relationship, and nothing in the README describes its terms.

The honest failure mode: JavaScript-heavy sites and the cost of the fallback

The HTTP-first strategy is only as good as the fraction of pages that render without JavaScript. Enable the chrome feature and call crawl_smart() and the engine will fall back, but the README does not say how it decides a page needs Chrome, how often the fallback fires in practice, or what the fallback costs in memory and startup time per browser instance. For a site that is entirely client-rendered, you are paying for a browser launch on nearly every page and the HTTP path is dead weight. That is the case where Spider is the wrong tool relative to a browser-first crawler, and the README offers no benchmark to tell you which side of the line your target sits on. A second limitation is politeness. with_respect_robots_txt(true) is opt-in in the example, not stated as a default, so a crawl built from the README's local snippet is not necessarily robots-aware. with_delay(500) is likewise something you set. A third is language surface: the Rust API is the documented one, and the Node and Python packages are listed as install targets with no API detail in the README, so if you are not writing Rust you are working from docs the README does not show.

Alternatives: browser-first crawlers and hosted scraping APIs

The nearest comparison in kind is a browser-first crawler such as Crawlee, where a headless browser is the primary fetch path and HTTP requests are the optimisation. Spider inverts that: HTTP is primary and Chrome is the exception. The difference shows up in your infrastructure. A browser-first crawler needs a browser available for every worker from the first request, so memory per worker is high and predictable. Spider's default worker is cheap and gets expensive only on pages that require rendering, which suits mixed sites where most pages are static HTML and a minority are app shells. If your targets are uniformly JavaScript-heavy, the inversion works against you. The other alternative is a hosted extraction API, and here Spider's own README makes the case: if you do not want to run proxies and browsers, Spider Cloud is the same engine with that layer managed, and the migration is a feature flag plus a config object. The real choice is therefore three-way, not two-way: browser-first open source, Spider local, or Spider Cloud. The README supports the first and third positions well and leaves the middle one to you to validate on your own targets.

Maintenance, release cadence, and what MIT does and does not cover

The repository is active, not archived, with a last push in September 2026 and releases through v2.52.2 in March 2026. The recent release titles are informative about where effort is going: v2.52.2 adds CLI authenticate for Spider Cloud, and v2.48.2 is labelled Parallel Crawl Backends. Cloud integration and crawl throughput are the visible themes, which is consistent with the README's framing. For upgrade cost, the crate is at major version 2 and the README's dependency line is spider = "2", so semver-compatible updates are the expected path; there is no migration guide in the material provided, so a major bump is the event to watch for. The licence is MIT, which is permissive and places few obligations on how you use or redistribute the crate. That is the whole of what the supplied material supports on licensing: it does not address the hosted service's terms, data handling, or what happens to a crawl's output. Treat the MIT grant as covering the code in the repository and nothing about spider.cloud, and take your own advice on the legal side rather than mine.

Editorial conclusion

Adopt spider if you are writing a Rust service that needs to turn a site into a stream of pages and you want the HTTP path to stay cheap, with headless Chrome reserved for the pages that break. Do not adopt it if you want a no-code scraper, or if you cannot operate proxies and browser binaries yourself: the README is explicit that proxy rotation, headless browsers, and anti-bot churn are the hard part, and those are what Spider Cloud sells. Verify two things before committing: that your target pages actually render under the plain HTTP path, since the README gives no field data on how often the Chrome fallback fires, and that your crawl respects the target site's terms, because the library will happily ignore robots.txt unless you pass with_respect_robots_txt(true).

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. spider-rs/spider on GitHub