Crawlee: one Node.js interface for HTTP and headless-browser crawling

The queue and storage layer is the actual product

Writing a scraper is not hard. Writing one that survives ten thousand URLs is. The failure modes are mundane: you lose the frontier when the process dies, you re-fetch pages you already have, you fan out to a hundred concurrent requests and get rate-limited, and you end up with results scattered across a folder of half-written JSON. Crawlee's README lists a persistent queue for URLs to crawl, breadth and depth first, plus pluggable storage of both tabular data and files. That is the part you are buying. The crawler classes are the visible API, but the queue and the dataset are what turn a script into something you can stop and restart. The README states that by default Crawlee stores data to ./storage in the current working directory, and that you can override this directory via Crawlee configuration. That default matters more than it looks: it means a crawl is resumable from disk without you designing a schema for it, and it means a CI job that runs a crawler will leave artifacts behind unless you point the configuration somewhere else.

One requestHandler, four engines underneath

The README's feature list puts it plainly: a single interface for HTTP and headless browser crawling, with Cheerio, JSDOM, Playwright and Puppeteer behind it. The manual-install example uses PlaywrightCrawler, and the handler receives request, page, enqueueLinks and log. The page object there is Playwright's, not an abstraction Crawlee invented, which is the design decision worth noting. Crawlee does not try to normalise page.title() into a vendor-neutral call. It gives you the real engine handle and lets you write engine-specific code inside a shared lifecycle. The upside is that you are never fighting a leaky wrapper. The downside is that switching from PlaywrightCrawler to PuppeteerCrawler is not a one-line change if your handler touches page methods that differ between the two. The README also notes that Playwright is not bundled with Crawlee, to reduce install size, so npm install crawlee playwright is two dependencies you manage separately. That is a deliberate split: the core stays small, and the browser driver is your choice and your upgrade problem.

Getting a first crawl running

The fastest path in the README is the CLI: npx crawlee create my-crawler, then cd my-crawler and npm start. The README says the CLI installs dependencies and adds boilerplate, and that you should choose the Getting started example. For an existing project, the manual route is npm install crawlee playwright, followed by a file that imports PlaywrightCrawler and Dataset from 'crawlee', constructs the crawler with a requestHandler, and calls await crawler.run(['https://crawlee.dev']). The README notes Crawlee requires Node.js 16 or higher. The handler in the example calls enqueueLinks() with no arguments to pull links off the current page and add them to the queue, and Dataset.pushData() to write a record. The README says results land as JSON in ./storage/datasets/default. Two options appear in the example: headless, commented out, which the README describes as the switch to see the browser window, and the log object passed into the handler. Configuration, request storage and result storage each have their own guide on crawlee.dev, so the defaults are documented rather than implied. If you also use the Apify SDK, the README gives a package.json overrides block mapping @crawlee/core, @crawlee/types and @crawlee/utils to $crawlee, to avoid ending up with multiple Crawlee versions installed. That override is specific to the Apify SDK path and is not needed for a standalone crawler.

Proxy rotation and fingerprinting are claims the README makes, not guarantees

The README states that Crawlee offers integrated proxy rotation and session management, zero-config generation of human-like fingerprints for browser crawling, and, on the HTTP side, automatic generation of browser-like headers, replication of browser TLS fingerprints, and zero-config HTTP2 support even for proxies. It also says crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Treat that last sentence as marketing copy rather than an engineering specification. Nothing in the supplied material quantifies a success rate against any anti-bot vendor, and defaults that work today are exactly the kind of thing that stops working after a detection vendor ships a release. The mechanism is real and worth understanding: HTTP2 support plus TLS fingerprint replication means the HTTP crawler is not sending the request signature of a default Node.js client, and header generation means it is not sending Node's default headers either. Those are the two cheapest signals a bot detector checks. Whether that is enough for a given target is something you have to measure against that target, not against the README.

The version situation is the adoption risk

The release list shows v3.18.0, then v3.18.1 a week later, then v4.0.0-rc.0. A release candidate for a major version, published while the previous major is still receiving patch releases, tells you the API surface is moving. The README documents an npm install crawlee@next path for beta builds, described as automated builds for every merged code change, and warns that installing beta builds alongside the Apify SDK requires the overrides block. Read that as a warning about dependency duplication, not just a tip. If you run crawlee@next and something else in your tree pulls a published @crawlee/core, you can end up with two copies of the queue and storage code in one process. The practical consequence is that a crawler is not a library you upgrade casually. Its behaviour depends on the queue, the storage layout and the browser driver version simultaneously, so a patch bump in @crawlee/core can change how a resumable crawl behaves on restart. Pin the version, and test the upgrade by re-running a crawl that was interrupted, not just a fresh one.

When a plain fetch call is the better answer

Crawlee's own feature list starts with a single interface for HTTP and headless browser crawling, and the HTTP side is genuinely capable: Cheerio and JSDOM parsers, HTTP2, header and TLS fingerprint generation. But the library's value is concentrated in the queue, the storage and the lifecycle hooks. If your job is to fetch one endpoint, parse the JSON and exit, Crawlee is the wrong shape. You get a storage directory, a request queue and a crawler object to solve a problem that a single fetch call and JSON.parse already solve. The same applies to scheduled jobs that hit a fixed list of ten URLs: the persistent queue is overhead when the list is a constant in your source file. The honest dividing line is statefulness. The moment you need to resume after a crash, deduplicate URLs across runs, or mix a cheap HTTP pass with a browser pass over the same site, the queue earns its place. Below that line, reach for the HTTP client you already have.

Licence and the maintenance surface

Crawlee is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant. That is more permissive than a copyleft licence would be for a library you embed in a product, and it does not impose a source-disclosure obligation on your crawler code. It is not legal advice; check the licence text and your own obligations. The maintenance cost is the dependency tree, not the licence. Crawlee deliberately does not bundle Playwright, so every browser engine you support is a separate upgrade you schedule, and browser binaries are large. The README points to Dockerfiles ready to deploy, which shifts some of that cost into image size and build time. On the HTTP side, Cheerio and JSDOM are dependencies you inherit. The README also notes that Crawlee is developed by Apify and is easy to set up on the Apify platform, while remaining open source and runnable anywhere. That is a fair description: the library does not require the platform, but the documentation, the SDK integration and the overrides advice all assume you might use it. Decide early whether you are on that path, because it changes which packages you pin.

Editorial conclusion

Adopt Crawlee if your team already writes TypeScript and you want one requestHandler shape across Cheerio, JSDOM, Playwright and Puppeteer, with the queue and dataset handled for you. Do not adopt it if you need a stateless single-request fetch, or if you are unwilling to pin a crawler version, since the v3.18.x line and the v4.0.0-rc.0 pre-release are both live. Before committing, run npx crawlee create my-crawler, check what lands in ./storage after one crawl, and confirm whether your production path needs the apify package's overrides block.

Official sources

1. apify/crawlee on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases