Crawlee for Python: A Crawler Framework That Bundles HTTP, Parsing, Storage and Proxy Rotation

What Crawlee for Python actually replaces

A hand-rolled Python crawler is usually four decisions glued together: an HTTP client, an HTML parser, a queue that tracks which URLs are done, and somewhere to write results. Crawlee for Python packages all four behind crawler classes. The README describes it as covering crawling and scraping end to end, with tools to crawl for links, scrape data and persistently store it in machine-readable formats. The library is published on PyPI as crawlee and licensed Apache-2.0. It targets Python 3.10 and above, per the badge in the README. The intended audience is engineers who want the plumbing handled but still want to control the request handler, because the framework calls your function per request rather than imposing a fixed extraction model. It is also aimed at teams that need both plain HTTP fetching and browser rendering in the same codebase, since the crawler classes share the same handler shape.

The crawler classes and what each one downloads with

Crawlee ships more than one crawler, and the choice between them is the main architectural decision. BeautifulSoupCrawler downloads pages over HTTP and hands you parsed HTML. According to the README, it uses ImpitHttpClient by default for HTTP communication and BeautifulSoup for parsing, and it requires installing crawlee with the beautifulsoup extra. The README is explicit that this crawler has very good performance since it does not use a browser, and equally explicit that it is not enough if you need client-side JavaScript to produce your content. PlaywrightCrawler covers that case: it uses a headless browser and is built on Playwright, and the README says it excels at retrieving pages that rely on client-side rendering. The repository topics also list Parsel and Selenium, so parser and driver choices exist beyond the two documented examples, but the README's worked examples only show BeautifulSoupCrawler and PlaywrightCrawler. Treat the rest of the matrix as something to confirm on the documentation site rather than something the README spells out.

How a request flows through the handler, dataset and link queue

The BeautifulSoup example in the README is the clearest view of the data flow. You construct a crawler, optionally with max_requests_per_crawl, and decorate a function with crawler.router.default_handler. That function receives a context object typed as BeautifulSoupCrawlingContext. Inside it, context.request.url gives the current URL, context.soup gives the parsed document, context.push_data writes a dictionary to the default dataset, and context.enqueue_links adds every link found on the page back into the crawl. Finally crawler.run takes the seed URL list. So the loop is: fetch, parse, your handler extracts and pushes, the framework enqueues discovered links, and the queue decides what runs next until the limit is hit. Two details matter in practice. The router is a named concept, meaning you can attach more than the default handler and dispatch by URL pattern rather than writing one large if-chain. And push_data is asynchronous, so storage writes are part of the same event loop as fetching. The README notes that a crawler run creates a storage/ directory in the current working directory, which is where datasets and related state land.

Installing it, from pip extras to the uv scaffolding command

The README gives two installation paths. The broad one is python -m pip install 'crawlee[all]', followed by playwright install for the browser binaries, then a version check with python -c 'import crawlee; print(crawlee.__version__)'. The extras system exists so you do not pull browser dependencies into a project that only needs HTTP parsing, which is why the BeautifulSoupCrawler example tells you to install the beautifulsoup extra specifically. The second path scaffolds a project. First confirm uv is present with uv --help, then run uvx 'crawlee[cli]' create my-crawler and pick a template. If crawlee is already installed, the equivalent is crawlee create my-crawler. That is the whole documented onboarding surface: one extras install, one browser binary install, one CLI generator. The CLI path is the one worth preferring for a new project, because the template decides the dependency set for you instead of leaving you to guess which extras a given crawler class needs.

Proxy rotation and the bot-detection claim in the README

The repository description lists proxy rotation as a feature, and the README claims crawlers will appear almost human-like and fly under the radar of modern bot protections even with default configuration. That second sentence is a marketing claim, not a documented mechanism. The material here does not describe how the rotation is configured, which proxy providers are supported, or what fingerprinting is applied. If evading bot detection is the reason you are evaluating Crawlee, this is the part to verify on the documentation site before you commit, because a default-configuration claim is only meaningful if you can see the defaults. The honest reading is that proxy rotation is a supported capability and the anti-detection behaviour is asserted rather than specified. Plan for the possibility that you will still need to tune headers, concurrency and session handling yourself, and budget time for that rather than assuming the defaults carry you.

Where Crawlee is the wrong tool, and what to use instead

Crawlee is the wrong tool when the crawl is trivial. If you need to fetch one page and parse one field, the framework's queue, storage directory and crawler lifecycle are overhead you will pay for and never use. A single httpx or requests call plus Parsel or BeautifulSoup is shorter and has fewer moving parts. The second case where it is the wrong tool is an existing Scrapy project. Scrapy is the closest real alternative and the difference in approach is structural. Scrapy is built around a Twisted reactor and a spider class with parse methods, and its item pipelines and middlewares are configured through settings. Crawlee is asyncio-native and built around a router of handler functions plus a context object passed into each call. If your team already reasons in Scrapy's middleware and pipeline model, porting to Crawlee means rewriting that layer, not adjusting it. Crawlee's advantage in that comparison is that browser rendering arrives through the same handler API as HTTP fetching, so a project that starts on plain HTTP and later needs Playwright does not have to change frameworks. A third case: if you want a managed extraction service rather than a library, none of this applies, since Crawlee is code you run and operate.

Maintenance surface, release cadence and licence terms

The release history shows v1.10.0 on 2026-08-31, v1.9.3 a week earlier and v1.9.2 the week before that, with the last push to master on 2026-09-10. That is a fast patch cadence, which cuts both ways. You get fixes quickly, and you also need a pinning policy, because a library releasing minor versions weekly can shift behaviour under an unpinned dependency. The Apache-2.0 licence permits commercial use and modification; it also carries a patent grant and requires that you preserve notices and state changes. That is a description of the licence text, not legal advice, and if you redistribute Crawlee inside a product you should have your own counsel read the notice and modification clauses. The other maintenance cost is environmental rather than code-level. PlaywrightCrawler depends on browser binaries installed via playwright install, so container images grow and CI pipelines need that step. The storage/ directory created in the working directory is a second operational detail: on ephemeral filesystems it disappears between runs, so anything you want to keep must be moved out by the end of the crawl.

Editorial conclusion

Adopt Crawlee for Python if you are building a multi-page crawler that needs queueing, retries, persistent datasets and both HTTP and browser fetching under one API. Do not adopt it if your job is a single fetch of one page, or if you already have a working Scrapy project and no browser requirement. Before committing, verify three things: that Python 3.10 or newer is available, that you can run playwright install in your deployment image, and where the storage/ directory will land, because the README states a crawler run creates it in the current working directory.

Official sources

1. apify/crawlee-python on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases