crawl4ai vs reader: a local Python crawler against a URL prefix service

A library in your process, and a service you call

crawl4ai is a Python package. The description says it turns web pages into clean Markdown for retrieval systems, agents and data pipelines, with browser control and structured extraction, and the mechanism is a real browser: the quick start installs Chromium through Playwright and the code opens an AsyncWebCrawler and awaits a crawl whose result carries a .markdown attribute. Everything is in your process. Your data stays in your process, the crawl is a Python object you can pass around, and a failure is an exception you can catch rather than a response code from somebody else's endpoint.

reader is a service with an open-source branch. The description is a prefix: convert any URL to an LLM-friendly input with https://r.jina.ai/. The README is equally plain that this repository is the open source branch of the codebase behind r.jina.ai and s.jina.ai, and that it runs in stateless or bucket-cached mode with the MongoDB-backed SaaS storage layer not included. So there are two products with one name. One is an endpoint you call, and its README says you are welcome to use it in production, that it is free and stable and scalable, and that it is maintained as one of Jina AI's core products, with rate limits linked from the page. The other is a TypeScript codebase you deploy yourself.

That difference propagates into everything else. A library decides your dependencies, your process model and your error handling. A service decides its availability, its rate limits and its version of the page it returns, and its self-hosted branch decides a different set again.

Getting each one running

crawl4ai has a documented setup path. pip install -U crawl4ai, then crawl4ai-setup, then crawl4ai-doctor to verify the installation, and if a browser problem appears, python -m playwright install --with-deps chromium installs it manually. There is a CLI as well: crwl https://www.nbcnews.com/business -o markdown for a basic crawl, crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10 for a breadth-first deep crawl, and crwl with -q for a question, as in -q "Extract all product prices". The project also ships a Docker API server, so the same crawl can be exposed over HTTP instead of imported. Our analysis puts the browser first in its checklist: verify that crawl4ai-doctor reports a working browser on your target machine, because Playwright is a core dependency and a headless Chromium is not always available in a container or a locked-down build image.

reader has two very different starting points. The hosted one has none: prepend the prefix, or open the live demo URL, and the README points at a Colab notebook for fetching a whole site and an interactive code snippet builder for exploring parameter combinations. Self-hosting is a deployment: it requires Node 22.15+, Chrome or Chromium, LibreOffice, and optionally MinIO for caching, and the README's local development path uses docker compose. The 2026-04 update records that the open source branch was re-synchronized with the SaaS code, that the MongoDB-backed storage layer is stripped, and that the branch runs in stateless mode out of the box with optional MinIO or S3-compatible bucket caching.

So the honest comparison is: one command plus a browser download, against either a URL you paste into a fetch call or a service stack with four prerequisites.

What each one can read and return

crawl4ai returns Markdown and describes it as LLM ready, with headings, tables, code and citation hints. Its feature list also names sessions, proxies, cookies, user scripts and hooks as controls, and adaptive behaviour that learns site patterns and explores only what matters. The release notes for v0.8.0 describe deep crawl crash recovery with resume_state and on_state_change callbacks for long-running crawls, and a prefetch mode that the project reports as five to ten times faster URL discovery. The CLI's -q option points the crawl at a question and uses LLM extraction to answer it, which is a different output from a page transcript.

reader reads four kinds of thing. Web pages are rendered with headless Chrome or fetched in a lightweight way with curl-impersonate, and the README says it picks intelligently between the two. PDFs are parsed with PDF.js for any URL, with a side-by-side example linked. MS Office documents, Word, Excel and PowerPoint, are converted through LibreOffice and then processed as HTML or PDF. Images are captioned by a vision-language model, so a text-only model downstream gets a description rather than nothing. The 2025-12 update records that PDFs and Office documents can be POSTed directly through a file body field, with no need to host them first.

The search side is the part reader has and crawl4ai's CLI does not mirror. s.jina.ai takes a query, searches the web, fetches the top five results, visits each URL and applies r.jina.ai to it, which the README contrasts with search function calling in agent and RAG frameworks that return only a title, a URL and a description. In-site search is a query parameter, as in ?site=jina.ai&site=github.com. A crawl4ai user who needs that composes it from a search API and a crawl, and owns both halves.

Operating a fetch at scale

Running a browser is work, and both projects are honest about it in different ways. crawl4ai's feature list names an async browser pool and caching as the reason it is fast in practice, and its Docker server exists so a fleet of workers can share a crawl API. The operational surface is visible in its release history: v0.8.7 was a security-hardening release that fixed critical Docker API vulnerabilities, naming remote code execution, SSRF, authentication bypass, file write, XSS and a hardcoded JWT secret, and added DomainMapper; v0.9.0 was a secure-by-default release in which auth is on by default, the server binds loopback unless it is given a token, and the request body became an untrusted trust boundary. Those are the notes of a component that is finding its edges, and our analysis says the server has only recently been hardened, so a deployment should read them and confirm it accepts the default authentication behaviour described in the v0.9.0 notes.

reader's self-hosted mode is stateless by design, with bucket caching as an option, and a stateless reader is easy to scale horizontally and easy to lose per-session state in. The hosted endpoint is a different operational proposition: the README invites production use and describes the service as actively maintained, with rate limits published, and our analysis calls it the lowest-friction starting point, requiring no setup at all. What it does not give you is a vendor SLA on your side. Our analysis tells you not to adopt crawl4ai if you need one, and by symmetry the hosted reader gives you a service relationship rather than a contract you control.

A useful way to frame it: crawl4ai's scaling problems are memory, browsers and the security surface of a server you expose. reader's self-hosted scaling problems are Node, Chromium, LibreOffice and the absence of a storage layer, while the hosted version trades all of that for someone else's uptime and a rate limit.

Where crawl4ai falls short

The browser is the first constraint. Playwright is a core dependency, so an environment without Chromium, or without the system libraries Chromium needs, is an environment where this project does not run, and that excludes some minimal images and some locked-down build agents. Our analysis says plainly not to adopt it if you cannot run Chromium in your environment.

Second, the Docker server has needed real work. The v0.8.7 release fixed critical vulnerabilities in the Docker API, including remote code execution, server-side request forgery, authentication bypass, file write, cross-site scripting and a hardcoded JWT secret, and v0.9.0 then changed the defaults so auth is on and the server binds loopback without a token. Our analysis describes the server as secure by default and only recently hardened, and the sensible reading is that the secure posture is young. An operator inheriting an older deployment should assume it needs review before it is exposed.

Third, Markdown filtering is a lossy stage. Our analysis tells you to verify that the pages you care about survive the Markdown filtering stage rather than the raw HTML stage, which is a way of saying that the cleaned output is the product and the raw HTML is not a fallback you can rely on. Code blocks, tables and citation hints are the parts most worth checking on your own pages.

Fourth, the hosted option from this project is not available. The README announces a Crawl4AI Cloud API in closed beta, launching soon, with an application form, so a team that wanted a managed crawl from this vendor is waiting. And there is no vendor SLA on the open-source path, which our analysis lists as a reason not to adopt it.

Where reader falls short

Self-hosting it is not one process. It needs Node 22.15+, Chrome or Chromium, LibreOffice, and optionally MinIO, and LibreOffice in particular is a large system dependency for anyone whose containers are built to stay small. The hosted endpoint removes that cost and replaces it with a network call.

Second, the open source branch is not the whole product. The README states that the MongoDB-backed SaaS storage layer is not included, and our analysis draws the consequence: session-based or account-based features are not present in self-hosted deployments. Anyone who reads the hosted behaviour and expects the same in a container will find a difference. The 2026-04 update explains the shape of that difference, re-synchronising the branch with the SaaS code with the storage layer stripped and running stateless out of the box.

Third, the language and the boundary. The repository is TypeScript, so a Python retrieval pipeline either shells out, calls over HTTP, or lives in another service. The free hosted API is the pragmatic answer and the README says to feel free to use it in production, with a rate limit to respect, but a rate limit is a shared resource and a dependency on someone else's schedule.

Fourth, maintenance is quieter on this side. The repository has no GitHub releases, its update history is recorded as dated entries such as 2026-04, 2025-12, 2025-03, 2024-05 and 2024-04 rather than versioned artefacts, and our analysis records the last push to this repository as 2026-05-22. It is not archived, and the entries show substantial work, but a team that pins versions is pinning to a commit rather than to a release.

Licence, maintenance and the choice for concrete situations

The licences match. Both are Apache-2.0, which permits commercial use, modification and redistribution with attribution and patent terms, and neither choice is settled by terms. If a policy screens for copyleft, both pass it.

Maintenance is the axis that separates them. crawl4ai's last recorded push is 2026-09-25 and it has a published release line, v0.9.0 on 2026-06-18, v0.9.1 on 2026-07-08 and v0.9.2 on 2026-07-15, with the v0.9.2 release described as a maintenance patch fixing a task and page leak in MemoryAdaptiveDispatcher when a streaming crawl is closed, along with Docker Playground and Monitor WebSocket authentication, Playwright headless-shell packaging and GPU Docker builds. reader's last recorded push is 2026-05-22 with no GitHub releases at all. Both are Apache-2.0 and neither is archived, so neither is a risk in the sense of abandonment; the difference is whether you can cite a version.

The situations below are clear.

A prototype that needs to read a handful of pages today. reader, because there is nothing to install and the answer arrives as a response body.

A RAG index over ordinary web pages, PDFs and Office files, where you want the fetch handled. reader, because PDF.js, LibreOffice conversion and image captioning are already in the path.

Search over the web as part of an agent, with the top results fetched for you. reader, because s.jina.ai fetches the top five results and applies r.jina.ai to each one.

A Python pipeline that must control the crawl, set cookies, run a user script, or resume a long deep crawl. crawl4ai, because sessions, proxies, user scripts, hooks, resume_state and on_state_change are its documented surface.

Pages only reachable from inside your network. crawl4ai, because the browser runs where your process runs.

A crawl exposed to other services. crawl4ai's Docker server, after reading the v0.9.0 defaults and confirming your deployment accepts them, as our analysis instructs.

A requirement for a vendor SLA on the fetch. Neither on its own, and our analysis says so for crawl4ai. The hosted reader is a service relationship, which is a different kind of answer and has to be judged as one.

Before committing, run the checks each project prescribes. For crawl4ai, confirm crawl4ai-doctor reports a working browser on the target machine, confirm the deployment accepts the Docker server's default authentication behaviour from the v0.9.0 notes, and confirm your pages survive the Markdown filtering stage rather than needing the raw HTML. For reader, read the security and rate limit information for the hosted endpoint before you depend on it, and if you self-host, plan for Node 22.15+, Chrome or Chromium, LibreOffice, and stateless behaviour without session storage.

Bottom line

crawl4ai is the right choice when the crawl belongs inside your process, because it gives you a browser you control, Markdown with structure, a CLI, deep crawl recovery and an API server, all under Apache-2.0 in Python; reader is the right choice when you want the fetch handled for you, because a prefix and a search endpoint turn URLs into Markdown with PDF, Office and image handling already in place. Both carry the same licence, so terms never decide this, and the deciding factors are whether you can run Chromium, whether you need to pin a version, and whether a shared rate limit is acceptable. Verify first that crawl4ai-doctor sees a working browser on your target machine, that your deployment accepts the Docker server's default authentication behaviour, and that your pages survive Markdown filtering, and for reader that LibreOffice and Chromium fit your container image and that you can live with stateless self-hosting. The judgement to take away is that these are not rivals in quality, since one is a component and the other is a service, so decide by who owns the browser.

Sources

1. unclecode/crawl4ai repository
2. unclecode/crawl4ai README
3. jina-ai/reader repository
4. jina-ai/reader README