Crawl4AI or Firecrawl for turning a docs site into Markdown for RAG. Which is easier to self-host?

For your scale, Crawl4AI as a library in a script you schedule is the smaller commitment by a wide margin. A few hundred pages a week is a loop with a concurrency limit, not a queue; you do not need Redis for it. The Playwright browser is the only heavy dependency and you have that either way.

Firecrawl's stack earns its weight when many unrelated jobs are submitted by many callers and you want an API in front of them, or when you want the hosted service and only self-host as a fallback. For one pipeline that is you, it is three containers to keep up for one cron job.

On Markdown quality: both produce usable output on documentation sites; the differences are in boilerplate removal (navigation, footers), where you will tune either one for the specific site. Crawl4AI's content-filtering options are configurable per crawl, which is where that tuning happens.

Whichever you choose, decide chunking before you crawl, because the crawler's output shape sets what the chunker sees. Documentation pages have headings; chunking on headings rather than fixed token windows keeps a section's code sample with its explanation. /tools/rag-text-chunker lets you try both on a real page before you commit, and /learn/embeddings has the background on why chunk boundaries matter more than the embedding model.

Library plus cron it is. Leaving this open for a week in case someone has run both on the same site and has quality notes; will accept after that.