TRAWL: a FlareSolverr replacement that adds a challenge-aware forward proxy
Self-hosted scraping engine — bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.
At a glance
- What is it?
- TRAWL is a TypeScript scraping engine that solves JS challenges and CAPTCHAs and exposes both a FlareSolverr-compatible API and an HTTP/HTTPS proxy for indexers that re-fetch pages with their own client. The proxy is the interesting part; the AGPL licence and the residential-proxy dependency are the parts to think about before adopting.
- Who is it for?
- Adopt TRAWL if you already run Prowlarr, Jackett, Sonarr or Radarr and you have indexers that FlareSolverr or Byparr cannot get past, particularly ones where the clearance cookie does not survive being replayed by another HTTP client. Do not adopt it if you cannot run Redis alongside it, if you are unwilling to install a generated CA into a client trust store, or if AGPL-3.0 is incompatible with how you ship software.
- Can I use it commercially?
- Yes, with strict conditions. AGPL-3.0 is a network copyleft licence: if people use a modified version over a network, for example as a hosted service, you must offer them its source code under the same licence.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly TypeScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem TRAWL targets: indexers that fail the challenge on the second request
Most self-hosted scraping setups in the *arr ecosystem hit the same wall. An indexer sits behind Cloudflare or a similar bot manager, Prowlarr or Jackett asks a solver service for a page, the solver gets a clearance cookie, and then the indexer client re-fetches the page with its own HTTP stack. Cloudflare sees a different connection fingerprint and challenges again. The cookie was never portable. The README states this explicitly for the /v1 flow: Prowlarr keeps only the cookie and user-agent and re-fetches the page with its own HTTP client, so the cookie is not portable and the page gets re-challenged.
TRAWL is aimed at that specific gap. It is a self-hosted scraping engine written in TypeScript, licensed AGPL-3.0, with dedicated flows for Cloudflare, Akamai Bot Manager and Imperva/Incapsula described as best effort, plus Turnstile, reCAPTCHA, hCaptcha and GeeTest handling. The audience is narrow and concrete: people running Prowlarr, Jackett, Sonarr, Radarr and the rest of the *arr stack who are currently pointing those tools at FlareSolverr or Byparr. The README positions TRAWL as a drop-in replacement for that role and as an alternative to a paid solver API, since reCAPTCHA audio can use Google's free STT endpoint or an optional local Whisper service.
Four tiers, a Redis session cache, and a Camoufox browser pool
The execution model is a four-tier escalation ladder: plain HTTP fetch, then a cached browser session, then a fresh challenge solve, then a residential proxy. Requests start cheap and only climb when a wall is detected. The native /scrape endpoint returns which tier was used, along with timings, a sessionCached flag and the full cookie list, so the escalation is observable rather than implicit.
The cache is Redis. Solved cookies and the browser identity are stored there, and an accepted session can avoid a fresh solve on a later request. That is the mechanism behind the speed claim in the README, which says TRAWL is 2-6x faster than FlareSolverr or Byparr with a higher success rate. I cannot verify that number from the repository material, and the README does not publish the methodology behind it, so treat it as a vendor claim rather than a measurement.
The browser is Camoufox, a Firefox build the README describes as fingerprint-patched at the C++/Juggler level to reduce automation signals. Captcha handling is native rather than delegated to a paid API: CF Turnstile and interstitial, reCAPTCHA v2 using free STT, hCaptcha, and GeeTest v4 Slide. The README does not document per-captcha success rates, which is the number that would actually matter for capacity planning.
Running it: docker compose, port 8191, and the FlareSolverr-compatible endpoint
The quick start is three commands. Clone the repository, copy .env.example to .env, and run docker compose up -d, which starts the scraper plus Redis. The README notes that the first boot takes 15 to 30 seconds while the browser pool warms up, and that subsequent starts are fast. Verification is a curl against http://localhost:8191/health.
The FlareSolverr-compatible surface lives at /v1 and takes the familiar request shape: a POST with cmd set to request.get, a url, and a maxTimeout in milliseconds. Pointing Prowlarr or Jackett at it means setting the FlareSolverr URL to http://localhost:8191 on the same host, or http://trawl:8191 when both run under Docker Compose on the same network. There is also a native /scrape endpoint with the richer response fields, and an /mcp endpoint that is off unless MCP_ENABLED=true is set, exposing Streamable HTTP tools for readable content, HTML, screenshots and browser diagnostics to MCP-compatible clients. The README is careful to note that those tools load known public URLs and that TRAWL does not provide web search or ranking.
For NAS users, the README points to community app catalogs for TrueNAS and Unraid, where TRAWL can be found by searching the app store rather than by running Compose by hand.
The forward proxy is the part that differs from FlareSolverr
The /v1 endpoint cannot fix the re-fetch problem, and the README says so. The answer is a challenge-aware HTTP/HTTPS forward proxy. Normal traffic is forwarded directly; detected walls trigger automatic tier escalation. WebSockets, binary bodies and Range/206 requests are supported. You enable it with MITM_ENABLED=true and set MITM_PORT, which defaults to 8192 in the example, plus MITM_CA_DIR pointing at a directory such as /data/proxy-ca that must be a mounted volume so the generated CA survives container restarts.
Two keys control behaviour in ways worth reading twice. MITM_MAX_TIER caps escalation, and the README gives MITM_MAX_TIER=3 as the way to stay off residential proxies. MITM_ALWAYS_SCRAPE=false is the default and opts in to bypassing the proxy's direct Tier 0 probe, meaning by default the proxy tries a plain forward first. The listener binds 0.0.0.0 by default so clients on a Docker bridge network can reach it; MITM_HOST=127.0.0.1 restricts it to loopback on bare metal.
There is a real operational cost here. Because the proxy issues per-host certificates, the client has to trust the proxy's CA. The README's instruction is to install that CA into the client's trust store, with the CA served from the TRAWL host. Anyone who has done this before knows it is the step that gets skipped, and skipping it produces TLS errors that look like the proxy is broken.
Where TRAWL is the wrong tool, and what the README does not tell you
The README uses the phrase best effort for Akamai Bot Manager and Imperva/Incapsula, and the repository description applies the same qualifier to captcha solving overall. That is honest labelling, but it means TRAWL is not a guarantee. A site that changes its challenge script, or one whose bot manager is not among the three named, may simply fail, and the four-tier ladder will have spent a browser session and possibly a residential proxy request before it does.
Redis is a hard dependency, not an optional cache. The quick start brings it up in the same Compose file, and the session cache is what makes the tier-two path useful. If you are running TRAWL without Redis, or with a Redis that loses its data on restart, you have removed the mechanism the speed claim rests on.
The residential proxy tier implies a proxy provider, and the README's sponsor section points at two commercial proxy vendors with discount codes. Whether the fourth tier works at all depends on credentials you supply; the repository material does not describe a bundled proxy. If you set MITM_MAX_TIER to cap escalation, you are choosing to fail rather than to escalate, which is a legitimate choice but one you should make deliberately.
Finally, the README does not document resource consumption for the browser pool, nor how many concurrent solves a single instance handles. For a homelab that is fine. For anything with throughput requirements, that is the first thing to establish on your own hardware.
The alternative, and the actual difference in approach
The obvious alternative is FlareSolverr, which TRAWL is explicitly built to replace. FlareSolverr exposes a solver API and hands back a cookie and user-agent. The consuming client then makes its own request. That architecture works when the challenge is satisfied by a cookie alone, and it fails when the site binds clearance to the solving browser's connection fingerprint, which is the case the TRAWL README calls out by name.
Byparr is the other named alternative, and the README groups it with FlareSolverr in the same comparison, describing TRAWL as faster and a drop-in replacement for both. The structural difference is the same: TRAWL adds a forward proxy mode so the client's traffic passes through the solving browser's connection rather than being replayed from a different stack. That is a different design, not a tuning difference, and it is why the CA installation step exists at all.
A paid unblocking service is the third option, and the README's sponsor links make the commercial relationship visible. Those services handle the proxy network and the challenge logic for you, at a per-request cost, and they do not require you to run a browser pool or trust a local CA. TRAWL's pitch is that you keep the traffic on your own hardware and pay for proxies only when the fourth tier is reached.
Licence and maintenance considerations
TRAWL is AGPL-3.0. That matters more than usual here, because the software is designed to be run as a network service and, in proxy mode, to sit in the request path of other applications. If you modify TRAWL and expose it to users over a network, the AGPL's source-availability obligation is the thing to understand before you do. I am not giving legal advice; if you plan to redistribute a modified TRAWL or offer it as a hosted service, get your own reading of the licence.
The release cadence visible in the material is tight: v1.5.0 on 2026-09-04, v1.4.2 and v1.4.1 both on 2026-08-21. Three releases in roughly two weeks suggests active development, and it also means you should expect to upgrade. The default branch is dev, not main, so if you build from source rather than pulling a release image, you are tracking a development branch.
Upgrade cost is dominated by the CA and the browser pool rather than by the application code. If you persist MITM_CA_DIR in a volume, as the README instructs, certificate trust survives container replacement and upgrades are mostly a matter of pulling a new image and restarting. Skip that volume and every restart regenerates the CA, which means reinstalling it in the client trust store each time. The 15 to 30 second warm-up on first boot applies to each fresh container, so restarts are not instant.
Editorial conclusion
Adopt TRAWL if you already run Prowlarr, Jackett, Sonarr or Radarr and you have indexers that FlareSolverr or Byparr cannot get past, particularly ones where the clearance cookie does not survive being replayed by another HTTP client. Do not adopt it if you cannot run Redis alongside it, if you are unwilling to install a generated CA into a client trust store, or if AGPL-3.0 is incompatible with how you ship software. Before trusting it, verify first that the indexers you care about actually work through the /v1 endpoint, and only then enable MITM_ENABLED and set MITM_MAX_TIER explicitly so escalation stops before the residential tier if you do not want that traffic.
Community notes