AnakinScraper OSS: a Go scraping API with a handler chain, Camoufox and Thompson Sampling proxies

The problem it targets: LLM pipelines that need page text, not HTML

Feeding a language model from the web is mostly a preprocessing problem. You fetch a page, strip navigation and cookie banners, and convert the remainder to something a model can consume. AnakinScraper OSS is aimed at that step. The README describes it as "the open-source web scraping API for AI" and says it turns a website into "LLM-ready markdown or structured data", naming RAG pipelines and AI agents as the intended consumers. That framing matters because the output is markdown by default, not a DOM tree. The project assumes you do not want to write a readability heuristic yourself, and it also assumes you want to own the infrastructure rather than call a hosted scraping vendor. The second assumption is what the self-hosting story is built around: the README states there are no cloud dependencies, no Redis, no AWS and no message queues, and that a zero-config mode runs with Go alone. If your pipeline already has a fetch-and-clean step you are happy with, this project is not replacing anything you lack. If you are assembling one from scratch and would rather not maintain the browser automation, the handler chain is the part worth evaluating.

The handler chain: HTTP first, browser second, paid API last

The central mechanism is a three-stage fallback chain. Per the README, each handler tries in order and if one fails the next picks up automatically: a free local HTTP fetch, then the anti-detect browser, then an external API. The README claims most pages resolve on the local HTTP handler and that paid APIs are called only for roughly the five percent that need them. Treat that percentage as the project's own estimate, not a measured figure. The fallback is not triggered by HTTP status codes alone. Domain configs let you define failure patterns and required patterns per domain, and the README gives CAPTCHA pages as the example of a failure pattern. If scraped content matches a failure pattern, or misses a required pattern, the job retries with the next handler. That is the part with real design thought in it: a 200 response containing a bot check is the common failure mode in scraping, and pattern matching is a reasonable way to detect it. The cost is that you now own a per-domain configuration surface. Every domain with unusual content needs its own required pattern, or the chain will either accept a broken page or retry unnecessarily and burn the external API budget the chain exists to protect.

Camoufox for the browser stage and Thompson Sampling for proxy choice

The browser handler uses Camoufox, described in the README as anti-detect Firefox with realistic fingerprints rather than headless Chrome. The comparison table in the README places Camoufox against headless Chrome for the project itself and Playwright for Crawlee. The distinction is fingerprint surface: a stock headless Chrome instance is a well-known signal, and Camoufox is a patched Firefox build intended to look like an ordinary browser. Whether that holds against any particular anti-bot vendor is not something the README claims, and I cannot verify it without running the stack. Proxy selection is the other learning component. The README states that Thompson Sampling picks the best proxy per domain and learns from success and failure in real time, and the web dashboard exposes a Proxy Scores page for the resulting per-proxy performance. The comparison table contrasts this with round-robin for Firecrawl and manual selection for Crawlee and Scrapy. Thompson Sampling is a sensible fit here because you are choosing among arms with unknown and drifting success rates, and it balances exploration against exploitation without a tuning pass. The limitation is cold start: with a fresh proxy pool the algorithm has no history, so early requests distribute close to randomly until enough outcomes accumulate. If your proxy pool rotates frequently, the scoring never converges on anything useful.

Running it: make up, or go run with no database

There are two start paths and they differ in what you get. The Docker path is the full stack: clone the repository, run make up, and three containers start. The README lists the server on port 8080 (REST API plus worker pool), the browser service on port 9222 (Camoufox over WebSocket) and PostgreSQL on port 5432 for job storage. The no-Docker path needs only Go 1.25 and two commands: cd server then go run cmd/server/main.go. In that mode jobs are held in memory and lost on restart, and the README says to set DATABASE_URL for persistence. JavaScript-heavy sites still need the browser service via Docker, so the lightweight path is genuinely limited to pages that resolve over plain HTTP. The API surface is three endpoints. POST /v1/scrape returns the full result synchronously, with a 30 second default timeout configurable through the timeout request field up to a maximum of 120 seconds. POST /v1/url-scraper submits an asynchronous job that you then poll at /v1/url-scraper/JOB_UUID, with batch support up to 10 URLs. Structured extraction is opt-in by setting generateJson to true in the request body and requires a GEMINI_API_KEY; the README notes no API keys are needed for scraping itself. The web dashboard is a separate React 19 app started with npm install and npm run dev from the webapp directory, listening on port 3000 and proxying to the server on 8080.

What the material does not settle: version maturity and the telemetry release

Two releases exist, v0.1.0 in March 2026 and v0.1.1 shortly after, and the v0.1.1 release is labelled Anonymous Telemetry. That is worth pausing on. A project whose selling point is self-hosting with no cloud dependencies added telemetry in its second release, and the README does not document what is collected or how to disable it. I cannot tell you from the supplied material whether it is opt-out, what endpoint receives it, or whether it fires in the zero-config mode. Anyone deploying this in an environment with egress restrictions or data handling requirements should read the v0.1.1 release notes and the relevant source before running make up, because the README as provided does not answer the question. The broader maturity picture is a v0.1 line: the API shape, the domain config schema and the handler interface are all plausible candidates for breaking changes. The README's own links point to docs/handlers.md, docs/domain-configs.md and docs/proxy-pool.md, which suggests the documentation lives outside the README and is the authoritative source. Treat the README as an introduction and those three files as the contract.

Where it is the wrong tool, and what the alternatives do differently

The clearest mismatch is a team that wants a scraping library, not a service. AnakinScraper OSS is a server plus worker pool plus optional browser container plus optional database. If you are scraping a handful of known pages inside an existing Python data pipeline, that footprint is hard to justify. Scrapy is the direct contrast: it is a Python framework you import and extend, with no HTTP API and no separate browser container, and its own concurrency model rather than a worker pool behind a REST endpoint. You write spiders; you do not POST JSON. Crawlee sits between the two, offering a Node.js library with Playwright-based browser automation and a request queue, and it is something you embed rather than deploy. The README's comparison table lists both as single-mode, meaning no handler chain fallback, and both as manual for proxy handling. That is the real differentiator: if you only ever need one fetch strategy, the chain adds configuration without adding capability. A second mismatch is cost control. The chain's value proposition depends on the external API handler being invoked rarely. If your target sites are consistently JavaScript-heavy or consistently behind bot detection, the chain degrades into an expensive passthrough and you are paying for a paid API plus the operational cost of the browser container. The domain configs let you force handler selection per domain, so the honest test is to point it at your actual targets and look at which handler resolves them before deciding the architecture pays for itself.

Licence and the cost of staying current

The project is AGPL-3.0. That is a copyleft licence with a network clause: if you modify the software and expose it to users over a network, the licence's terms extend to the modified version you are serving. For internal scraping behind a private endpoint this is usually unremarkable. For anyone embedding the server into a product they sell, the licence is the first thing to check with counsel, and I am not giving legal advice here. The upgrade cost is harder to estimate from the material. Two releases in the v0.1 line means the API and the domain config schema are early, and the README's documentation links suggest behaviour is specified in separate files rather than in the README itself. A team adopting this should pin a version, watch the release notes for changes to the handler interface and the domain config keys, and be prepared to re-read docs/domain-configs.md when they upgrade. The dependency surface is small in the no-Docker mode, which helps: Go 1.25 and nothing else. The Docker mode pulls in Camoufox and PostgreSQL, and Camoufox is an external browser build whose fingerprinting behaviour changes on its own schedule, outside this project's control.

Editorial conclusion

Adopt AnakinScraper OSS if you want a self-hosted scrape endpoint in Go that returns markdown for RAG ingestion and you are willing to run Camoufox and PostgreSQL alongside it. Do not adopt it if you need a stable public API contract: two releases, both in the v0.1 line, with the second one adding anonymous telemetry. Before committing, read docs/domain-configs.md and confirm the failure-pattern and required-pattern syntax, because that is the mechanism the entire retry design rests on, and check what the telemetry introduced in v0.1.1 sends.

Official sources

1. Anakin-Inc/anakin on GitHub
2. License: AGPL-3.0
3. Project website
4. README
5. Releases