everywall/ladder: a self-hosted CORS and HTML proxy for paywall testing

The problem Ladder addresses: header and markup control on someone else's page

Ladder is a HTTP web proxy aimed at developers, researchers and publishers who need to see how a page is actually delivered. The README frames the use case narrowly: simulating different client environments, such as browsers and crawlers, and observing how content is served under those conditions. That is a debugging and QA framing, not a reading-a-paywalled-article framing, even though the repository topics include paywall and paywall-blocker.

The practical problem is that a page's behaviour is often decided by headers you do not control. CORS headers decide whether your own script can read a response. Content-Security-Policy decides whether injected code runs. The User-Agent and X-Forwarded-For values decide which variant of the page the server returns. Ladder puts all of those knobs in one place and lets you change them per domain through a YAML ruleset. If your job is to confirm that a paywall configuration behaves consistently across clients, this is the layer you would otherwise rebuild by hand with a local proxy and a header-rewriting script.

Request and response modification, in that order

The README's sequence diagram is short and worth taking literally. A client sends GET to Ladder. Ladder applies RequestModifications. Ladder sends GET to the target website. The website returns 200 OK. Ladder applies ResultModifications. Ladder returns 200 OK to the client.

So the tool has two hooks, not one. Request modifications act on the outbound request (the User-Agent, the forwarded IP, the requested URL). Result modifications act on the response body and headers (CORS headers, Content-Security-Policy, injected HTML, CSS or JavaScript). Domain-based rules select which modifications apply. The ruleset is loaded on startup, which means a change to a rule file requires a restart rather than a hot reload; the README does not document a reload endpoint.

The Go dependencies in go.mod line up with that design: gofiber/fiber for the HTTP server, PuerktoBio/goquery for HTML manipulation, yaml.v3 for the ruleset. There is no headless browser in the dependency list, which is why the README points at FlareSolverr as an external, optional service rather than bundling rendering.

Installing Ladder with Docker and making a first request

The README gives three install paths: a release binary, a Docker image, and a Helm chart in the helm-chart subdirectory. The Docker route is the shortest. This command runs the published image, maps port 8080, and points RULESET at the ruleset file maintained in the separate ladder-rules repository.

docker run -p 8080:8080 -d --env RULESET=https://raw.githubusercontent.com/everywall/ladder-rules/main/ruleset.yaml --name ladder ghcr.io/everywall/ladder:latest

After the container starts, open http://localhost:8080 in a browser, paste a URL into the form and press Enter. The README also documents the direct form, where the target URL is appended to the proxy URL, and a bookmarklet that rewrites the current location through the proxy. If you want the response as data rather than a rendered page, the API endpoint returns it under /api/:

curl -X GET "http://localhost:8080/api/https://www.example.com"

The /raw/ prefix returns the fetched HTML without the page being made browsable, and /ruleset shows the rules currently in effect. The binary path is similar: download the release, unpack it, and run it with the same RULESET flag, for example ./ladder -r https://raw.githubusercontent.com/everywall/ladder-rules/main/ruleset.yaml. The README's own warning is the part to read twice: if the instance is publicly reachable and USERPASS is empty, anyone can route traffic through it.

Configuration that decides whether this is safe to run

The environment variable table is the real operational surface. PORT defaults to 8080. USER_AGENT defaults to a Googlebot string, and X_FORWARDED_FOR defaults to 66.249.66.1, which is a Google address; those defaults are chosen to emulate a crawler, and leaving them in place means every request your instance makes carries that identity. USERPASS, in admin:123456 format, turns on Basic Auth. LOG_URLS defaults to true and logs fetched URLs.

The scoping variables matter more than the rest. ALLOWED_DOMAINS takes a comma-separated list. ALLOWED_DOMAINS_RULESET allows domains taken from the ruleset. The README states that the two are joined, and that if both are empty no limitations are applied. EXPOSE_RULESET defaults to true, which makes your ruleset readable by other Ladder instances through the /ruleset endpoint. If you are running a private instance, that default is worth turning off. BASE_PATH exists for running the proxy under a subpath, and DISABLE_FORM plus FORM_PATH let you replace or remove the front page.

FLARESOLVERR_HOST points at an optional FlareSolverr service, defaulting to http://localhost:8191. The docker-compose.yaml ships a commented-out FlareSolverr service on port 8191 with CAPTCHA_SOLVER=none, so enabling it is an explicit edit rather than a default.

Where Ladder stops: fingerprinting, rate limits and cloaking

The README's Limitations section is unusually direct. It says many sites use fingerprinting, rate limiting or behavioural analysis to restrict automated access, and that Ladder does not circumvent such protections and may not function correctly against services that actively control access. That is the boundary of the tool. It rewrites headers and markup; it does not execute JavaScript challenges, rotate identities at scale, or defeat a bot-management system that is looking at TLS fingerprints and mouse movement.

The second limitation is collateral damage. The feature list itself includes the entry that Ladder might break tracking, ads and other third-party content. That follows from stripping CORS and CSP headers: scripts that depended on those policies can fail, and pages that assemble themselves from third-party origins may render partially. For a testing workflow that is acceptable, because you are inspecting the response. For anything resembling normal browsing, it is a defect.

The third is operational. Rules are loaded at startup, so a running instance does not pick up ruleset edits until restarted. And the Windows binary is marked untested in the feature list, so a Windows deployment is an assumption rather than a supported path.

How Ladder differs from a general-purpose scraping stack

The closest comparison in the README's own text is FlareSolverr. They are not competitors so much as adjacent layers, and the difference is the mechanism: FlareSolverr renders pages in a headless browser to get past a JavaScript challenge, while Ladder performs HTTP-level header and body rewriting and has no browser in its dependency list. Ladder's README explicitly says FlareSolverr is not part of Ladder and that its use may carry its own legal and contractual restrictions. The two are wired together through FLARESOLVERR_HOST, which is the honest way to present it: Ladder is the rewriting proxy, and browser rendering is somebody else's process.

Against a general scraping framework, the difference is scope. A scraping stack gives you scheduling, extraction and storage; Ladder gives you a browsable proxied view, a raw HTML endpoint, an API endpoint and per-domain rules. It is a debugging instrument with a UI, not a pipeline. If your goal is to collect a corpus, Ladder is the wrong shape. If your goal is to see what one URL returns to a Googlebot user agent versus a normal browser, and to strip the headers that get in the way, it is precisely the right shape.

Licence, maintenance and the cost of upgrading

Ladder is GPL-3.0. If you run it as a service for yourself, that is unremarkable. If you embed it in a product you distribute, the copyleft terms apply to the combined work, and you should read the licence text rather than take a summary from an article. This is not legal advice.

The repository is not archived, and the last push was on 2026-09-04. Releases tell a more uneven story: v0.0.21 shipped on 2023-12-07, then v0.0.22 on 2026-04-11 and v0.0.23 on 2026-05-04, with no release since. So the code has moved recently while tagged releases have not. The version is still 0.0.x, which is a fair signal about API and ruleset stability: the docker-compose.yaml and README are the contract, and a ruleset format change would land as a breaking change for anyone maintaining a private ruleset directory.

Upgrade cost is dominated by the ruleset, not the binary. Because RULESET accepts a local path, a directory of YAML files, or a URL, teams that point at the upstream ladder-rules repository inherit whatever changes land there. Pinning a local ruleset file and reviewing diffs before restarting is the cheaper posture. The Docker image is built from golang:1.26 and runs on gcr.io/distroless/static-debian13:nonroot, so the runtime image is small and has no shell, which is good for attack surface and mildly annoying for debugging.

Editorial conclusion

Adopt Ladder if you need a small, self-hosted proxy to inspect how a site serves content to different user agents and to strip headers for local testing, and if you can put it behind Basic Auth and an ALLOWED_DOMAINS list. Do not adopt it if you expect it to defeat fingerprinting, rate limiting or behavioural analysis; the README says it does not. Before exposing it, verify that USERPASS is set, that ALLOWED_DOMAINS or ALLOWED_DOMAINS_RULESET restricts the target set, and that your ruleset file loads at startup.

Frequently asked questions

Official sources

1. everywall/ladder on GitHub
2. Issues
3. License: GPL-3.0
4. README
5. Releases