caddy-defender: A Caddy Module for Blocking AI and Cloud Crawlers

What caddy-defender Solves, and for Whom

The README describes caddy-defender as "a middleware for Caddy that allows you to block or manipulate requests based on the client's IP address," with the stated goal of "preventing unwanted traffic or polluting AI training data by returning garbage responses." That is a narrow problem with a narrow audience. The project is for people who already run Caddy as their web server or reverse proxy and want a filter that sits inside the existing configuration rather than in front of it.

The audience is narrower than "anyone worried about AI scraping." Because the mechanism is IP range matching, caddy-defender is useful when the traffic you want to stop originates from infrastructure you can name: a cloud provider's published ranges, a specific service's egress addresses, a VPN pool. It is not a user-agent filter and it does not fingerprint browsers. If a crawler runs from a residential connection or rotates through addresses outside the embedded lists, this module will not see it as anything unusual. That is the honest boundary of the design.

How the IP Matching and Responder Pipeline Works

The repository layout makes the architecture legible. The plugin.go file defines the Caddy directive, config.go holds the configuration parsing, matchers/ contains the matching logic, ranges/ contains the embedded IP data, and responders/ contains one implementation per response strategy. The go.mod file lists github.com/gaissmai/bart, a data structure for fast IP prefix lookups, and github.com/viccon/sturdyc, a caching library. Those two dependencies are the core of the request path: an incoming client IP is checked against a prefix table built from the configured ranges, and the result is cached so repeat addresses do not pay the lookup cost again.

The data flow is straightforward. Caddy receives a request, the defender middleware extracts the client IP, and the matcher tests it against the active range set. If the IP falls inside a range, the configured responder takes over. If it does not, the request passes to the next handler untouched. The ranges themselves come from two sources. Embedded ranges are compiled into the binary from the files in ranges/fetchers/, covering providers such as OpenAI, DeepSeek, GitHub Copilot, Google Cloud, Azure, AWS, Oracle Cloud, Mistral, Vultr, Cloudflare, DigitalOcean, Linode and Datadog. Custom ranges are supplied in the Caddyfile.

The responder choice is where the project's character shows. Block returns 403. Custom returns a message you write. Drop closes the connection. Redirect returns 308 to a URL you specify. Ratelimit defers to caddy-ratelimit. Garbage returns fabricated data. Tarpit streams slowly. The last two are the ones that make this project different from a plain firewall rule: they are designed to waste a crawler's time and budget rather than just turn it away.

Installing caddy-defender with Docker or xcaddy

The README gives Docker as the easiest path. Pull the prebuilt image, then run it with your Caddyfile mounted. The commands below are copied from the README; replace the path with your own.

docker pull ghcr.io/jasonlovesdoggo/caddy-defender:latest

docker run -d \
  --name caddy \
  -v /path/to/Caddyfile:/etc/caddy/Caddyfile \
  -p 80:80 -p 443:443 \
  ghcr.io/jasonlovesdoggo/caddy-defender:latest

After the container starts, Caddy reads the mounted Caddyfile as usual. The README points to the online documentation for other installation methods, and the repository's Dockerfile shows the build path for anyone compiling their own image: it uses the caddy:builder stage and runs xcaddy build with --with pkg.jsn.cam/caddy-defender=/defender, then copies the resulting binary into a caddy:latest image.

For a first real use, the Caddyfile syntax is the defender directive followed by a responder name and an optional block. The README documents three optional keys: message, ranges and url. The README gives this syntax block for the directive.

defender <responder> {
    message <custom message>
    ranges <ip_ranges...>
    url <url>
}

With that shape, requests from the ranges you list are handed to the responder you named, and everything else proceeds normally. The README notes that if you omit ranges entirely, the directive defaults to aws azurepubliccloud deepseek gcloud githubcopilot openai. That default is broad, and it is worth reading plugin.go before relying on it, because blocking all of AWS and Azure will affect far more than AI crawlers.

The Tarpit and Garbage Responders Are Not Free Wins

The README lists tarpit as a responder that streams data at a slow, configurable rate to stall bots, and garbage as one that returns fabricated data. Both are appealing if your goal is to make scraping expensive. Both also hold connections open on your server for as long as the client stays connected. A tarpit converts a crawler's request into a long-lived connection on your side, which means a sufficiently determined or distributed crawler can consume connection slots and worker capacity. The README does not document connection limits, timeouts or concurrency caps for the tarpit responder, so the operator has to reason about that exposure themselves.

The same caution applies to drop. Closing a connection without a response is cheap for you, but it gives the client no signal, and some legitimate clients inside a cloud range will simply retry. If your site serves API consumers hosted on AWS or Azure, a default configuration that blocks those ranges will break them, and the failure will look like a network problem rather than a policy decision. The README does not document an exclusion or allowlist key in the defender directive, though the repository does contain an examples/whitelist/ directory, which suggests the pattern exists. Check that directory before assuming you can carve out exceptions.

How caddy-defender Differs from a General WAF or Fail2ban

The obvious alternative is a general-purpose web application firewall or a tool like Fail2ban. The difference is in what triggers a block. Fail2ban watches logs for repeated failures and bans addresses reactively, after the behaviour has already happened. A WAF typically matches on request patterns, headers, user agents and signatures, and is usually deployed as a separate layer with its own configuration language.

caddy-defender is proactive and static. It does not observe behaviour; it checks membership in a set of IP prefixes that you or the maintainers have decided are worth blocking. That makes it fast and predictable, and it makes it blind to anything outside those prefixes. A WAF can catch a scraper that spoofs a browser user agent from a residential IP. caddy-defender cannot. Conversely, caddy-defender needs no signature updates, no log parsing, and no separate process, because it runs inside Caddy and the ranges are compiled in. If you already run Caddy and your threat model is "known cloud and AI providers," this is less machinery than a WAF. If your threat model includes adaptive adversaries, it is the wrong layer.

Maintenance, Versioning and Licence

The repository is not archived, and the last push was on 2026-09-13, one day before this writing. Releases are infrequent: v0.9.0 in June 2025, v0.10.0 in December 2025, and v0.10.1 in May 2026. The version numbers suggest a project that ships when there is something to ship rather than on a schedule. Because the embedded ranges are compiled into the binary, updating those ranges requires a new build or a new image pull; the README does not describe a mechanism for refreshing ranges at runtime. That is the main upgrade cost to plan for. If a provider changes its egress addresses, you are waiting on a release or supplying custom ranges yourself.

The licence is MIT, which permits commercial and private use, modification and redistribution with the licence and copyright notice retained. That is permissive and unsurprising for a Caddy module. The go.mod pins caddy/v2 v2.11.4 and Go 1.25.10, so building from source requires a toolchain at least that new. This is not legal advice; read the LICENSE file in the repository for the actual terms.

Editorial conclusion

caddy-defender suits operators already running Caddy who want a low-effort way to keep AI and cloud crawlers off their content, and who accept that the match is by IP range rather than by user agent. It is the wrong tool if you need per-request identity, if your traffic arrives through a CDN that hides the real client IP, or if you cannot tolerate blocking legitimate users who happen to sit inside a cloud provider's ranges. Before deploying, verify which ranges the default configuration actually covers by reading plugin.go, and confirm that Caddy's remote_ip handling sees the true client address in your setup.

Frequently asked questions

Official sources

1. JasonLovesDoggo/caddy-defender on GitHub
2. License: MIT
3. Project website
4. README
5. Releases