ai.robots.txt: One JSON File, Six Server Configs for Blocking AI Crawlers
A list of AI agents and robots to block.
At a glance
- What is it?
- The ai-robots-txt project keeps a single list of AI crawlers in robots.json and generates robots.txt, .htaccess, Nginx, Caddy, HAProxy and lighttpd configs from it. It is a good fit for operators of small and mid-sized sites who want a copy-paste block and are willing to update it manually.
- Who is it for?
- Adopt it if you run your own web server, want a maintained block list without writing one, and accept that updates are a manual copy from a GitHub release. Skip it if you are on a CDN or managed host where you cannot edit server config, or if you need the block to update itself without you.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 8 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What the list actually blocks, and who it is for
The repository describes itself as a list of AI-related crawlers of all types, regardless of purpose. That phrasing matters. It is not a list of malicious scrapers or of crawlers that ignore robots.txt. It is a list of user-agent strings associated with AI, and the project does not sort them by whether they train models, answer live queries, or do something else. If you want to allow a search crawler that also feeds an AI product, this list does not make that distinction for you. You will have to edit it.
The audience is site operators who control their own server configuration. The README provides six artefacts: robots.txt, .htaccess, nginx-block-ai-bots.conf, Caddyfile, haproxy-block-ai-bots.txt and lighttpd-block-ai-bots.conf. That spread tells you the intended user: someone with shell access to a web server, not someone on a hosted platform with a settings panel. A WordPress user, for example, gets no plugin here. The README points at an RSL plugin for licensing content instead, which is a different goal.
Some entries are sourced from Known Agents, which the README credits. The list also links to external datasets that measure how widely this kind of blocking is deployed, including one covering German .de domains on a fixed panel of 600 domains and another covering the Tranco top 5,000. Those are third-party projects, not part of this repository, but their existence is a useful signal about how the list is being used in practice.
robots.json is the source; everything else is generated
The architecture is a single-source generator. Contributors edit robots.json. A GitHub action then generates robots.txt, table-of-bot-metrics.md, .htaccess and nginx-block-ai-bots.conf. The README says this explicitly in the contributing section, and it is the most important structural fact about the project.
That design has consequences worth spelling out. First, the generated files are not the place to make local edits. If you patch robots.txt on your server, the next release will not carry your change forward, and you have no merge path. Second, the generated set is narrower than the published set: the README lists six artefacts under Usage but names only four as generated by the action. The Caddyfile and the HAProxy and lighttpd files are published in the repository, but the README does not say the action regenerates them. Whether they are kept in sync by hand is not stated in the material, so treat that as an open question and check the repository before relying on it.
The release cadence is fast. The supplied release history shows v1.50 on 2026-08-17, v1.51 on 2026-08-26, and v1.52 on 2026-09-07. Roughly one release every one to three weeks, with titles like "add ExaSearchBot and Reflectionbot" and "Diffbot-User, OAI-Adsbot, and various others". v1.50 also mentions improved pattern matching, which suggests the list is not purely additive; entries get refined. A static copy you took six months ago is meaningfully out of date.
Installing the Nginx, Apache, Caddy, HAProxy and lighttpd variants
The README gives the mechanics for each server, and the level of detail varies.
For Nginx, nginx-block-ai-bots.conf is a snippet included inside any virtual host server {} block via the include directive. The README does not print the include line itself, so you supply the path.
For Apache httpd, .htaccess returns an error page when a listed crawler requests the server. The README adds a caveat worth repeating: the httpd documentation states that more performant methods than an .htaccess file exist. If you have access to the main server config, use it there instead.
For Caddy, the Caddyfile contains a Header Regex matcher group you copy or import, and rejection is handled with abort @aibots. That is the whole instruction. It assumes you already know where matcher groups go in a Caddyfile.
For HAProxy, the steps are concrete. Put haproxy-block-ai-bots.txt in the config directory, then add two lines to the frontend section:
acl ai_robot hdr_sub(user-agent) -i -f /etc/haproxy/haproxy-block-ai-bots.txt http-request deny if ai_robot
The README notes the path may differ in your environment. The -i flag makes the match case-insensitive, which matters because user-agent strings are not consistently cased.
For lighttpd, include the fragment with include "fragments/lighttpd-block-ai-bots.conf", either globally or inside any conditional section.
The robots.txt file itself implements the Robots Exclusion Protocol per RFC 9309. That is the polite path, and it only works against crawlers that choose to honour it. The server-level configs are the enforcement path. You may want both: robots.txt for well-behaved agents and a server block for the rest.
The Bing gap and what a robots.txt block does not cover
The README carries a one-line warning that deserves more weight than its placement gives it: Bing uses the data it crawls for AI and training, and you may opt out by adding a meta tag to the head of your site, with a link to docs/additional-steps/bing.md.
The implication is that blocking a user-agent named for an AI product does not necessarily stop a general search crawler from feeding the same pipeline. A robots.txt entry for an AI crawler and a meta tag for Bing are two separate mechanisms targeting two separate things. If your goal is to keep your content out of AI training data, adding this list to robots.txt and stopping there leaves a gap the README itself flags.
The same logic applies to the Cloudflare note. The README suggests using Cloudflare's hard block alongside the list and offers a form for reporting abusive crawlers that do not respect robots.txt. That is an acknowledgement that the list alone is not enforcement. It is a set of user-agent strings. A crawler that lies about its user-agent, or ignores robots.txt entirely, is unaffected by anything in this repository. The README does not claim otherwise, but the framing around the list can lead readers to assume more protection than it delivers.
Maintenance cost: manual copies, fast releases, MIT terms
There is no package manager here. You do not pip install a running service. The pip install -r requirements.txt step in the README is for contributors running code/tests.py, not for deployment. Deployment means downloading a file from a release and placing it on your server.
The release process is documented and manual on the maintainer side: create a tag v1.n, write a release title, generate release notes, publish. A GitHub action then attaches robots.txt as a release asset. Notably, the action attaches robots.txt only. The other five artefacts are not listed as release assets, so if you want the Nginx or HAProxy file you take it from the repository at a given commit or tag rather than from the release page.
Updates are announced through an Atom feed at the releases.atom URL, and the README lists several readers you can subscribe with, plus a GitHub watch setting. Subscribing is easy. Applying the update is not automated by anything in this repository. On a single server that is a file copy. Across a fleet, or behind a CDN, it is a change-management task you now own, on a one-to-three-week cadence.
The licence is MIT. That permits commercial use, modification and redistribution with the licence and copyright notice retained. The list is a data file, not code, so the practical question is less about licence compatibility and more about whether you redistribute it. If you do, keep the notice. This is a description of the licence text, not legal advice; check the LICENSE file for the exact terms.
Where this approach is the wrong tool
The clearest failure mode is architectural. If your site sits behind a CDN or a managed host, you may not be able to edit the server config at all. The six artefacts all assume you can. A Netlify or Vercel deployment, for instance, does not read an .htaccess file. The README links to a third-party write-up about blocking bots on Netlify edge functions, which is a sign that the list's own artefacts do not cover that case and you are on your own for the translation layer.
A second limitation is the scope of the list itself. It blocks AI-related crawlers of all types regardless of purpose. If you want to allow an AI-adjacent crawler that drives referral traffic to your site, or one your own tooling depends on, you have to carve out an exception by hand. The repository does not offer a curated subset or a category filter in the material provided.
A third is the update model. A block list is only as good as its freshness, and freshness here depends on you noticing a release and re-copying a file. There is no auto-update mechanism in this repository. The Traefik plugin mentioned in the README is the closest thing to an automated path, and it is a separate project, not part of this one.
Finally, the list is a heuristic. User-agent matching is trivially evadable. The README's own pointer to a form for reporting crawlers that ignore robots.txt is the honest version of this: the project knows the list is not a wall.
How it compares to Cloudflare's hard block and to writing your own rules
The most direct alternative the README itself names is Cloudflare's hard block, which it describes as blocking AI bots and scrapers with a single click. The difference in approach is where enforcement happens. Cloudflare's block runs at the edge, before a request reaches your origin, and Cloudflare maintains the list. This project runs at your origin, in your Nginx or Apache or HAProxy config, and you maintain the copy. Cloudflare gives you less control and less work. This project gives you more of both, and it works on servers that are not behind Cloudflare at all.
The README suggests using the two together, which is a reasonable reading: Cloudflare's edge block for the traffic it can see, this list for the origin. Cloudflare also publishes a verified bots list that the README points to as useful, which is the inverse dataset: crawlers you might want to allow.
The other alternative is writing your own rules. That is what most people do before finding a list like this, and the cost is obvious. You have to discover each new crawler, decide whether to block it, and add a pattern. This project externalises the discovery step and the pattern-writing, at the cost of accepting someone else's judgement about what counts as an AI crawler and how to match it. Given a release cadence of roughly one to three weeks, the discovery step is not trivial to replicate alone.
A third option the README raises is RSL, Really Simple Licensing, which lets you license content to AI companies through robots.txt, with an option of collective bargaining. That is the opposite posture: monetise access rather than deny it. A WordPress plugin implementing RSL and payment processing is linked. If your goal is compensation rather than exclusion, this project is not the tool.
Verifying a release before you deploy it
The project ships tests. The README says you can run them by installing Python 3, running pip install -r requirements.txt, and then executing code/tests.py. If you are about to push a new robots.json-derived file to production, running those tests against the version you intend to deploy is the cheapest check available, and it is a step the documentation supports directly.
Beyond that, the README's own links are the verification surface. The table-of-bot-metrics.md file describes the listed crawlers. The FAQ is linked at the top. Known Agents is credited as a source. The external datasets (KI-Zugangsindex on a fixed panel of 600 .de domains, and the AI Crawler Census over the Tranco top 5,000) are third-party measurements of how widely this kind of blocking is deployed, and they are published per-run so the same domains can be compared over time. They will not tell you whether a given user-agent string is correct, but they will tell you whether the blocking pattern is common enough to be worth the effort.
The practical pre-deployment check is to read robots.json and confirm the patterns against your own access logs. The list is a general-purpose one, and your traffic is specific. A pattern that matches a crawler you want to keep, or that misses one hitting you daily, is something only your logs will show.
Editorial conclusion
Adopt it if you run your own web server, want a maintained block list without writing one, and accept that updates are a manual copy from a GitHub release. Skip it if you are on a CDN or managed host where you cannot edit server config, or if you need the block to update itself without you. Before rolling it out, read robots.json to see whether the user-agent patterns match the traffic in your logs, and check the FAQ for the note on Bing, since a robots.txt block does not cover Bing's use of crawled data for AI and training. That last point is the one most people miss.
Community notes