Result
The result will appear here.Deciding which AI bots to let in is now a real trade-off: training crawlers take content with no link back, while AI search crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot are how ChatGPT search, Perplexity and Claude find pages to cite. This generator starts from the list maintained by ai-robots-txt/ai.robots.txt — 175 user agents in the snapshot it ships — and writes rules for three policies: block training only and explicitly allow search and user-requested fetches, block every AI bot, or block only the names you list.
How it works
- Each crawler in the snapshot is sorted into training, search, user-requested assistant, agent or link preview from the purpose the upstream list records.
- Blocked crawlers share one group of User-agent lines with a single Disallow rule, which RFC 9309 allows and keeps the file short.
- facebookexternalhit is never blocked by the ready-made policies, because blocking it breaks link previews when your pages are shared.
Where your data goes
Nowhere. This tool runs entirely in your browser: the text you paste is processed by the page and is never transmitted to a server or written to a log.
This tool is free and needs no account. Its results exist only in your open page and are not saved anywhere.
What it costs
This tool is free, with no sign-in and no points.
Common questions
- Does blocking Google-Extended remove me from Google Search?
- No. Google-Extended is a control token, not a crawler: it tells Google not to use your content for training Gemini and Vertex AI, and it does not affect whether pages appear in Search. Crawling is still done by Googlebot, which this list does not include and which you should not block unless you mean to leave Google.
- Will every bot obey these rules?
- robots.txt is a request. Major operators document their user agents and say they honour it, but the upstream list records compliance as unclear for many entries, and a scraper can simply lie about its user agent. For bots that ignore the file, blocking by user agent or IP at your CDN or firewall is the only enforcement.
- Why paste the output instead of merging it into my existing file?
- Append it to your existing robots.txt rather than replacing it: these groups name specific bots, so they do not change what your User-agent: * rules say to other crawlers. The snapshot date is written into the file; re-generate every few months, because new AI crawlers appear regularly.
The open-source behind it
This tool runs on ai-robots-txt/ai.robots.txt, released under MIT. If you need the same behaviour inside your own program, that is the library to reach for.
ai-robots-txt/ai.robots.txtAlso known as
- block ai crawlers robots.txt
- block gptbot
- robots.txt ai bots
- disallow claudebot
- google-extended robots.txt