Textpattern CMS support forum
You are not logged in. Register | Login | Help
- Topics: Active | Unanswered
Blocking AI bots and crawlers
As AI stays in the news in not so good way some ways to try to protect your sites
(there is never certainty that those bots will respect the directives they themselves suggest… caveat emptor). The below list is base on data from this Reuters Institute article and links therein
# AI spiders & crawlers
# https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# https://platform.openai.com/docs/gptbot
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /Please share additional possibilities.
Where is that emoji for a solar powered submarine when you need it ?
Sand space – admin theme for Textpattern
Offline
Re: Blocking AI bots and crawlers
More bots & crawlers can be found in the NYtimes robots.txt (towards the end, before the sitemap list; or search for “Amazonbot”).
Where is that emoji for a solar powered submarine when you need it ?
Sand space – admin theme for Textpattern
Offline