Go to main content

Textpattern CMS support forum

You are not logged in. Register | Login | Help

#1 2024-02-28 02:57:13

phiw13
Plugin Author
From: Japan
Registered: 2004-02-27
Posts: 3,076
Website

Blocking AI bots and crawlers

As AI stays in the news in not so good way some ways to try to protect your sites
(there is never certainty that those bots will respect the directives they themselves suggest… caveat emptor). The below list is base on data from this Reuters Institute article and links therein

# AI spiders & crawlers
# https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# https://platform.openai.com/docs/gptbot
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /

Please share additional possibilities.


Where is that emoji for a solar powered submarine when you need it ?
Sand space – admin theme for Textpattern

Offline

#2 2024-03-02 02:08:44

phiw13
Plugin Author
From: Japan
Registered: 2004-02-27
Posts: 3,076
Website

Re: Blocking AI bots and crawlers

More bots & crawlers can be found in the NYtimes robots.txt (towards the end, before the sitemap list; or search for “Amazonbot”).


Where is that emoji for a solar powered submarine when you need it ?
Sand space – admin theme for Textpattern

Offline

Board footer

Powered by FluxBB