Textpattern CMS support forum
You are not logged in. Register | Login | Help
- Topics: Active | Unanswered
Re: Apache: redirecting deep links
robots.txt is entirely voluntary, and all the evil crawlers ignore it. Also, seemingly legitimate crawlers ignore robots.txt because they say that they aren’t ‘crawling’ but rather ‘fetching’. I have…let me count…51 crawlers blocked in my .htaccess file right now, each of which has, in the past, ignored robots.txt.
(My comment about modifying Txp was motivated by the existence of plugins; I know that they exist, but I have no idea what they can and cannot do.)
Offline