Textpattern CMS support forum
You are not logged in. Register | Login | Help
- Topics: Active | Unanswered
Re: Googlebot goes berserk on my site
Kjeld wrote:
and
?rah_sitemap=sitemap
Rah_sitemap also automatically answers to example.com/sitemap.xml and example.com/sitemap.xml.gz if you prefer clean URLs :-) Example.com being your domain.tld.
You could also add Sitemap: row to your Robots.txt to tell bots that the sitemap infact is there, even that GoogleBot should, from time to time, automatically check it.
Last edited by Gocom (2010-10-15 12:47:23)
Offline
Re: Googlebot goes berserk on my site
Gocom wrote:
You could also add
Sitemap:row to your Robots.txt to tell bots that the sitemap infact is there, even that GoogleBot should, from time to time, automatically check it.
Thanks, Jukka, for this great tip, too. Have added it to my robots.txt file.
• Old Photos of Japan – Japan in the 1850s~1960s (100% txp)
• MeijiShowa – Stock photos of Japan in the 1850s~1960s (100% txp)
• JapaneseStreets.com – Japanese street fashion (mostly txp)
Offline
Re: Googlebot goes berserk on my site
Yesterday, as I mentioned in an earlier entry, I started using the following lines on my robots.txt:
Disallow: /*keywords=*
Disallow: /*place=*
Disallow: /*artist=*
Disallow: /*period=*
Disallow: /*year=*
Disallow: /*c=*So far, googlebot hasn’t touched these links yet. But it has only been half a day and googlebot just looked at a few links, so it’s wait and see for now.
• Old Photos of Japan – Japan in the 1850s~1960s (100% txp)
• MeijiShowa – Stock photos of Japan in the 1850s~1960s (100% txp)
• JapaneseStreets.com – Japanese street fashion (mostly txp)
Offline
Re: Googlebot goes berserk on my site
Kjeld wrote:
So far, googlebot hasn’t touched these links yet. But it has only been half a day and googlebot just looked at a few links, so it’s wait and see for now.
Did you set up the crawl frequency in Webmaster Tools, yet? Site Configuration -> Settings -> Crawl Rate -> Set Custom Crawl Rate.
Offline
Re: Googlebot goes berserk on my site
gaekwad wrote:
Did you set up the crawl frequency in Webmaster Tools, yet? Site Configuration -> Settings -> Crawl Rate -> Set Custom Crawl Rate.
Thanks, Pete. I didn’t actually. But I have no problems with the current crawl rate. And more than 30 hours after I last adjusted robots.txt, googlebot (or any other robot) still has not touched the urls that I disallowed with the double asterisks (Disallow: /*keywords=*), so it seems to be working.
Thanks to all for the tips and advice!
• Old Photos of Japan – Japan in the 1850s~1960s (100% txp)
• MeijiShowa – Stock photos of Japan in the 1850s~1960s (100% txp)
• JapaneseStreets.com – Japanese street fashion (mostly txp)
Offline
Re: Googlebot goes berserk on my site
Update: the strategy below definitely works. Googlebot has left all links that include the strings between the asterisks completely alone during the past week. I can recommend it for your site. Thanks again to all for the input on this!
Disallow: /*keywords=*
Disallow: /*place=*
Disallow: /*artist=*
Disallow: /*period=*
Disallow: /*year=*
Disallow: /*c=*
Disallow: /*q=*
Disallow: /*s=*
Disallow: /*id=*Last edited by Kjeld (2010-10-20 22:38:58)
• Old Photos of Japan – Japan in the 1850s~1960s (100% txp)
• MeijiShowa – Stock photos of Japan in the 1850s~1960s (100% txp)
• JapaneseStreets.com – Japanese street fashion (mostly txp)
Offline
Re: Googlebot goes berserk on my site
Kjeld, roliviu smells badly like an spammer :)
I’ll report the user.
Offline
Re: Googlebot goes berserk on my site
maniqui wrote:
Kjeld, roliviu smells badly like an spammer :) I’ll report the user.
Thanks. I missed that. I have deleted my response.
• Old Photos of Japan – Japan in the 1850s~1960s (100% txp)
• MeijiShowa – Stock photos of Japan in the 1850s~1960s (100% txp)
• JapaneseStreets.com – Japanese street fashion (mostly txp)
Offline