Googlebot goes berserk on my site

colak · 2010-10-15 05:11:21

Did you consider adding some meta information?

In the 404 page add

<meta name="revisit-after" content="30 days" />
<meta name="robots" content="noindex,follow" />

In the article list pages

<meta name="revisit-after" content="7 days" />
<meta name="robots" content="index,follow" />

for individual articles

<meta name="revisit-after" content="60 days" />
<meta name="robots" content="index,follow" />

Kjeld · 2010-10-15 12:33:11

Gocom wrote:

Google’s own Robots.txt manual is pretty helpful.

Might be that the rules are failing because the parameter all doesn’t have a value. Didn’t check but that might be the case. Don’t quote me on that.

Anyhow, to cut the amount of rules, as there is plenty, you could just block URLs that use question mark. For example something like might work (not tested):

User-agent: *
Disallow: /*?
Disallow: /present/
Disallow: /featured-images/
Disallow: /explore/
Disallow: /checkout/
Disallow: /confirmation/

If it works, it would also block some other unwantedish pages, like ?id=, ?q= and ?s= which are part of TXP’s URL structure.

Thanks, Jukka.

Had somehow overlooked the bottom of Manually create a robots.txt file on Google’s Robots.txt manual. Thanks for directing my attention to it.

Also thanks for the suggestion to use Disallow: /*?. However, this would also disallow access to many of my legit pages and ?rah_sitemap=sitemap.

I am now trying the following:

User-agent: *
Allow: /
Disallow: /*keywords=*
Disallow: /*place=*
Disallow: /*artist=*
Disallow: /*period=*
Disallow: /*year=*
Disallow: /*c=*
Disallow: /present/
Disallow: /featured-images/
Disallow: /explore/
Disallow: /checkout/
Disallow: /confirmation/

The googlebot explanations seem to imply that this should work, but I am not very confident.

Last edited by Kjeld (2010-10-15 12:33:45)

Kjeld · 2010-10-15 12:34:56

colak wrote:

Did you consider adding some meta information?

Very basic information to add, but I had not considered it at all. Thanks!

Gocom · 2010-10-15 12:42:28

Kjeld wrote:

and ?rah_sitemap=sitemap

Rah_sitemap also automatically answers to example.com/sitemap.xml and example.com/sitemap.xml.gz if you prefer clean URLs :-) Example.com being your domain.tld.

You could also add Sitemap: row to your Robots.txt to tell bots that the sitemap infact is there, even that GoogleBot should, from time to time, automatically check it.

Last edited by Gocom (2010-10-15 12:47:23)

Kjeld · 2010-10-15 23:13:40

Gocom wrote:

You could also add Sitemap: row to your Robots.txt to tell bots that the sitemap infact is there, even that GoogleBot should, from time to time, automatically check it.

Thanks, Jukka, for this great tip, too. Have added it to my robots.txt file.

Kjeld · 2010-10-15 23:19:15

Yesterday, as I mentioned in an earlier entry, I started using the following lines on my robots.txt:

Disallow: /*keywords=*
Disallow: /*place=*
Disallow: /*artist=*
Disallow: /*period=*
Disallow: /*year=*
Disallow: /*c=*

So far, googlebot hasn’t touched these links yet. But it has only been half a day and googlebot just looked at a few links, so it’s wait and see for now.

gaekwad · 2010-10-16 19:08:05

Kjeld wrote:
So far, googlebot hasn’t touched these links yet. But it has only been half a day and googlebot just looked at a few links, so it’s wait and see for now.

Did you set up the crawl frequency in Webmaster Tools, yet? Site Configuration -> Settings -> Crawl Rate -> Set Custom Crawl Rate.

Kjeld · 2010-10-16 23:23:17

gaekwad wrote:

Did you set up the crawl frequency in Webmaster Tools, yet? Site Configuration -> Settings -> Crawl Rate -> Set Custom Crawl Rate.

Thanks, Pete. I didn’t actually. But I have no problems with the current crawl rate. And more than 30 hours after I last adjusted robots.txt, googlebot (or any other robot) still has not touched the urls that I disallowed with the double asterisks (Disallow: /*keywords=*), so it seems to be working.

Thanks to all for the tips and advice!

Kjeld · 2010-10-20 22:38:37

Update: the strategy below definitely works. Googlebot has left all links that include the strings between the asterisks completely alone during the past week. I can recommend it for your site. Thanks again to all for the input on this!

Disallow: /*keywords=*
Disallow: /*place=*
Disallow: /*artist=*
Disallow: /*period=*
Disallow: /*year=*
Disallow: /*c=*
Disallow: /*q=*
Disallow: /*s=*
Disallow: /*id=*

Last edited by Kjeld (2010-10-20 22:38:58)

maniqui · 2010-10-21 23:04:40

Kjeld, roliviu smells badly like an spammer :)
I’ll report the user.

Kjeld · 2010-10-21 23:17:49

maniqui wrote:

Kjeld, roliviu smells badly like an spammer :) I’ll report the user.

Thanks. I missed that. I have deleted my response.

Textpattern CMS

Textpattern CMS support forum

#13 2010-10-15 05:11:21

Re: Googlebot goes berserk on my site

#14 2010-10-15 12:33:11

Re: Googlebot goes berserk on my site

#15 2010-10-15 12:34:56

Re: Googlebot goes berserk on my site

#16 2010-10-15 12:42:28

Re: Googlebot goes berserk on my site

#17 2010-10-15 23:13:40

Re: Googlebot goes berserk on my site

#18 2010-10-15 23:19:15

Re: Googlebot goes berserk on my site

#19 2010-10-16 19:08:05

Re: Googlebot goes berserk on my site

#20 2010-10-16 23:23:17

Re: Googlebot goes berserk on my site

#21 2010-10-20 22:38:37

Re: Googlebot goes berserk on my site

#22 2010-10-21 23:04:40

Re: Googlebot goes berserk on my site

#23 2010-10-21 23:17:49

Re: Googlebot goes berserk on my site

Board footer