Textpattern CMS

#1 2006-09-26 18:25:13

Myjestic

Member

Registered: 2006-04-22

Posts: 17

no index following

Hi all,

i realized, that google does not index the sub pages.
Only the main page. On this page all other links are shown like <code>http://www.mydomain.de/abc</code>

Robots.txt contains:
<code>
User-agent: *
Disallow:
</code>
Sitemap is okay.

And the meta contains the following:
<code>
<meta name=“robots” content=“index,follow” />
</code>
I tried a spider software and it follows all links correctly.

What’s wrong with my site?

#2 2006-09-26 21:58:06

jm

Plugin Author

From: Missoula, MT

Registered: 2005-11-27

Posts: 1,746

Website

Re: no index following

If it’s new, it’ll take a little while to get indexed. It took about a week for my last project to get indexed.

---

What? My name is… | GitHub

#3 2006-09-27 03:33:41

Mary

Sock Enthusiast

Registered: 2004-06-27

Posts: 6,236

Re: no index following

Why are you disallowing the robots in the first place, if you want them to index your pages?

#4 2006-09-27 07:56:38

Myjestic

Member

Registered: 2006-04-22

Posts: 17

Re: no index following

@ jm
The mainpage is indexed correctly. And the upload is about 5 weeks ago.
So that can´t be the problem.

@mary
I think i made it right
http://www.seoconsultants.com/robots-text-file/

What else can cause this problem?

#5 2006-09-28 03:49:24

Mary

Sock Enthusiast

Registered: 2004-06-27

Posts: 6,236

Re: no index following

Note what that page says:

To allow all robots complete access:

User-agent: *
Disallow:

Important Note: The above format is the common and acceptable standard for allowing all spiders access to the site. We’ve recently learned (2002-06-09), that the practice of having just a User-agent: * and Disallow: without a trailing forward slash (empty robots.txt file) may not be recommended. Some spiders may incorrectly interpret this as blocking all content.

I’m pretty sure that’s what is happening.

The good news: it’s also pointless – robots will index all content by default, unless you tell them otherwise. So, just drop the robots file and meta tags unless you want to specifically restrict indexing something. :)

#6 2006-09-28 07:41:27

Myjestic

Member

Registered: 2006-04-22

Posts: 17

Re: no index following

Ah okay,

Thanks. I will give it a try.