Textpattern CMS

#1 2007-12-30 20:40:10

AndrijaM

Member

From: Belgrade, Serbia

Registered: 2007-12-22

Posts: 190

Website

Question about robots.txt file

First time I’m making robots.txt file. Something I dont understand:

this is what it looks like:

User-agent: *
Disallow: /phpMyAdmin/
Disallow: /images/
Disallow: /files/
Disallow: /textpattern/

What I dont understand is how can I exclude a single page from my txp site?

For instance, page contact I dont want to be indexed, and as i understand I cant write something like:

Disallow: /contact/

becouse it is not a folder, it is a section on my txp site.

#2 2007-12-30 21:31:22

Gocom

Developer Emeritus

From: Helsinki, Finland

Registered: 2006-07-14

Posts: 4,533

Website

Re: Question about robots.txt file

becouse it is not a folder, it is a section on my txp site.

It works by the url, so you can use it. Search engines won’t matter if it’s folder, file or dynamic-page (as it is in this case), only matter is the url.

Cheers!

---

Rah | GitHub

#3 2007-12-30 21:35:12

zero

Member

From: Lancashire

Registered: 2004-04-19

Posts: 1,470

Website

Re: Question about robots.txt file

Wikipedia has an answer

---

BB6 Band My band
Gud One My blog

#4 2007-12-31 08:12:15

colak

Admin

From: Cyprus

Registered: 2004-11-20

Posts: 9,091

Website GitHub Mastodon Twitter

Re: Question about robots.txt file

there’s one problem i found with the robots.txt. Hackers type in the url directly so as to find out which folders you are trying to protect.

>Edit typos

Last edited by colak (2007-12-31 16:46:08)

---

Yiannis
——————————
NeMe | hblack.art | EMAP | A Sea change | Toolkit of Care
I do my best editing after I click on the submit button.

#5 2007-12-31 09:36:13

AndrijaM

Member

From: Belgrade, Serbia

Registered: 2007-12-22

Posts: 190

Website

Re: Question about robots.txt file

Thanks!