[SOLVED] prevent indexing of urls with ?

colak · 2020-12-30 15:28:38

Is there a way to add <meta name="Robots" content="noindex,nofollow" /> to all urls with a ? in their structure? These include cookie consents, not just search results which we have the specific tags to detect.

I’ll rephrase this as I remembered that the ? is used more widely. Is there a way to add <meta name="Robots" content="noindex,nofollow" /> to urls with ?accept_vimeo_cookies=yes or similar for other content providers in the NeMe site?

Last edited by colak (2020-12-30 15:53:36)

gaekwad · 2020-12-30 18:09:55

Would canonical help here?

colak · 2020-12-31 06:20:25

gaekwad wrote #327901:

Would canonical help here?

I do have it but I am still getting some accept_cookie urls listed in the search engines. All cookie concession links have rel="nofollow noindex noodp noydir nofollow".

I’m not sure what else I can do.

Pat64 · 2020-12-31 07:53:59

Yiannis. It seems you need that:

<txp:if_section name="your-section"><txp:evaluate query='contains("<txp:page_url />","?accept_vimeo_cookies=yes")'>
<meta name="Robots" content="noindex,nofollow" />
</txp:if_section></txp:evaluate>

Tested and working fine ;)

colak · 2020-12-31 09:22:37

Hi Patrick, this appears to be exactly what I need. The problem is to nest it now, but I’m sure that I’ll figure it out:)

colak · 2020-12-31 09:45:54

OK… I was too fast.

I thought of using this

<txp:if_individual_article>
<txp:evaluate query='contains("<txp:page_url />","?accept_vimeo_cookies=yes")'>
<meta name="Robots" content="noindex,nofollow" />
<txp:else />
<meta name="Robots" content="index,follow" />
</txp:evaluate>
</txp:if_individual_article>

but the issue is that I have more than one cookie consent.

ie I also have a couple of places with youtube videos

<txp:evaluate query='contains("<txp:page_url />","?accept_youtube_cookies=yes")'>
<meta name="Robots" content="noindex,nofollow" />
...

As such the first solution would deem the YouTube cookie acceptance page as negative and index those pages.

I use the accept_ prefix for all those urls so I’m wondering if

<txp:evaluate query='contains("<txp:page_url />","?accept_+")'>
<meta name="Robots" content="noindex,nofollow" />
...

would work or if there is another wildcard I could use for that.

Pat64 · 2020-12-31 12:24:46

colak wrote #327917:

<txp:evaluate query='contains("<txp:page_url />","?accept_+")'>...

I’m loving it! Tell us if it works.

colak · 2020-12-31 16:06:03

Pat64 wrote #327925:

I’m loving it! Tell us if it works.

Unfortunately

<txp:evaluate query='contains("<txp:page_url />","?accept_+")'>
<meta name="Robots" content="noindex,nofollow" />
<txp:else />
<meta name="Robots" content="index,follow" />
</txp:evaluate>

and

<txp:evaluate query='contains("<txp:page_url />","?accept_","+")'>
<meta name="Robots" content="noindex,nofollow" />
<txp:else />
<meta name="Robots" content="index,follow" />
</txp:evaluate>

return <meta name="Robots" content="index,follow" /> when visiting a url ending in ?accept_vimeo_cookies=yes. :(

Myusername · 2020-12-31 17:22:59

Does something like that help?

<txp:php>
   $url = parse('<txp:page_url/>');
   $cookies = "/accept_youtube_cookies=yes|accept_fb_cookies=yes|accept_vimeo_cookies=yes/i";
   if (preg_match($cookies, $url)){
      parse('<txp:variable name="no_index_cookies" value="1"/>');
   }
</txp:php>

<txp:if_variable name="no_index_cookies" value="1">
    <meta name="Robots" content="noindex,nofollow" />
<txp:else/>
     <meta name="Robots" content="index,follow" />
</txp:if_variable>

Last edited by Myusername (2020-12-31 17:23:24)

etc · 2020-12-31 20:06:20

Why +?

<txp:evaluate query='contains("<txp:page_url />","?accept_")'>
    <meta name="Robots" content="noindex,nofollow" />
<txp:else />
    <meta name="Robots" content="index,follow" />
</txp:evaluate>

colak · 2021-01-01 05:43:21

etc wrote #327953:

Why +?

Because I am and idiot!!! Thanks so much to all! Oleg’s suggestion solved it!

Pat64 · 2021-01-03 09:36:01

etc wrote #327953:

Why +?

<txp:evaluate query='contains("<txp:page_url />","?accept_")'>...

Oleg:

Maybe it’s simpler to do:

<txp:evaluate query='contains("<txp:page_url />","?accept_")'>
<txp:header name="HTTP/1.1 503 Service Temporarily Unavailable" />
<txp:header name="Status" value="503 Service Temporarily Unavailable" />
<txp:header name="Retry-After: 3600" /></txp:evaluate>

But unfortunately, header response in my browser isn’t correct. Can you explain to me why? Thanks.

Textpattern CMS

Textpattern CMS support forum

#1 2020-12-30 15:28:38