Go to main content

Textpattern CMS support forum

You are not logged in. Register | Login | Help

#1 2020-12-30 15:28:38

colak
Admin
From: Cyprus
Registered: 2004-11-20
Posts: 9,006
Website GitHub Mastodon Twitter

[SOLVED] prevent indexing of urls with ?

Is there a way to add <meta name="Robots" content="noindex,nofollow" /> to all urls with a ? in their structure? These include cookie consents, not just search results which we have the specific tags to detect.

I’ll rephrase this as I remembered that the ? is used more widely. Is there a way to add <meta name="Robots" content="noindex,nofollow" /> to urls with ?accept_vimeo_cookies=yes or similar for other content providers in the NeMe site?

Last edited by colak (2020-12-30 15:53:36)


Yiannis
——————————
NeMe | hblack.art | EMAP | A Sea change | Toolkit of Care
I do my best editing after I click on the submit button.

Offline

#2 2020-12-30 18:09:55

gaekwad
Server grease monkey
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 4,132
GitHub

Re: [SOLVED] prevent indexing of urls with ?

Would canonical help here?

Offline

#3 2020-12-31 06:20:25

colak
Admin
From: Cyprus
Registered: 2004-11-20
Posts: 9,006
Website GitHub Mastodon Twitter

Re: [SOLVED] prevent indexing of urls with ?

gaekwad wrote #327901:

Would canonical help here?

I do have it but I am still getting some accept_cookie urls listed in the search engines. All cookie concession links have rel="nofollow noindex noodp noydir nofollow".

I’m not sure what else I can do.


Yiannis
——————————
NeMe | hblack.art | EMAP | A Sea change | Toolkit of Care
I do my best editing after I click on the submit button.

Offline

#4 2020-12-31 07:53:59

Pat64
Plugin Author
From: France
Registered: 2005-12-12
Posts: 1,595
GitHub Twitter

Re: [SOLVED] prevent indexing of urls with ?

Yiannis. It seems you need that:

<txp:if_section name="your-section"><txp:evaluate query='contains("<txp:page_url />","?accept_vimeo_cookies=yes")'>
<meta name="Robots" content="noindex,nofollow" />
</txp:if_section></txp:evaluate>

Tested and working fine ;)


Patrick.

Github | CodePen | Codier | Simplr theme | Wait Me: a maintenance theme | [\a mi.ni.ma]: a “Low Tech” simple Blog theme.

Offline

#5 2020-12-31 09:22:37

colak
Admin
From: Cyprus
Registered: 2004-11-20
Posts: 9,006
Website GitHub Mastodon Twitter

Re: [SOLVED] prevent indexing of urls with ?

Hi Patrick, this appears to be exactly what I need. The problem is to nest it now, but I’m sure that I’ll figure it out:)


Yiannis
——————————
NeMe | hblack.art | EMAP | A Sea change | Toolkit of Care
I do my best editing after I click on the submit button.

Offline

#6 2020-12-31 09:45:54

colak
Admin
From: Cyprus
Registered: 2004-11-20
Posts: 9,006
Website GitHub Mastodon Twitter

Re: [SOLVED] prevent indexing of urls with ?

OK… I was too fast.

I thought of using this

<txp:if_individual_article>
<txp:evaluate query='contains("<txp:page_url />","?accept_vimeo_cookies=yes")'>
<meta name="Robots" content="noindex,nofollow" />
<txp:else />
<meta name="Robots" content="index,follow" />
</txp:evaluate>
</txp:if_individual_article>

but the issue is that I have more than one cookie consent.

ie I also have a couple of places with youtube videos

<txp:evaluate query='contains("<txp:page_url />","?accept_youtube_cookies=yes")'>
<meta name="Robots" content="noindex,nofollow" />
...

As such the first solution would deem the YouTube cookie acceptance page as negative and index those pages.

I use the accept_ prefix for all those urls so I’m wondering if

<txp:evaluate query='contains("<txp:page_url />","?accept_+")'>
<meta name="Robots" content="noindex,nofollow" />
...

would work or if there is another wildcard I could use for that.


Yiannis
——————————
NeMe | hblack.art | EMAP | A Sea change | Toolkit of Care
I do my best editing after I click on the submit button.

Offline

#7 2020-12-31 12:24:46

Pat64
Plugin Author
From: France
Registered: 2005-12-12
Posts: 1,595
GitHub Twitter

Re: [SOLVED] prevent indexing of urls with ?

colak wrote #327917:

<txp:evaluate query='contains("<txp:page_url />","?accept_+")'>...

I’m loving it! Tell us if it works.


Patrick.

Github | CodePen | Codier | Simplr theme | Wait Me: a maintenance theme | [\a mi.ni.ma]: a “Low Tech” simple Blog theme.

Offline

#8 2020-12-31 16:06:03

colak
Admin
From: Cyprus
Registered: 2004-11-20
Posts: 9,006
Website GitHub Mastodon Twitter

Re: [SOLVED] prevent indexing of urls with ?

Pat64 wrote #327925:

I’m loving it! Tell us if it works.

Unfortunately

<txp:evaluate query='contains("<txp:page_url />","?accept_+")'>
<meta name="Robots" content="noindex,nofollow" />
<txp:else />
<meta name="Robots" content="index,follow" />
</txp:evaluate>

and

<txp:evaluate query='contains("<txp:page_url />","?accept_","+")'>
<meta name="Robots" content="noindex,nofollow" />
<txp:else />
<meta name="Robots" content="index,follow" />
</txp:evaluate>

return <meta name="Robots" content="index,follow" /> when visiting a url ending in ?accept_vimeo_cookies=yes. :(


Yiannis
——————————
NeMe | hblack.art | EMAP | A Sea change | Toolkit of Care
I do my best editing after I click on the submit button.

Offline

#9 2020-12-31 17:22:59

Myusername
Member
Registered: 2019-12-12
Posts: 162

Re: [SOLVED] prevent indexing of urls with ?

Does something like that help?

<txp:php>
   $url = parse('<txp:page_url/>');
   $cookies = "/accept_youtube_cookies=yes|accept_fb_cookies=yes|accept_vimeo_cookies=yes/i";
   if (preg_match($cookies, $url)){
      parse('<txp:variable name="no_index_cookies" value="1"/>');
   }
</txp:php>

<txp:if_variable name="no_index_cookies" value="1">
    <meta name="Robots" content="noindex,nofollow" />
<txp:else/>
     <meta name="Robots" content="index,follow" />
</txp:if_variable>

Last edited by Myusername (2020-12-31 17:23:24)

Offline

#10 2020-12-31 20:06:20

etc
Developer
Registered: 2010-11-11
Posts: 5,028
Website GitHub

Re: [SOLVED] prevent indexing of urls with ?

Why +?

<txp:evaluate query='contains("<txp:page_url />","?accept_")'>
    <meta name="Robots" content="noindex,nofollow" />
<txp:else />
    <meta name="Robots" content="index,follow" />
</txp:evaluate>

Offline

#11 2021-01-01 05:43:21

colak
Admin
From: Cyprus
Registered: 2004-11-20
Posts: 9,006
Website GitHub Mastodon Twitter

Re: [SOLVED] prevent indexing of urls with ?

etc wrote #327953:

Why +?

Because I am and idiot!!! Thanks so much to all! Oleg’s suggestion solved it!


Yiannis
——————————
NeMe | hblack.art | EMAP | A Sea change | Toolkit of Care
I do my best editing after I click on the submit button.

Offline

#12 2021-01-03 09:36:01

Pat64
Plugin Author
From: France
Registered: 2005-12-12
Posts: 1,595
GitHub Twitter

Re: [SOLVED] prevent indexing of urls with ?

etc wrote #327953:

Why +?

<txp:evaluate query='contains("<txp:page_url />","?accept_")'>...

Oleg:

Maybe it’s simpler to do:

<txp:evaluate query='contains("<txp:page_url />","?accept_")'>
<txp:header name="HTTP/1.1 503 Service Temporarily Unavailable" />
<txp:header name="Status" value="503 Service Temporarily Unavailable" />
<txp:header name="Retry-After: 3600" /></txp:evaluate>

But unfortunately, header response in my browser isn’t correct. Can you explain to me why? Thanks.


Patrick.

Github | CodePen | Codier | Simplr theme | Wait Me: a maintenance theme | [\a mi.ni.ma]: a “Low Tech” simple Blog theme.

Offline

Board footer

Powered by FluxBB