how TXP parses templates & related stuff (URL queries, security, etc)

Gocom · 2011-12-15 14:10:02

maniqui explains:

Important: this conversation started in smd_wrap thread.
I moved it here to avoid hijacking the plugin’s thread. Gocom joined there to share his deep knowledge on TXP & security.

————

maniqui wrote:

Aaaah… I hope one day I’ll learn how (in which order) all the TXP tag parsing really works, and get to see how a page template or form really looks like after it has been parsed and re-parsed and re-re-parsed by TXP…

Bloke wrote:

note how big the output is, in characters

Lengths can collide very easily, where md5() hashes can not (as easily). Not to good idea to compare resulting lengths, if by lengths you really meant length. In addition to md5 (which you could use to cache the results) is to compare the strings as it is. I.e.

preg_replace('@(</?txp:\w+(?:\s+\w+\s*=\s*(?:"(?:[^"]|"")*"|\'(?:[^\']|\'\')*\'|[^\s\'"/>]+))*\s*/?'.chr(62).')@s', '', $thing) === parse($thing)

$thing would be the content which you have already parsed with your curly tag parser.

{replacements} (or their <txp:smd_some_data /> counterparts

Is there way to turn curly tags off, so that your parser ignores them completely? You could actually probably do that for the user with simple strpos($content, '{') === false in addition to setting setting it with a tag attribute.

maniqui wrote:

I hope one day I’ll learn how (in which order) all the TXP tag parsing really works

Parser “basically” replaces tags with PHP’s function calls, one tag level at time. One parser call replaces one level of <txp:tags />, to the first container’s tags wrapped content. I.e.

<txp:tag foo="bar" />
<txp:tag>
	<txp:tag />
</txp:tag>

Becomes:

tag( (array) $atts, (string) $thing = NULL)
tag( (array) $atts, (string) $thing = NULL)

The $atts contains the tag attributes as an associative array (‘foo’ => ‘bar’), where values (bar) surrounded by single quotes are already parsed (in the same manner). Second tag’s (container’s) contents will stay as tags, and it’s the tag()’s responsibility to invoke the parser to parse the wrapped tags.

after it has been parsed and re-parsed and re-re-parsed by TXP…

Re-parsing after the function is evaluated would just parse the returned contents, I suppose. For example following would give you an error page:

parse(parse(
	'<txp:variable name="404" value="<txp:txp_die />" />'.
	'<txp:variable name="404" />'
));

And not that <txp:txp_die /> as a plain text as suggested by the double quotes. On the first pass it does return the content, but on the second, it gets parsed.

Textpattern does actually do two passes with the parser, so if you were to place same type of variable snippet directly to your page template, you should get the same results. Which is one thing I’m not that fond in Textpattern, second being that you can’t map functions or methods as tags, but every function is potential tag.

One of Textpattern’s documented features, second pass, is the reason why I mentioned as long as parser is secure to Bloke. Tags are harmless as long as you can’t inject anything as an attribute, or make an tag to return tags otherwise. If you can, once again you can execute any code you want. For example consider this:

<txp:variable name="date" value='<txp:php> echo gps("date"); </txp:php>' />
<txp:variable name="date" />

There is nothing wrong with it when it’s placed directly to page template, right? It just does return date or something else that is requested, right? There is no possibility it executing server side code, right?

http://example.com?date=<txp:txp_die />

Textpattern’s second pass. If date contains code, it will get executed once <txp:variable name="date" /> is called. And the code can be anything, from PHP to tags.

Even if you just pass something to tag structure, like Bloke experiments with replacing curly tags with <txp:tags />, he will still need to escape the returned values as Textpattern has the second pass feature which powers custom_fields in addition to other stuff.

I just hope no one populates variables with URL data, as that code will get executed if you return it and don’t escape it first.

Last edited by maniqui (2011-12-15 23:00:21)

maniqui · 2011-12-15 14:33:13

Gocom, thanks for sharing your knowledge on the magical inner workings of TXP.
I’ll have to parse, re-parse, and re-re-parse your post a few times until I fully get it :)

I just hope no one populates variables with URL data, as that code will get executed if you return it and don’t escape it first.

Without trying to hijack this plugin thread (can we use our moderation powers to split/move this discussion somewhere else?), I think everyone here non-PHP/security savvy will appreciate some extra insights on the above statement.
This practice (populating variable with URL data) may be more common practice than you think.
In fact, isn’t that exactly what plugins like adi_gps do? Pick up some URL/GET/POST data and convert it in (txp) variables?

Gocom · 2011-12-15 15:52:35

maniqui wrote:

In fact, isn’t that exactly what plugins like adi_gps do? Pick up some URL/GET/POST data and convert it in (txp) variables?

Yes. And, for example, adi_gps allows code injection if escape="html" isn’t set (which by default it isn’t) and you return the variables. For example doing this would be very bad:

<txp:adi_gps name="hello" />
<txp:variable name="hello" />

With hello parameter placed in the URL attacker can do anything ?hello=<txp:php> /* bad code here* /</txp:php>, from deleting all content from the database, to stealing passwords.

But then this wouldn’t be as risky:

<txp:adi_gps name="hello" />
<txp:if_variable name="hello" value="hi">
	Hello World!
</txp:if_variable>

As we are not returning the variable. But if we return it, it will get parsed as Textpattern passes content with parser twice. Meaning that tags returned by single tags will get parsed too.

Now talking specifically about adi_gps, there are various things about the beast;

Never use it without name set. If you use it without name attribute, at least set global to 0. If you don’t, your site can be compromised as attacker can use adi_gps to overwrite all global variables, i.e. Textpattern’s preferences.
Always set escape="html" if you are going to return the values as is and you don’t plan doing escaping by yourself. Otherwise you have code injection hole on your site which can be used to remotely execute PHP code (as shown above – this doesn’t only apply to adi_gps but all tags that return content directly from URLs).
Also keep in mind that adi_gps will overwrite existing TXP’s variables and PHP’s global variables. If you haven’t set global="0" then be careful to not to overwrite any important globals, including prefs, pretext i.e. This is why you will have to use name.
To sum it up; Never use adi_gps tag without any attributes. Using the tag in non-attribute format compromises your site — big time.

Last edited by Gocom (2011-12-15 15:58:57)

maniqui · 2011-12-15 22:52:15

Again: this conversation started in smd_wrap thread.
I moved it here to avoid hijacking the plugin’s thread. Gocom joined there to share his deep knowledge on TXP & security.

I thought this very post you are reading was going to appear as the opening post of this thread, but then, when I moved my posts & Gocom’s posts (the posts above this one), those were listed first (because of its date, probably), ups! :)

Gocom · 2011-12-16 00:18:40

About adi_gps. It may sound that I was trash talking about the plugin. Don’t worry I’m not. I actually like the way it works, populating variables and all that. It’s very clever concept. Strange thing is that I haven’t used it before (as in, really used in a real project), this was the first time I’ve looked at its source code.

Anyways, I’ve contacted and talked with Adi today, and just submitted a patch (for the worst or better). Unfortunately it changes the plugins behavior, but it should make the default configuration more securer among some small additions.

Last edited by Gocom (2011-12-16 00:31:27)

Gocom · 2011-12-16 00:50:11

Uh, and as the topic says, about the parsing itself. If there was a correct time for it, second pass should (in my honest opinion) be removed. I remember the days when the old developer team wondered why there even was a such thing. That second parser pass can have dangerous effects, way more than just XSS’ing HTML somewhere.

Wondering how many things would break if it was removed. Singular article tags at least, I think.

Last edited by Gocom (2011-12-16 00:53:37)

Textpattern CMS

Textpattern CMS support forum

#1 2011-12-15 14:10:02

how TXP parses templates & related stuff (URL queries, security, etc)

#2 2011-12-15 14:33:13

Re: how TXP parses templates & related stuff (URL queries, security, etc)

#3 2011-12-15 15:52:35

Re: how TXP parses templates & related stuff (URL queries, security, etc)

#4 2011-12-15 22:52:15

Re: how TXP parses templates & related stuff (URL queries, security, etc)

#5 2011-12-16 00:18:40

Re: how TXP parses templates & related stuff (URL queries, security, etc)

#6 2011-12-16 00:50:11

Re: how TXP parses templates & related stuff (URL queries, security, etc)

Board footer