Garbage generated by doSlash() function

zem · 2006-06-26 22:40:48

Also, what are the possible knock-on effects in doing this?

Potentially: security holes, failed queries.

In practice: hard to say without testing, or at least a more certain diagnosis.

Once more, the problem isn’t with doSlash. My best guess is that PHP is automatically adding slashes; but Textpattern doesn’t know this, because get_magic_quotes_gpc() is returning an incorrect value, and so adds a second layer of slashes. (If get_magic_quotes_gpc is not the problem, the gist of it is the same: something else is automatically adding extra slashes). The solution might be as simple as turning off magic quotes, or it might not.

I’d suggest talking to your hosting company. Textpattern needs a reliable, predictable environment in which to run. That appears not to be the case here.

(FWIW, if the problem is what I think it is, you’ll almost certainly have problems with any modern PHP software)

In case it helps at all, I have created a user on the site should you wish to see the problem in action. You can put php code into the content if you want to test anything.

You are aware this is a public forum, right? You should remove that asap.

Last edited by zem (2006-06-26 22:42:31)

zem · 2006-06-26 22:47:37

There’s a sample script here you can use to test whether get_magic_quotes_gpc is working.

daveh · 2006-06-27 18:49:03

There’s a sample script here you can use to test whether get_magic_quotes_gpc is working.

I ran the test.php page (from the bug report) and got the expected result of \”test\”

Yeah, that’s the one I ran.

Once more, the problem isn’t with doSlash.

Yeah, I realise it’s only a wrapper for mysql_real_escape_string. Maybe this is the problem? (which is possibly where the \r\n comes from. Presumably the get_magic_quotes_gpc setting has no affect on this combined PHP/mySql function?)

I’d suggest talking to your hosting company.

I would but I don’t really know what to ask them yet. I’d like to get to the root of the problem first.

(FWIW, if the problem is what I think it is, you’ll almost certainly have problems with any modern PHP software)

I’m installing WordPress as an alternative to TextPattern to see if I get on any better. (I need a site up and functioning by the middle of August).

You are aware this is a public forum, right? You should remove that asap.

The only thing on the website at the moment is TextPattern. If it won’t work then it’ll be deleted, so no immediate security issues.

Many thanks,
Dave

daveh · 2006-06-27 19:06:23

I’ve put the following test file on the site…

<a href=“http://www.tabularasaarts.com/magic.php” target=”_blank”>http://www.tabularasaarts.com/magic.php</a>

This test file gives the same results on both the above URL and my local PC.

Where in TextPattern does it remove the \r\n stuff that mysql_real_escape_string puts in?
(ie. how would I convert it back to an unescaped value?)

Ta,
Dave

Last edited by daveh (2006-06-27 19:09:51)

zem · 2006-06-27 22:58:10

Where in TextPattern does it remove the \r\n stuff that mysql_real_escape_string puts in?

It doesn’t. MySQL does. mysql_real_escape_string is used because without it quotes and other characters would cause SQL errors, and open up security holes.

Again, the problem is not with that part of the code. The problem is that PHP is automatically adding extra slashes first. Textpattern includes code to detect this and remove them. That detection code appears to be failing, or something else is adding them back in, for reasons that are specific to your server or installation — you’re the only one with this problem.

You should ask your hosting company how to turn off the PHP “magic_quotes_gpc” setting, and refer them to the PHP bug report.

FYI, Textpattern handles magic quotes using the method recommended in the PHP manual (see the gps() function in txplib_misc.php).

Last edited by zem (2006-06-27 23:25:29)

daveh · 2006-06-27 23:51:19

Zem, I’m losing the will to live here ;-)

I know (and almost understand) what you’re saying about the magic_quotes_gpc setting, but I can’t see how that relates to my problem.

On the test file I put up <a href=“http://www.tabularasaarts.com/magic.php”>here</a> it mirrors exactly the problem I’m having with TP, and the problem isn’t being caused (seemingly) by the magic_quotes_gqc setting, it is being caused by the mysql_real_escape_string function.
<br />
<blockquote>What about ‘this string’.

And “on a new line” here.
</blockquote>
If you type (can’t copy/paste as this forum has converted the quotes) the text above (or any with multiple lines) into my test page the output created by magic_quotes_gpc being on is as I would expect it, with the ‘ and “ being escaped, but with the carriage returns remaining intact.

However, the value being returned by the mysql_real_escape_string is exactly the problem I have, in that the carriage returns are being escaped as \r\n. (I understand that it needs to do the conversion in order to safely store this in the DB, but that’s not what I want pulled back out of the DB again.)

Which, in turn, is what TP is pulling back out of the database and putting into the default page content textarea instead of first convering the \r\n back into the appropriate carriage returns.

Am I on completely the wrong track here? (Are you saying that the removal of the \r\n is done automatically based on the magic_quotes_gpc setting?)

(I can post the source for my magic.php test page if you were able to run it to see what results you get returned on your local machine.)

Cheers,
Dave

daveh · 2006-06-28 00:15:10

OK, possibly ignore the above :-)

Using the following sql (using the converted slashed text from the mysql_real_escape_string on my test page)…

<blockquote>select “What about \‘this string\’.\r\n\r\nAnd \“on a new line\” here.” as test</blockquote>

I get on my local PC (with the \r\n nicely and automatically converted back as expected)…

<blockquote>test ——————————————————————————— What about ‘this string’.

And “on a new line” here.

1 record(s) selected [Fetch MetaData: 0/ms] [Fetch Data: 0/ms] </blockquote>

While on the hosting sevice mySQL I get…

<blockquote>>[Error] Script lines: 1-1 ————————————— You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘on a new line\” here.” as test’ at line 1

Warnings: —-> W (1): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘on a new line\” here.” as test’ at line 1 <—- [Executed: 28/06/06 01:03:01 BST ] [Execution: 0/ms] </blockquote>

So it looks like it’s a mySQL problem.

Now then, any idea what it could be? :-)

zem · 2006-06-28 00:38:52

Aha! Well spotted, thanks. Sounds like it could be related to this – the MySQL ‘NO_BACKSLASH_ESCAPES’ setting, used as a hack to fix a security bug without upgrading to the latest version of MySQL.

mysql_real_escape_string() (and hence Textpattern) should work correctly regardless of that setting, but it appears there were bugs in some versions of the MySQL client libraries. Looks like that could be the case on your server.

daveh · 2006-06-28 17:48:18

Zem, superb!

From Brinkster…
<quote> sql_mode: NO_BACKSLASH_ESCAPES,STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
</quote>
And my local one…
<quote>
sql_mode: STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
</quote>
<br />Thanks for all your help, I’m pretty sure this is what’s causing the effect. It doesn’t solve my problem but at least I know what is wrong now.

Dave

daveh · 2006-06-28 18:15:32

OK, did some testing on Brinkster…

select @@local.sql_mode

The above gives you the value of the sql_mode variable (containing the no_backslash_escapes setting).

But if you do this…
<blockquote>
set sql_mode = replace(local.sql_mode, ‘NO_BACKSLASH_ESCAPES,’, ‘’)

select “What about \‘this string\’.\r\n\r\nAnd \“on a new line\” here.” as test
</blockquote>

…then Brinkster returns the proper string (without giving the error I quoted earlier). However, as the ‘unfix’ above is only valid for the current connection/session it would have to be done every time data is retrieved from the DB. I don’t know if, from PHP, you can run a double query (combining the setting change and the necessary SELECT statement) in one go (I know it can be done in ASP with MS SQLserver).

Sooooo, could there possibly be a patch written for TextPattern that checks for the inclusion of no_backslash_escapes and, if present, switches it off again?
This seems a possibly easier approach that writing a custom unslash function.

ALSO: If someone reading this also has an account on the WordPress blog that Zem gave a link to, could you possibly post a link on there to this blog. Maybe it will help someone over there.

(Hopefully Brinkster will upgrade and fix this soon but maybe TextPattern needs a permanent check for this situation built into its code.)

Dave

zem · 2006-06-28 23:01:47

Sooooo, could there possibly be a patch written for TextPattern that checks for the inclusion of no_backslash_escapes and, if present, switches it off again?

That would probably do more harm than good. This is only a problem on certain buggy versions of MySQL (on specific platforms only, if the MySQL release notes are accurate).

You could add a mysql_query() to your copy of txplib_db.php. See the ‘SET NAMES’ query for an example.

daveh · 2006-06-28 23:19:59

Zem, I’ll look into putting something together in txplib_db.php.

<blockquote>That would probably do more harm than good. This is only a problem on certain buggy versions of MySQL (on specific platforms only, if the MySQL release notes are accurate).</blockquote>

Surely this is now a problem in all future versions of mySQL. As the ability now exists to switch off the escaping in mySQL then TextPattern (and other apps) will need to check for this and work around it – in the same way the checks and appropriate changes are made to account for the magic_quotes_gpc setting.

Anyway, Brinkster are not being helpful…. “it will be fixed when and if we get around to it”. There’s a lot of unhappy Brinkster people out there!

Now I have to go to bed, it’s tomorrow already! :-)

Cheers,
Dave

Mary · 2006-06-29 05:25:02

Surely this is now a problem in all future versions of mySQL. As the ability now exists to switch off the escaping in mySQL then TextPattern (and other apps) will need to check for this and work around it – in the same way the checks and appropriate changes are made to account for the magic_quotes_gpc setting.

magic quotes are a PHP setting, easily detected and compensated for. It was invented, originally, as a feature for beginners that try screwing around with PHP but don’t know what they’re doing.

no_backslash_escape makes all escape characters useless, not just slashes. It’s a setting intended to prevent sql injection in buggy MySQL versions, meant as an interim protection while you upgrade, not something you turn on for the heck of it. If someone turns it on, it’s their responsibility to upgrade, not expect everyone else to jump through hoops for them while they sit on their laurels. It’s not a good idea to let hosts think they do that.

WordPress users on Brinkster are having the same trouble.

zem · 2006-06-29 06:08:58

Surely this is now a problem in all future versions of mySQL.

No, it’s fixed in 5.0.22, according to the MySQL release notes. Textpattern is handling slashes and mysql escaping in absolutely the correct way. The problem is that, in a few specific MySQL releases on certain platforms, the mysql_real_escape_string() function ignores the NO_BACKSLASH_ESCAPES setting.

daveh · 2006-06-29 18:42:35

zem wrote:

<blockquote>Surely this is now a problem in all future versions of mySQL.

No, it’s fixed in 5.0.22, according to the MySQL release notes. Textpattern is handling slashes and mysql escaping in absolutely the correct way. The problem is that, in a few specific MySQL releases on certain platforms, the mysql_real_escape_string() function ignores the NO_BACKSLASH_ESCAPES setting.
</blockquote>
Sorry Zem, I didn’t make myself clear. What I meant was, now that the setting exists (for all future versions of mySQL), hosts have the ability of switching it on and the possibility (and hence the problem) remains of TextPattern (and other apps of course) not working because of it.

Mary, surely this is the developers responsibility too not just the hosting services. At the very least TextPattern (and others) can do a very simple check during setup to warn users that this setting is enabled on the specified server and that TextPattern will not work while it is on.

While we wait for the ‘other person’ to fix it, TextPattern remains unusable for me as a potential user.

Cheers,
Dave.

(I’ll post a ‘quick fix’ if I manage to create one for other sufferers.)

Textpattern CMS

Textpattern CMS support forum

#16 2006-06-26 22:40:48

Re: Garbage generated by doSlash() function

#17 2006-06-26 22:47:37

Re: Garbage generated by doSlash() function

#18 2006-06-27 18:49:03

Re: Garbage generated by doSlash() function

#19 2006-06-27 19:06:23

Re: Garbage generated by doSlash() function

#20 2006-06-27 22:58:10

Re: Garbage generated by doSlash() function

#21 2006-06-27 23:51:19

Re: Garbage generated by doSlash() function

#22 2006-06-28 00:15:10

Re: Garbage generated by doSlash() function

#23 2006-06-28 00:38:52

Re: Garbage generated by doSlash() function

#24 2006-06-28 17:48:18

Re: Garbage generated by doSlash() function

#25 2006-06-28 18:15:32

Re: Garbage generated by doSlash() function

#26 2006-06-28 23:01:47

Re: Garbage generated by doSlash() function

#27 2006-06-28 23:19:59

Re: Garbage generated by doSlash() function

#28 2006-06-29 05:25:02

Re: Garbage generated by doSlash() function

#29 2006-06-29 06:08:58

Re: Garbage generated by doSlash() function

#30 2006-06-29 18:42:35

Re: Garbage generated by doSlash() function

Board footer