NON-ASCII obfuscated since MySQL/PHP "upgrade"

jayrope · 2016-11-28 11:28:45

This affects all kinds of Textpattern versions from 4.2.0 to 4.6.2 here:

My server provider upgraded their MySQL and PHP installs two days ago, now all Textpattern installs on my account display non-ASCII characters such as Ä Ö Ü ä ö ü ç ø ~ñ etc as ? or ? in black squares.
Very annoying.
Also, some of the articles in backends don’t show their body text.
Some strange PHP errors pop up on some sites, but none consistently related to the same TXP version at least.
Databases checked via PHPMyAdmin check out good, all content is there at least, incl. non-ASCII.

Has anyone an idea of a simple cure? The server support went from level 1 to 3, but all clueless, and me, kinda, too.

Server runs PHP 5.4.45 (cpanel report) / 5.5.38 (diagnostics report) – and MySQL 10.0.28-MariaDB

Someone suggested a false setting in php.ini, related to the correct setting of mbstring.http_input …

Open for ideas!

Last edited by jayrope (2016-11-28 12:02:13)

jayrope · 2016-11-28 11:53:40

Typical diagnostics:

Textpattern version: 4.6.1 (fb9a5683991545def3c704efce499c18)
Last update: 2016-09-14 19:19:40/2016-10-08 09:17:57
Document root: /home/.../jayrope.com
$path_to_site: /home/.../jayrope.com
Textpattern path: /home/.../jayrope.com/textpattern
Article URL pattern: title_only
Temporary directory path: /home/.../jayrope.com/textpattern/tmp
Site URL: www.jayrope.com
PHP version: 5.5.38
GD Graphics Library: bundled (2.1.0 compatible); supported formats: GIF, JPG, PNG.
Server TZ: America/Chicago
Server local time: 2016-11-28 05:31:56
Daylight Saving Time enabled?: 0
Automatically adjust Daylight Saving Time setting?: 0
Time zone (GMT offset in seconds): Europe/Berlin (3600)
MySQL: 10.0.28-MariaDB
Database server time: 2016-11-28 05:31:56
Database server time offset: 1 s
Database server timezone: SYSTEM
Database session timezone: SYSTEM
Locale: en_GB.UTF-8
Server: LiteSpeed
PHP server API: litespeed
RFC 2616 headers: 
Server OS: Linux 2.6.32-673.8.1.lve1.4.3.el6.x86_64
Active plugins: zem_contact_reborn-4.5.0.0, smd_xml-0.40
Admin-side theme: hiveneutral 4.6.1

Pre-flight check: 
------------------------
New Textpattern version 4.6.2 available for download.
Clean URLs are only supported on Apache, use at your own risk.
------------------------

.htaccess file contents: 
------------------------
# BEGIN Textpattern
#DirectoryIndex index.php index.html

#Options +FollowSymLinks
#Options -Indexes
#ErrorDocument 403 default

<IfModule mod_rewrite.c>
    RewriteEngine On
    #RewriteBase /relative/web/path/

    RewriteCond %{HTTP_HOST} ^jayrope.com
	RewriteRule (.*) http://www.jayrope.com/$1 [R=301,L]

    RewriteCond %{REQUEST_FILENAME} -f [OR]
    RewriteCond %{REQUEST_FILENAME} -d
    RewriteRule ^(.+) - [PT,L]

    RewriteCond %{REQUEST_URI} !=/favicon.ico
    RewriteRule ^(.*) index.php

    RewriteCond %{HTTP:Authorization}  !^$
    RewriteRule .* - [E=REMOTE_USER:%{HTTP:Authorization}]
</IfModule>

#php_value register_globals 0

<IfModule mod_mime.c>
    AddType image/svg+xml  svg svgz
    AddEncoding gzip       svgz
</IfModule>


order allow,deny
deny from 188.143.232.*
allow from all


# END Textpattern

------------------------

Charset (default/config): latin1/utf8
character_set_client: latin1
character_set_connection: latin1
character_set_database: latin1
character_set_filesystem: binary
character_set_results: latin1
character_set_server: latin1
character_set_system: utf8
character_sets_dir: /usr/share/mysql/charsets/
17 Tables: textpattern is utf8, txp_category is utf8, txp_css is utf8, txp_discuss is utf8, txp_discuss_nonce is utf8, txp_file is utf8, txp_form is utf8, txp_image is utf8, txp_lang is utf8, txp_link is utf8, txp_log is utf8, txp_page is utf8, txp_plugin is utf8, txp_prefs is utf8, txp_section is utf8, txp_token is utf8, txp_users is utf8

10 Custom fields: custom_1, custom_2, custom_3, custom_4, custom_5, custom_6, custom_7, custom_8, custom_9, custom_10 (10)

PHP extensions: Core/5.5.38, date/5.5.38, ereg, libxml, openssl, pcre, sqlite3/0.7-dev, zlib/2.0, bz2, calendar, ctype, curl, hash/1.0, filter/0.11.0, ftp, gettext, gmp, SPL/0.2, iconv, pcntl, readline/5.5.38, Reflection/$Id: dc76d2fe0f3e9c327c1d4ca617d94e26c7fae98d $, session, standard/5.5.38, shmop, SimpleXML/0.1, mbstring, tokenizer/0.1, xml, litespeed, posix, mysql/1.0, tidy/2.0, mysqli/0.1, bcmath, imagick/3.1.2, mysqlnd/mysqlnd 5.0.11-dev - 20120503 - $Id: 15d5c781cfcad91193dceae1d2cdd127674ddb3e $, imap, dom/20031129, json/1.2.1, soap, sockets, xmlreader/0.1, PDO/1.0.4dev, xmlrpc/0.51, ffmpeg/0.7.0, xmlwriter/0.1, gd, pdo_mysql/1.0.2, exif/1.4 $Id: 74b652b3ebe78882b8006d49c17cac029fe8a5d0 $, mcrypt, pdo_sqlite/1.0.1, Phar/2.0.2, mhash, ionCube Loader

Apache modules: mod_rewrite, mod_mime, mod_headers, mod_expires, mod_auth_basic

pretext_data: array (
  'status' => '200',
  'id' => '',
  's' => '',
  'c' => '',
  'context' => '',
  'q' => '',
  'm' => '',
  'pg' => '',
  'p' => '',
  'month' => '',
  'author' => '',
  'request_uri' => '/2bf8d83a3d9702c30cc8d3b37cccd4c0/?txpcleantest=1',
  'qs' => 'txpcleantest=1',
  'subpath' => '\\/',
  'req' => '/2bf8d83a3d9702c30cc8d3b37cccd4c0/?txpcleantest=1',
)

/../index.php: 
	91b2dbb550c73fe2a7f8bad6c0c165b3
/../css.php: 
	1872823dbb091f1a3e28d37018a25917
/admin-themes/classic/classic.php: 
	45ba2ffb11e1c2c077092fb8276ecff6
/admin-themes/hive/hive.php: 
	567d1a57fd0e2d969d0d1992e188272f
/admin-themes/hiveneutral/hiveneutral.php: 
	d3db7d4ed35b487f6029d27e717d6c5d
/css.php: 
	15b5eb655ba4f162361c30ed083502d0
/include/txp_admin.php: 
	6e1b2ec212f673fae4022da9e1dee03e
/include/txp_article.php: 
	12a156ffb905daafc55bb76d16f3b3c4
/include/txp_auth.php: 
	3c53dc12daf2818a58b65d26dc995358
/include/txp_category.php: 
	b82bf6f9668fb3953786cc8ea6c72e26
/include/txp_css.php: 
	275baaa28814a719d890a895a730e78e
/include/txp_diag.php: 
	1236dbc50fcd292555e17a5357d257b0
/include/txp_discuss.php: 
	28d572167f576f7ababc3138a23fc77b
/include/txp_file.php: 
	b24d3c19ed68ab8d0ba9dcfdbf9822bf
/include/txp_form.php: 
	aa456de76a14187399fcf113b6843973
/include/txp_image.php: 
	43e37f80bce83a3cc8a4b5058328bba2
/include/txp_lang.php: 
	6e0c06bd366837d4b23bd755f7a54d9b
/include/txp_link.php: 
	6e8fa57564b1cc72181ee54b3bd9b10f
/include/txp_list.php: 
	e69e39f2dae2f81c814dc88b8a09b8b7
/include/txp_log.php: 
	6474bae0873b4b66a4ffab7b0fafe86b
/include/txp_page.php: 
	4f84cb835a5a4c0974b3359ec5ac691d
/include/txp_pane.php: 
	a6fd0eb91f9a9d2454f558ce82dd236c
/include/txp_plugin.php: 
	c10be88d56fe2daa3a297d6a5a4bc7ae
/include/txp_prefs.php: 
	6951674c8e3267b9171acf90e52a28a4
/include/txp_section.php: 
	9a9bb075ec2ae48a8898392a28f80c10
/include/txp_tag.php: 
	b7ce7f8161b8e160cc227601414171d7
/index.php: 
	3db87f1daa6c506c8e8403a40d0cbe6e
/lib/IXRClass.php: 
	a1b525b4a9ed12dcb5677d5232a3d92e
/lib/PasswordHash.php: 
	40e06a9c70b96309728e0a02822a6722
/lib/admin_config.php: 
	e81e53b7c3083ac599967072df52c49f
/lib/class.thumb.php: 
	385ecc1a5c59561db667c2490a1d45c8
/lib/class.trace.php: 
	0ee318488a6f48da87f0c0cf6fe323f9
/lib/classTextile.php: 
	bc332cd871af9aea59d1f0a20e6e47cb
/lib/constants.php: 
	be11b7315b4a9505e2d2ab0ab04c2ab7
/lib/taglib.php: 
	f1b871e7ff90028d3cd12a944b4c2153
/lib/txplib_admin.php: 
	fb66c1420b71d18416181df1f09dfcb3
/lib/txplib_db.php: 
	91704ee8946940bf0ac08764cef8ec92
/lib/txplib_forms.php: 
	c61b2af91351854c8df777d2b8097e9f
/lib/txplib_head.php: 
	c74b4a61144f6020fb513fbe2d446ce2
/lib/txplib_html.php: 
	2a6520ac0de472e03f48963570359ea2
/lib/txplib_misc.php: 
	44e673f4994eee5a65552e496ae1c578
/lib/txplib_publish.php: 
	b5707fa079d073cb8f2ba138c09c0d70
/lib/txplib_theme.php: 
	6de524114ee36d72ce424a77b92051ed
/lib/txplib_update.php: 
	477518d6ecc2ab603ffa9439df4b3720
/lib/txplib_validator.php: 
	7bf016bf3449424b84c15dfe3544e329
/lib/txplib_wrapper.php: 
	be2a8ee0e5a16a67cfb366802d5d0176
/publish.php: 
	805fd06782ecd44b3ed42fabc54a954c
/publish/atom.php: 
	9dffcb3fa3a6c7990f731e8c8073b76c
/publish/comment.php: 
	49004f3d722f096741eb685fa1817a45
/publish/log.php: 
	d99b408d6bf88b39865dab18cfaf73a2
/publish/rss.php: 
	af420738b7d5c6e1e97fdbcffa527a62
/publish/search.php: 
	3d7e0fcea6f40d31dae8c7fb208e35e6
/publish/taghandlers.php: 
	5653e0dcd45db67c5607db6d386738fb
/textpattern.js: 
	d7654c4b09d4b59f08d72a0701e91771
/update/_to_1.0.0.php: 
	038fee881209aae809666177852f0c64
/update/_to_4.0.2.php: 
	2fda6b67e48855a503d31b226739c1e8
/update/_to_4.0.3.php: 
	0ed87e575b17dc7a5c96ae68a3d16397
/update/_to_4.0.4.php: 
	bf3676db9029bcd37c2df43bef3443f0
/update/_to_4.0.5.php: 
	fa4026347c767cfbd6450cd98c9e37f6
/update/_to_4.0.6.php: 
	6e9a6eb010ba628e77fb00bfb5a0c62a
/update/_to_4.0.7.php: 
	f72fc3ee07beb2a63a56057187daf136
/update/_to_4.0.8.php: 
	0d96bca3d788b371b54e925bb3b006a0
/update/_to_4.2.0.php: 
	efcc7e8a390968de49fd8e060f5e01a4
/update/_to_4.3.0.php: 
	6a819428ad8c1952a7ce5e394abe2823
/update/_to_4.5.0.php: 
	328e41695d6fd6be1c3e18b86e91ab08
/update/_to_4.5.7.php: 
	12f67c98bc6479b54dc7f02cade86638
/update/_to_4.6.0.php: 
	993a0ce529653a6cc4de3a4c454514a8
/update/_update.php: 
	9339f4bca4bc46b0aadf8a0039142e23
/vendors/dropbox/zxcvbn/zxcvbn.js: 
	0592f04743d1d2b81b21b6c57c2e9dc7
/vendors/jquery/jquery/jquery.js: 
	4f252523d4af0b478c810c2547a63e19
/vendors/jquery/jquery-ui/jquery-ui.js: 
	0a497d4661df7b82feee14332ce0bdaf
/vendors/PrismJS/prism/prism.js: 
	9987be40687dad6b26f400b8ce7d4a54
/vendors/Netcarver/Textile/DataBag.php: 
	63e6f6033871535df2494a0253ceb691
/vendors/Netcarver/Textile/Parser.php: 
	f78b64f1082064fa92524e5f8a0682b4
/vendors/Netcarver/Textile/Tag.php: 
	a7f254b354148a092dbb7ae2f91482ee
/vendors/Textpattern/Adaptable/AdapterInterface.php: 
	de8792292a384f8a51f07d8c1e24c1e5
/vendors/Textpattern/Adaptable/Providable.php: 
	db58e36723a5e6169cdab51407ec3e61
/vendors/Textpattern/Adaptable/ProvidableInterface.php: 
	fe415c23619c580543a415c8872d2761
/vendors/Textpattern/Admin/Theme.php: 
	eede702ee31d32d67b9ed3ea58287dd1
/vendors/Textpattern/Container/Container.php: 
	8aa23f3dd76ae9515e01f92210fca272
/vendors/Textpattern/Container/ContainerInterface.php: 
	73fdfc8a970ba41953207f18e245436b
/vendors/Textpattern/Container/FactorableInterface.php: 
	7a5493a74358c0f9cb362119565c0b8a
/vendors/Textpattern/Container/FactoryInterface.php: 
	231dca8874a1e894afbf57de8b343620
/vendors/Textpattern/Container/ReusableInterface.php: 
	df4edfb5a1ae3d43f0ad257139647dbd
/vendors/Textpattern/Date/Timezone.php: 
	d83b6ea7db588599b81709f85c600b1b
/vendors/Textpattern/Filter/Exception.php: 
	9dfad03467f770dfe5fea7f4c3886cfb
/vendors/Textpattern/Filter/FilterCallable.php: 
	ff71a066bde804ec70a4ef5a32ed7d79
/vendors/Textpattern/Filter/StringFilter.php: 
	aa39802e3186f455cf46071ea4e21cf9
/vendors/Textpattern/Http/Request.php: 
	eb603f3a3aee03861cf57f59851ff5a6
/vendors/Textpattern/Iterator/FileIterator.php: 
	0b92ccce27d3dc6684914c14a645bf54
/vendors/Textpattern/L10n/Locale.php: 
	da92218be29aa7573b5768c2f6d4cf7e
/vendors/Textpattern/Loader.php: 
	5132ee7aec8a6be0e0a19984c5fb68ca
/vendors/Textpattern/Mail/Adapter/Mail.php: 
	7d09ff9ec0f6c042bc37af0521af255f
/vendors/Textpattern/Mail/AdapterInterface.php: 
	5c374a13da2b48765337daf6564608c8
/vendors/Textpattern/Mail/Compose.php: 
	c26a947697ba18e05aa6dcab94baee5d
/vendors/Textpattern/Mail/Encode.php: 
	0dc0530c9033491519a097130402c711
/vendors/Textpattern/Mail/Exception.php: 
	ce0c04106b33126de84b777ba80c9950
/vendors/Textpattern/Mail/Message.php: 
	e35c47b7bf47cbcbd9f96864941b3689
/vendors/Textpattern/Password/Adapter/PasswordHash.php: 
	53d6862f0d1dada12218ab3aace85162
/vendors/Textpattern/Password/AdapterInterface.php: 
	a23c46382eff2c784f5e549eb17160b4
/vendors/Textpattern/Password/Generator.php: 
	0b2697558cda93525de4bf4671e49453
/vendors/Textpattern/Password/Hash.php: 
	7404c7e10f304be5089b6607c235a602
/vendors/Textpattern/Password/Random.php: 
	e2d21125ea9fc2567157080a137c9473
/vendors/Textpattern/Search/Filter.php: 
	8b4ef3fd8511467ee5f6026c748e259c
/vendors/Textpattern/Search/Method.php: 
	9de58b5033c9c0b61bef5d5f92bef75a
/vendors/Textpattern/Security/Exception.php: 
	1271a06d808208d3fc208cbe675555a0
/vendors/Textpattern/Security/Filter.php: 
	5a15e72dd32ab2f616c219742fbdee7c
/vendors/Textpattern/Server/Config.php: 
	33ae325df3a4c5788908ad49bab86c04
/vendors/Textpattern/Tag/Registry.php: 
	9bd60f0c914e97f5ec894f46437adda1
/vendors/Textpattern/Tag/Syntax/Authors.php: 
	c29c9244f8a1009483c634d78df01bf5
/vendors/Textpattern/Tag/Syntax/File.php: 
	705562c46df238e113a3c29a2f1cb3c1
/vendors/Textpattern/Tag/Syntax/Image.php: 
	d8917b9156ec10930bacdd209ae93bc7
/vendors/Textpattern/Tag/Syntax/Link.php: 
	80f8b01a48682d20649b739a574ba690
/vendors/Textpattern/Tag/Syntax/Partial.php: 
	657fa060189b64a08e2cbd241b759b77
/vendors/Textpattern/Textfilter/Base.php: 
	0496350b655278e555c139d6a00d559d
/vendors/Textpattern/Textfilter/Constraint.php: 
	03b065d525667b360b31a346e539c926
/vendors/Textpattern/Textfilter/Nl2Br.php: 
	00c2e5eb5bf332688a9c50f4e5ed73f1
/vendors/Textpattern/Textfilter/Plain.php: 
	e3c7f63b9a1940cc798855b9967f23de
/vendors/Textpattern/Textfilter/Registry.php: 
	4170a43b7973f1120de914a3fc0576fb
/vendors/Textpattern/Textfilter/TextfilterInterface.php: 
	30811632e0c2f491726995e95c579087
/vendors/Textpattern/Textfilter/Textile.php: 
	082ebbdb2f7a49fdb9765c4c6d2f5ef9
/vendors/Textpattern/Textile/Parser.php: 
	9d949632792c6b4d06989d9e2850744f
/vendors/Textpattern/Textpack/Parser.php: 
	2a1f718b9b11e97086161060c8e76b17
/vendors/Textpattern/Textpack/String/Iterator.php: 
	135cf5d86e191bae3139703c51cdc217
/vendors/Textpattern/Textpack/StringInterface.php: 
	b6cc517b7ec5f0e1995916878d5ee358
/vendors/Textpattern/Type/StringType.php: 
	c322366173b359c6d14223eee5b425bc
/vendors/Textpattern/Type/TypeCallable.php: 
	af1e2c7330803292e81aec97a198a373
/vendors/Textpattern/Type/TypeInterface.php: 
	b1b40316a4aaa7fc99f43078f215ab16
/vendors/Textpattern/Validator/BlankConstraint.php: 
	fa03c1d5576cbb03d62113e84617016b
/vendors/Textpattern/Validator/CategoryConstraint.php: 
	7983c8612d3289f20ee02c07657102e3
/vendors/Textpattern/Validator/ChoiceConstraint.php: 
	3308767c97cfb14f608a01d7583c79a6
/vendors/Textpattern/Validator/Constraint.php: 
	ac476eae8cd4acbf30b1f1ed80f59240
/vendors/Textpattern/Validator/FalseConstraint.php: 
	fa81622e336dc47e5706c860983fe79a
/vendors/Textpattern/Validator/FormConstraint.php: 
	8f5c5ececcc8b9b45ab9d237f1d1ba60
/vendors/Textpattern/Validator/SectionConstraint.php: 
	18a6828594b47798bf7db56f9e01c40c
/vendors/Textpattern/Validator/TrueConstraint.php: 
	415c2ec0adc2a37b0c9bfb82af66b000
/vendors/Textpattern/Validator/Validator.php: 
	ff06c7f8ccdc75c195c8cc38c624ea0f
/vendors/Txp.php: 
	3cbca9376577717d0a4cfc80903013ff

------------------------

Bloke · 2016-11-28 14:22:51

Just out of curiosity (and this is a long shot), what happens if you alter lib/txplib_db.php to reflect this bugfix. You can apply that one-line change to any of the Txp versions you mentioned (though the line numbers will of course differ).

Does it change anything?

jayrope · 2016-11-28 14:46:58

Hi Bloke! No it doesn’t change anything.

Actually the server provider now wants me to change all mysql collations to latin1_swedish_ci … i have no idea if that makes sense and actually wonder, why a server provider does updates deviating from the path to utf8 at all?

Bloke · 2016-11-28 14:57:48

jayrope wrote #303040:

No it doesn’t change anything.

Okay. Worth a shot.

the server provider now wants me to change all mysql collations to latin1_swedish_ci

Whaaaat?! That’ll mash your carefully-crafted UTF-8 content, surely? Unless when they upgraded things, they automatically changed the collation of all your data in the process? A backup would be prudent, then compare a few rows from that backup to one of your previous backups to see if anything’s been altered.

Normally this kind of thing manifests itself when the collation/character set of the DB connection, DB, and table(s) disagree somewhere along the line. Unfortunately it’s not my area of expertise so I can’t advise a fix offhand. Hopefully someone else here can.

EDIT: if your config.php uses utf8mb4, what happens if you switch it to utf8?

Last edited by Bloke (2016-11-28 15:00:36)

jayrope · 2016-11-28 15:24:09

It seems they upgraded and introduced disagreement, yes. They seem to think, that they’d cause more trouble presumably to the other 900 websites on that shared hosting server, if they convert to utf8 all the way. In Chicago maybe there is a lack of conciousness sometimes for international language issues, weirdly among a bunch of highly-skilled Russians there running that thing.
I am a bit shocked of those guys. Maybe have to look for another server host.

jayrope · 2016-11-28 15:49:27

When i look at the databases in PHPMyAdmin i notice, that table collation is utf8_general_ci, but database collatino is latin1_swedish_ci … doesn’t that introduce more troubles? Also the server support started converting my database table collation to latin1, not mentioning the risk of data loss …

gaekwad · 2016-11-28 15:54:25

If your hosting includes backups, you could request a copy of the most recent one before the upgrade so you at least have known-good data to deploy on another hosting company.

Bloke · 2016-11-28 15:56:43

jayrope wrote #303043:

table collation is utf8_general_ci, but database collatino is latin1_swedish_ci … doesn’t that introduce more troubles?

I would expect so. You may get spurious admin-side errors like ‘this is not a safe operation’ or the white screen of death, which (if you delve into log files or put your site in debugging mode) often reveals Illegal mix of collations.

Also the server support started converting my database table collation to latin1, not mentioning the risk of data loss …

Oh, boy…

jayrope · 2016-11-28 16:29:15

Ah Bloke. Thanx for the compassion. Would you or someone else have a recommendation for a full service server host in Europe, where i can hold unlimited everything and fiddle with intestines for around a 100-200 $ € or anything else per year?

jakob · 2016-11-28 16:47:51

In Germany, you might want to try Ueberspace if you want to play with the innards a bit. They give you plenty of time to try it out too before committing.

If you want a reliable and affordable shared host, also in Germany (Sachsen), where they take care of the setup, you could try all-inkl.com. They’ve served my txp needs for years, have capable and knowledgeable staff that you can get on the telephone without long waiting times, and when they do a server update (they are presently phasing out php 5.4 and below), they give you ample notice (presently coming up to 6 weeks) and point you to all the instances on your server that you need to check up on (even special php directives hidden in htaccess files). Likewise, they offer 90 days for you to try out before your commit yourself. I have no problems with UTF-8 encoded sites, and you can access databases remotely too (e.g. with sequel pro if you’re on a mac). Regular MySQL at present, i.e. no MariaDB as yet. PHP7 and Let’s Encrypt certificates are also available. SSH is available on the larger plans too. It’s a typical LAMP hoster, so no ruby or node.

BTW: I vaguely remember that Ruud had a script somewhere for converting between latin and utf8 encoding somewhere on the forum. It’s quite old (i.e. from the times of mysql 4 upgrades to v5) but maybe it helps. Do a search on the forum…

jayrope · 2016-11-28 16:53:00

Jakob, gaekwad, danke, super Tips.

Textpattern CMS

Textpattern CMS support forum

#1 2016-11-28 11:28:45