many problems after server update by the hosting provider

GugUser · 2017-11-23 16:49:25

After a server update one of my customer’s websites (it was still running with Textpattern 4.5.7) showed only question marks instead of umlauts. Another problem was that all text contents containing an umlaut were shown empty in the backend. However, the contents are available in the database (Server-Version: 10.1.24-MariaDB-cll-lve – MariaDB Server).

The umlauts of the texts in the database are written as such, for example, “Mädchen”. The only word with umlaut that is displayed correctly on the website is stored in the database like this “KÃ¼nstler”, is displayed as “Künstler”. I have no idea why this umlaut was coded differently.

At first, I made an upgrade to Textpattern 4.6.2, but the problem persisted. All plugins that I updated were marked as modified, what I’ve never seen before in this context.

Some tables in the database had latin1 character set, so I run the rvm_latin1_to_utf8 plugin. Now all tables are uft8_general_ci but nothing changed , everything stayed the same.

I discussed all this with the hosting provider support (www.serverprofis.de) and I made a reference to this forum post, but they don’t want nothing to do with it.

I’ve been working on it for several hours now and have no idea what else to do.

bici · 2017-11-23 18:43:43

i think I would try installing the whole shebang in another server as a test to see if the problem goes away.

When they updated the server which version of php is now the default?

And finally is there a way to make the DB truly utf8. i.e go from showing special characters to actually show what it should be: ie NOT this “KÃ¼nstler” but this “Künstler”.

jakob · 2017-11-23 19:13:48

I’ve struggled with this in the past with very old database from the mysql 4 age. If all else fails, you can export your db to SQL, open that in a text editor, set you file’s character encoding to what you need (a lot of text editors let you do that) and then search and replace the wrong characters, re-save and re-import with the correct collation.

It’s not great fun but in German, there are not too many special characters to search for – äöüßÄÖÜ etc. – and they are mostly easy enough to spot, but it may also affect other symbols such as en-dashes, quotes etc.

Important: Try first on a small sample to ensure it actually works before you get going on the whole file, and then work carefully and methodically.

GugUser · 2017-11-23 19:29:49

Thank you both.

bici: Textpattern version 4.5.7 was with PHP 5.6.+ and now 4.6.2 runs with PHP 7.1.

jakob: I will try to replace te wrong characters. But, what I need? The strange thing is that ü in the DB is not showed as ü on the site, but Ã¼ is ü on the site. And the strangest thing is that all the umlauts in titles and articles in the database cause the text fields in the edit area to appear blank. There is no title and no text, but on the site I can see te text, only with the wrong characters.

bici · 2017-11-23 20:37:32

Interesting. My local host is running PhP 7.1.8 and when i imported my Italian language site it had many issues with text display. But then again I have often had issues. This site was originally latin_1 and i had thought i had successfully translated it to UTF8, but i have to still update content with the the actual text that is correct. i am slowly translating all the HTML entities used for special characters.

jakob · 2017-11-23 20:56:21

GugUser wrote #307965:

I will try to replace te wrong characters. But, what I need? The strange thing is that ü in the DB is not showed as ü on the site, but Ã¼ is ü on the site.

I’m no expert on this, but have seen similar things before, though long ago. When I look at a current db-dump in utf-8 format, I see the regular symbol, e.g. ü, in the title, body and body_html (and so on). In the DB dump I have this for the table DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci.

I wonder if you can do this:

Make a DB-dump to sql (unzipped), make a safe backup copy of that first in case you need to go back to square one and start over, then load that in your text-editor and set to UTF-8 encoding in your editor and re-save.
Check that your Ã¼ are identifiable in the text and distinct, i.e. you can match that to ü and another string combination matches ö. If they’re not, search and replace won’t be viable, and you may have to do that in step 1 first.
Search and replace your character combinations to the proper utf-8 symbol.
Search all your CHARSET and COLLATE/COLLATION infos in your dump and make utf8 (or utf8mb4, should be okay).
Create a new database on your server, making sure the connection is utf8. If it’s not, alter that if you can. Then upload you new cleaned database to that.
Reconnect textpattern to the new database and see if it is now okay.

And the strangest thing is that all the umlauts in titles and articles in the database cause the text fields in the edit area to appear blank.

I’ve not seen that before, but you can probably edit the title down until it reappears to discover what it doesn’t like. I vaguely remember sometimes the Ã¼ string having an invisible character in there.

You can try out the principle with just one article first to test the principle before you do everything.

GugUser · 2017-11-23 22:01:45

I think there’s a misunderstanding. The Ã¼ is the only that works. I have no idea where it comes from, there’s only one. All the other umlauts are umlauts in the database too, like äöü.

What is the connection between the database und Textpattern? What makes that Textpattern shows in the backend an empty field if the text in the database contains on or more umlauts?

I made the side in 2014. I’ve never been logged in in a year. The site has always worked. And suddenly, for a few days now, it doesn’t work anymore. The only thing the provider says that has been changed is a security update on the server.

GugUser · 2017-11-23 23:45:15

I don’t understand what’s happening:

In a fin working installation, for example in the database table txp_lang there is a data “Einen neuen Autor hinzufügen”. In the backend it’s showed as “Einen neuen Autor hinzufügen”.

In the problematic installation there is in the database table txp_lang there a data “Einen neuen Autor hinzufÃ¼gen”. In the backend it’s showed as “Einen neuen Autor hinzufügen”. If I change it in the database to “Einen neuen Autor hinzufügen”, then it’s showed in the backend as “Einen neuen Autor hinzuf?gen”. I tried it with kollation utf8_general_ci and utf8mb4_unicode_ci and it’s all the same.

That’s how it is with all these characters in the database. I have no idea where these characters come from and why this works the other way round than in other installations.

If the content of the database is fine, for example ü in UTF-8 for ü, then it doesn’t work in the backend and neither in the frontend.

If I edit in a fine working Installation a new article with “Ein Versuch über äöü” then it’s stored in the DB as “Ein Versuch über äöü”. But the same in the problematic installation ist stored as “Ein Versuch Ã¼ber Ã¤Ã¶Ã¼” and showed on the site as “Ein Versuch über äöü”.

This is only the case for a short time now, before that the site had been running smoothly for three years.

jakob · 2017-11-23 23:48:53

Ah, check what you have set in config.php too.

GugUser · 2017-11-23 23:52:07

$txpcfg['dbcharset'] = 'utf8';

define('admin_custom_css', 'admin_custom.css');

# rvm_utf8_to_utf8mb4 plugin added the following line to ensure a correct dbcharset
$txpcfg['dbcharset'] = 'utf8mb4';

In the .htaccess file AddDefaultCharset utf-8

jakob · 2017-11-24 08:54:01

It’s definitely an encoding problem. I see those character combinations in other situations too sometimes (I have a workflow for datamerging training course certificates from a spreadsheet file of course participants to Indesign, and I have to set MacRoman as the encoding in InDesign for the characters to come through properly … That’s on my machine only; I wouldn’t recommend that for an online site).

The only thing I can suggest is:

get consistent encoding working at all levels:
- in config.php
- in the database connection/collation (you can see and set that in database properties/settings in PHPMyAdmin or similar)
- in each of your database table settings (there should be per-table settings)
- (perhaps check your host’s settings too, if you have access to them)
Find out which characters – or character combinations – display properly for you on the front end and the back end through trial and error. That might even be the weird ones, though I hope not.
Apply that combination consistently throughout by either editing your articles (.ie., the long way) or searching and replacing your exported .sql file.

CeBe · 2017-11-24 09:12:11

GugUser wrote #307960:

After a server update one of my customer’s websites (it was still running with Textpattern 4.5.7) showed only question marks instead of umlauts.

I think it is related to this old bug.

I ran into this issue and had to patch textpattern/lib/txplib_db.php as Bloke said.

Textpattern CMS

Textpattern CMS support forum

#1 2017-11-23 16:49:25