Odd Characters, Blank Templates

jrmartin · 2018-07-08 15:34:53

About ten days ago certain pages on Martha’s Italy started to get odd characters, usually in place of apostrophes. Here’s a sample

On these pages, the article is not visible for editing. Worse yet, all used template pages are also blank. But the site is working. Article pages that display properly CAN be edited.

I didn’t mess with anything. I didn’t upgrade the software. No errors are showing in diagnostics. Languages are up to date.

Any clues?

james

Last edited by jrmartin (2018-07-08 15:35:45)

colak · 2018-07-08 16:06:08

hi James,

Does phpMyAdmin provide any clues?

jrmartin · 2018-07-08 17:43:25

colak wrote #312910:

hi James,

Does phpMyAdmin provide any clues?

I’m no expert in these things, but I compared it to another site and nothing seems too different that I noticed.

jakob · 2018-07-08 21:06:12

jrmartin wrote #312909:

About ten days ago certain pages on Martha’s Italy started to get odd characters, usually in place of apostrophes. Here’s a sample

If I load that page in Safari, I see the problem character in the “it<>s” of “while it’s currently free”. If I change the text encoding (view menu I think) to ISO Latin 1, it displays correctly, so there may be an encoding mismatch somewhere. No idea why it has appeared out of the blue, but one reason might be that the host updated either php or mysql. I suggest you make a backup first before trying any special character set collation changes.

jrmartin · 2018-07-08 23:04:40

jakob wrote #312912:

If I load that page in Safari, I see the problem character in the “it<>s” of “while it’s currently free”. If I change the text encoding (view menu I think) to ISO Latin 1, it displays correctly, so there may be an encoding mismatch somewhere. No idea why it has appeared out of the blue, but one reason might be that the host updated either php or mysql. I suggest you make a backup first before trying any special character set collation changes.

Well, it’s interesting. I’ve checked three of my textpattern sites and the collation is all all different. Martha’s Italy is utf8_general_ci.

Wandering Italy, which seems to work fine and is the oldest site, is latin1_swedish_ci, and my new site, Italian Menu Master, is utf8mb4_unicode_ci. That’s working fine, too.

jrmartin · 2018-07-09 01:07:12

Just to throw this out, here’s a bit from diagnostics on character encoding. I’m not sure what the first line means because it seems contradictory.

Charset (default/config): latin1/utf8
character_set_client: latin1
character_set_connection: latin1
character_set_database: latin1
character_set_filesystem: binary
character_set_results: latin1
character_set_server: latin1
character_set_system: utf8
character_sets_dir: /usr/share/mysql/charsets/
17 Tables: textpattern is utf8, txp_category is utf8, txp_css is utf8, txp_discuss is utf8, txp_discuss_nonce is utf8, txp_file is utf8, txp_form is utf8, txp_image is utf8, txp_lang is utf8, txp_link is utf8, txp_log is utf8, txp_page is utf8, txp_plugin is utf8, txp_prefs is utf8, txp_section is utf8, txp_token is utf8, txp_users is utf8

bici · 2018-07-09 04:48:37

any clue for you here

jrmartin · 2018-07-09 05:20:56

bici wrote #312916:

any clue for you here

Hmmmm, well, that could be it—but why it’s started all of a sudden is still a mystery.

phiw13 · 2018-07-09 12:22:46

jrmartin wrote #312917:

… but why it’s started all of a sudden is still a mystery.

Has the database been updated recently (by your hosting provider)?
Or did this start when you updated from TXP 4.6.x to 4.7.x?

jrmartin · 2018-07-09 12:51:30

phiw13 wrote #312925:

Has the database been updated recently (by your hosting provider)?
Or did this start when you updated from TXP 4.6.x to 4.7.x?

I’ll ask if they updated the database. I haven’t updated the site to 4.7 yet.

james

bici · 2018-07-09 15:28:01

In Safari when i change the Text Encoding view to Western ISO Latin 1 the odd characters disappear.

I hope you are not copying text from Word documents and then pasting into your Write panel in Textpattern.

Years ago i had a similar problem with an older format etting – unicode swedish_ci i think – I solved by running the DB through a script – see this:

Converting your MySQL database to UTF8

P.S. I NEVER copy/paste from Word documents. Place all text into a neutral text editor. BBedit or Notepad or Textmate … before pasting into your CP

jrmartin · 2018-07-09 15:32:27

bici wrote #312928:

In Safari when i change the Text Encoding view to Western ISO Latin 1 the odd characters disappear.

I hope you are not copying text from Word documents and then pasting into your Write panel in Textpattern.

Years ago i had a similar problem with an older format etting – unicode swedish_ci i think – I solved by running the DB through a script – see this:

Converting your MySQL database to UTF8

P.S. I NEVER copy/paste from Word documents. Place all text into a neutral text editor. BBedit or Notepad or Textmate … before pasting into your CP

Don’t worry, I don’t even have Word on my computer. Everything is written on the keyboard.

jrmartin · 2018-07-10 18:16:56

So the hosting provider reports that a database upgrade happened at the time we noticed the odd characters.

I’ve gone through the pages with those characters (which were indeed pasted quotes and things) and replaced them. Now they’re good.

But there are still issues. For example, if I write “the emperors’ hats,” the trailing apostrophe causes that paragraph not to display on the html page.

And…my page templates included ©, which caused them not to display in the form. Scary. I went into the database and copied default and archive forms and changed the character to & copy; and pasted them into the form and all was well.

So I’m still worried about upgrading, given these problems.

Last edited by jrmartin (2018-07-10 18:19:04)

colak · 2018-07-10 18:24:20

There are plugins which help with the upgrade of the db character set.

Try: rvm_latin1_to_utf8 and rvm_utf8_to_utf8mb4 after that, both are downloadable from the same page. Remember to backup your db before you install the plugins.

jakob · 2018-07-22 07:28:21

I read you did a database upgrade on this thread but saw no difference in the files. You can check if that is really the case by exporting your updated database and then comparing the old and new using a diff tool.

One further idea – do you have the line:

$txpcfg['dbcharset'] = 'utf8';

as shown here in your config.php file?

If you want some off-forum assistance, send me an email…

Textpattern CMS

Textpattern CMS support forum

#1 2018-07-08 15:34:53