Quotes in pasted blocks of text

StFual · 2016-10-03 06:05:14

Bloke wrote #301904:

Your diagnostics really would help us reproduce this.

Can you check which PCRE version you have with your installed PHP? If you can write an article (without Textile switched on!) that contains this:

PHP version 7 seems to be the obvious item.
Ive just noted that textpattern requires PHP 5.4+ My host offers 5 as an option so i can downgrade

The host date and timezone dont match textpattern and reading the requirements timezone seems to be important. I can look at that

iconv directives are blank

Multibyte support has some no value lines

Multibyte Support enabled
mbstring.detect_order no value no value
mbstring.encoding_translation Off Off
mbstring.func_overload 0 0
mbstring.http_input no value no value
mbstring.http_output no value no value
mbstring.http_output_conv_mimetypes ^(text/|application/xhtml\+xml) ^(text/|application/xhtml\+xml)
mbstring.internal_encoding no value no value
mbstring.language neutral neutral
mbstring.strict_detection Off Off
mbstring.substitute_character no value no value

StFual · 2016-10-03 06:38:01

I believe i have the solution.

Installing and selecting the US English language pack in Textpattern Double and Single quotes are parsed correctly .
With the British English language pack it doesnt.

I can recreate the problem by selecting British English again. Quotes display correctly with BE but dont work. Also with the British English language pack enabled any double byte languages dsplay as garbage text under the Language select drop down menu.

I can live with Zees instead of Zeds but let me know if you want other details

I have corrected the host date.time zone but PHP is still on version 7 . Should i downgrade ?

StFual · 2016-10-03 07:52:39

colak wrote #301903:

This is indeed true! In such cases I use a bitly short url and then add <txp:hide>https://web.archive.org/web/20151019021621/http://textpattern.com/documentation</txp:hide> so as to keep a record of the web archive’s url just in case bitly decides to close down at some point. Alternatively you can use raw html: <a rel="external" href="https://web.archive.org/web/20151019021621/http://textpattern.com/documentation">your link</a>

Thanks. That was a “holy crap” moment!. While i’ve cut and pasted them many times I didnt know how those short links were generated before.

Bloke · 2016-10-03 07:57:39

StFual wrote #301907:

Installing and selecting the US English language pack in Textpattern Double and Single quotes are parsed correctly. With the British English language pack it doesnt.

Thanks for the diagnostics info. Very helpful.

I’m not the most well-versed at locale issues, so hopefully one of the more multi-lingual-aware forum members can help out. On the surface, it looks like there are several forces at play, but PHP 7 probably isn’t one of them. I’d leave that alone for now.

A few things that look anomalous in the diagnostics:

Last update: 1970-01-01 00:00:00 (might be normal for new installs, can’t remember).
Locale: en_GB.UTF-8
Charset (default/config): latin1/utf8mb4
character_set_client: latin1
character_set_connection: latin1
character_set_database: latin1
character_set_filesystem: binary
character_set_results: latin1
character_set_server: latin1
character_set_system: utf8

That’s a strange concoction of charsets that I would expect contribute to the various effects you’re seeing: garbled characters and perhaps the Textile problems. Quite how to alter it without data corruption I’m not entirely sure. Hopefully Ruud or someone else here on the forum will be able to advise.

Regarding why en-us vs en-gb fixes it: maybe the fact that the server is on US soil and, presumably, configured only with that locale installed, has some bearing.

The other stuff like Litespeed and MariaDB, although not officially supported at the moment, are probably not a factor here if the rest of the installation behaves itself.

jakob · 2016-10-03 08:21:37

Bloke wrote #301909:

That’s a strange concoction of charsets that I would expect contribute to the various effects you’re seeing: garbled characters and perhaps the Textile problems. Quite how to alter it without data corruption I’m not entirely sure. Hopefully Ruud or someone else here on the forum will be able to advise.

Yes, that was in the back of my mind earlier. It looks like you don’t have all that many entries in your installation yet (you’ve just been testing, right?) and are using the standard template, so maybe it’s easiest to reinstall the whole thing completely using a fresh database with collation set to utf8mb4 / utf8mb4_unicode_ci or utf-8 / utf8_general_ci. If you can influence your mysql setup on the server, try and make sure that is also set to utf-8.

phiw13 · 2016-10-03 08:27:11

StFual wrote #301907:

Installing and selecting the US English language pack in Textpattern Double and Single quotes are parsed correctly .
With the British English language pack it doesnt.

I can recreate the problem by selecting British English again. Quotes display correctly with BE but dont work. Also with the British English language pack enabled any double byte languages dsplay as garbage text under the Language select drop down menu.

That is weird. I routinely use the British English language pack (that is the default) and have no issues with quotes of any type, nor problems with double byte languages. My wife runs a website with en-gb as the admin (back-end) language, all articles are in Japanese.

As Bloke notes, I suspect your issues have more to do with the mix of character_sets in the database. Ideally you’d have them set to utf8mb4 nowadays (except probably@character_set_server@, which is out of your control, I think).

Here are mines:

Charset (default/config): latin1/utf8
character_set_client: utf8
character_set_connection: utf8
character_set_database: utf8
character_set_filesystem: binary
character_set_results: utf8
character_set_server: latin1
character_set_system: utf8

(that is running on a US based server – Dreamhost)

StFual · 2016-10-03 08:45:19

The diagnostic character set items you mention remain the same whether i use US or UK english in Textpattern

Looking on the host there are multiple language options under PHPadmin for language settings, these are the current settings .

Server connection collation : UTF8_general_ci
Under PHP appearance settings Language : US English.

Knowing nothing about PHP id assume the collation setting would be the one but there doesnt appear to be anything for UK English.

Edit: Sorry this came in parallel but UTF8_general_ci is OK ?

Last edited by StFual (2016-10-03 08:54:38)

StFual · 2016-10-03 08:49:28

phiw13 wrote #301911:

(that is running on a US based server – Dreamhost)

Any chance you could look at your PHPadmin collation settings on Dreamhost ?

jakob · 2016-10-03 09:02:39

Yes utf8 and UTF8_general_ci is okay (newer versions automatically now set utf8mb4), but there are several places to set this. You also need to make sure:

your database charset is set to utf8. In phpmyadmin, the collation is in the “Operations” tab but be warned, changing the character set can have adverse effects on an existing installation. That’s why I suggested – if you haven’t got much in your setup just yet – starting over with a new database, as it generally asks about character set and collation when creating a new database.
in your config.php, you also need $txpcfg['dbcharset'] = 'utf8'; (or utf8mb4).

StFual · 2016-10-03 11:30:50

phiw13 wrote #301911:

That is weird. I routinely use the British English language pack (that is the default) and have no issues with quotes of any type, nor problems with double byte languages. My wife runs a website with en-gb as the admin (back-end) language, all articles are in Japanese.

While i’m waiting for my hosting support to come online to see if i can change PHP collation permanently to “utf8mb4” I did another install using the US character set. It seems that whichever language I install with wont work properly. Subsequently loading the UK english character set inside Textpattern works both for textile and drop down languages. Switching back to US doesnt. So it seems to be an install mismatch .

Bloke · 2016-10-03 11:33:41

StFual wrote #301917:

whichever language I install with wont work properly. Subsequently loading the UK english character set inside Textpattern works both for textile and drop down languages. Switching back to US doesnt.

Freaky. Well that blows my lack-of-insatalled-locale theory out the window.

Loading and unloading languages won’t change collations or character sets on the server so yeah, maybe something going wrong during install. This is getting weirder by the hour…

StFual · 2016-10-03 12:16:23

Sorry if this is a dumb question

When i run the install script it gets me to load database details then generates text and asks me to copy it to create config.php which includes the line

$txpcfg[‘dbcharset’] = ‘utf8mb4’;

Is “utf8mb4” a default or is it generated by the installation programs understanding of the host system database ? I tried changing it to utf8 but it makes no difference to the problem (no better no worse).

Getting good at installing though.

Textpattern CMS

Textpattern CMS support forum

#25 2016-10-03 06:05:14

Re: Quotes in pasted blocks of text

Bloke wrote #301904:

#26 2016-10-03 06:38:01

Re: Quotes in pasted blocks of text

#27 2016-10-03 07:52:39

Re: Quotes in pasted blocks of text

colak wrote #301903:

#28 2016-10-03 07:57:39

Re: Quotes in pasted blocks of text

StFual wrote #301907:

#29 2016-10-03 08:21:37

Re: Quotes in pasted blocks of text

Bloke wrote #301909:

#30 2016-10-03 08:27:11

Re: Quotes in pasted blocks of text

StFual wrote #301907:

#31 2016-10-03 08:45:19

Re: Quotes in pasted blocks of text

#32 2016-10-03 08:49:28

Re: Quotes in pasted blocks of text

phiw13 wrote #301911:

#33 2016-10-03 09:02:39

Re: Quotes in pasted blocks of text

#34 2016-10-03 11:30:50

Re: Quotes in pasted blocks of text

phiw13 wrote #301911:

#35 2016-10-03 11:33:41

Re: Quotes in pasted blocks of text

StFual wrote #301917:

#36 2016-10-03 12:16:23

Re: Quotes in pasted blocks of text

Board footer