Quotes in pasted blocks of text

StFual · 2016-10-03 08:45:19

The diagnostic character set items you mention remain the same whether i use US or UK english in Textpattern

Looking on the host there are multiple language options under PHPadmin for language settings, these are the current settings .

Server connection collation : UTF8_general_ci
Under PHP appearance settings Language : US English.

Knowing nothing about PHP id assume the collation setting would be the one but there doesnt appear to be anything for UK English.

Edit: Sorry this came in parallel but UTF8_general_ci is OK ?

Last edited by StFual (2016-10-03 08:54:38)

StFual · 2016-10-03 08:49:28

phiw13 wrote #301911:

(that is running on a US based server – Dreamhost)

Any chance you could look at your PHPadmin collation settings on Dreamhost ?

jakob · 2016-10-03 09:02:39

Yes utf8 and UTF8_general_ci is okay (newer versions automatically now set utf8mb4), but there are several places to set this. You also need to make sure:

your database charset is set to utf8. In phpmyadmin, the collation is in the “Operations” tab but be warned, changing the character set can have adverse effects on an existing installation. That’s why I suggested – if you haven’t got much in your setup just yet – starting over with a new database, as it generally asks about character set and collation when creating a new database.
in your config.php, you also need $txpcfg['dbcharset'] = 'utf8'; (or utf8mb4).

StFual · 2016-10-03 11:30:50

phiw13 wrote #301911:

That is weird. I routinely use the British English language pack (that is the default) and have no issues with quotes of any type, nor problems with double byte languages. My wife runs a website with en-gb as the admin (back-end) language, all articles are in Japanese.

While i’m waiting for my hosting support to come online to see if i can change PHP collation permanently to “utf8mb4” I did another install using the US character set. It seems that whichever language I install with wont work properly. Subsequently loading the UK english character set inside Textpattern works both for textile and drop down languages. Switching back to US doesnt. So it seems to be an install mismatch .

Bloke · 2016-10-03 11:33:41

StFual wrote #301917:

whichever language I install with wont work properly. Subsequently loading the UK english character set inside Textpattern works both for textile and drop down languages. Switching back to US doesnt.

Freaky. Well that blows my lack-of-insatalled-locale theory out the window.

Loading and unloading languages won’t change collations or character sets on the server so yeah, maybe something going wrong during install. This is getting weirder by the hour…

StFual · 2016-10-03 12:16:23

Sorry if this is a dumb question

When i run the install script it gets me to load database details then generates text and asks me to copy it to create config.php which includes the line

$txpcfg[‘dbcharset’] = ‘utf8mb4’;

Is “utf8mb4” a default or is it generated by the installation programs understanding of the host system database ? I tried changing it to utf8 but it makes no difference to the problem (no better no worse).

Getting good at installing though.

Bloke · 2016-10-03 12:34:57

StFual wrote #301919:

Is “utf8mb4” a default

Yep. You can change it if you like, right there as you create the config file. Since the rest of your installation seems (mostly) to be using latin1, you could try that value and see what happens. Not a very long-term solution, though, unless you know you’re only ever going to be publishing in standard Western / European characters.

StFual · 2016-10-04 10:26:18

These were the settings from a known working system from phiw13 (thankyou)

Charset (default/config): latin1/utf8
character_set_client: utf8
character_set_connection: utf8
character_set_database: utf8
character_set_filesystem: binary
character_set_results: utf8
character_set_server: latin1
character_set_system: utf8

On my buggy system they are currently

Charset (default/config): latin1/utf8mb4
character_set_client: latin1
character_set_connection: latin1
character_set_database: utf8mb4
character_set_filesystem: binary
character_set_results: latin1
character_set_server: latin1
character_set_system: utf8

I note that the default on both systems is latin1. It seems that most hosting systems are set with a default of swedish latin1

I have been able to change “character_set_database” from ‘latin1’ to ‘utf8mb4’ by changing the collation setting in Phpmyadmin after database creation, prior to running install script. It hasn’t changed anything re. the problem

Presumably the settings on phiw13 host were either configured by the install script, or manually, or get set in a session script or .ini file.
Can i set the the others as session variables in .htaccess ? Anyone have any idea how they get configured on textpattern “supported” hosts like Arvixe and Dreamhost ?

StFual · 2016-10-05 13:36:40

After much testing what appears to be happening in my environment is the installation language pack causes the problem.

Do an install with any of the english language packs
Problems with double and single quotes
Empty the “txp_lang” table,
Reinstall the same language pack from within Textpattern
It works OK.

I can actually see a difference in the stored data in “txp_lang” between the install and post install language packs for the same double byte fields. Its the same data but the coding is different.

The only things I can imagine are:-

1: The installation population of the “txp_lang” table is using a different encoding method than population by installing a language pack post installation. Maybe the application is initialising something the installation script isnt ?
2: Its a different language pack used during Installation than is used post installation. Perhaps one is embedded and the other comes from the web ?

So my installation work around for quotes problem is empty the “txp_lang” table after install and reinstall the language.

Last edited by StFual (2016-10-05 13:38:10)

Bloke · 2016-10-05 14:26:48

StFual wrote #301980:

I can actually see a difference in the stored data in “txp_lang” between the install and post install language packs for the same double byte fields. Its the same data but the coding is different.

Thanks for your continued testing (and patience!)

This sounds intriguing. I’d love to get my hands on some (not all!) of this data so I can compare it side-by-side. Maybe, as you suggest, the installation is doing it differently to the lang injection. Installation uses raw SQL inserts, while any time thereafter it uses the API to install lang packs.

There could be something (line endings? encoding?) in the installed lang pack that your system chokes on for some reason, or the SQL statements might be mangling things, or there could be a bug under certain conditions.

Any chance you could send me some of the data from both installed and post-installed environments? Even via some freebie cloud storage provider would be ace. Thanks.

StFual · 2016-10-06 08:28:07

Thanks for the continued effort to support this. I do understand that this is almost certainly an environment problem however there are only so many times I can ask the hosting company support to change things and i’ve used up my silver bullets there i think. It would be good to know what it is so i can be specific in any request. I do think its worth investigation as Textpattern is a good application, shared hosting users dont have full control over the environment and typically like me dont always have detailed technical knowledge of how to configure the environment.
Id be happy to write a shared hosting install pitfall guide for this if I can make it work.

Just one note before i dump data. If I divide the “txp_lang” table contents as AW “application working” and ANW “application not working” data.

For the “txp_lang” “zh-cn” field (Chinese characters)
ANW data displays correctly using host Pgmyadmin tools suggesting that the intial data load is using environment default coding. This same data appears as garbage in the ANW Textpattern Language drop down.
AW data displays Chinese characters as garbage in Pgmyadmin but correctly in Textpattern.

I can set a temporary area on my site and post the data there or email them to you. Do you just want AW and ANW database dumps or anything else ? As i can’t be confident of what coding export tools will use i’ll include screen prints of what i see.

Textpattern CMS

Textpattern CMS support forum

#31 2016-10-03 08:45:19

Re: Quotes in pasted blocks of text

#32 2016-10-03 08:49:28

Re: Quotes in pasted blocks of text

phiw13 wrote #301911:

#33 2016-10-03 09:02:39

Re: Quotes in pasted blocks of text

#34 2016-10-03 11:30:50

Re: Quotes in pasted blocks of text

phiw13 wrote #301911:

#35 2016-10-03 11:33:41

Re: Quotes in pasted blocks of text

StFual wrote #301917:

#36 2016-10-03 12:16:23

Re: Quotes in pasted blocks of text

#37 2016-10-03 12:34:57

Re: Quotes in pasted blocks of text

StFual wrote #301919:

#38 2016-10-04 10:26:18

Re: Quotes in pasted blocks of text

#39 2016-10-05 13:36:40

Re: Quotes in pasted blocks of text

#40 2016-10-05 14:26:48

Re: Quotes in pasted blocks of text

StFual wrote #301980:

#41 2016-10-06 08:28:07

Re: Quotes in pasted blocks of text

Board footer