Quoted text does not appear in articles

gaekwad · 2017-10-12 11:45:40

Hi, everyone. This is an odd one I’d appreciate advice on.

Textpattern 4.6.2 on Litespeed (I know) moved from a Textpattern 4.6.2 on Apache.

Textile is enabled globally. Take the following test article body:

This is a plain paragraph.

"This is a sentence with inverted commas, no encoding."

&quot;This is a sentence with inverted commas, HTML encoding.&quot;

'This is a sentence with apostrophes, no encoding.'

&apos;This is a sentence with apostrophes, HTML encoding.&apos;

I use the Write area to save that, and it appears to save correctly. However, the front end only renders:

This is a plain paragraph.

"This is a sentence with inverted commas, HTML encoding."

'This is a sentence with apostrophes, HTML encoding.'

Hmm. So, the non-HTML encoded lines just don’t appear.

Checking the database, this is the `Body` column value:

This is a plain paragraph.

"This is a sentence with inverted commas, no encoding."

&quot;This is a sentence with inverted commas, HTML encoding.&quot;

'This is a sentence with apostrophes, no encoding.'

&apos;This is a sentence with apostrophes, HTML encoding.&apos;

So far, so good. This is the `Body_html` column value:

<p>This is a plain paragraph.</p>



<p>&quot;This is a sentence with inverted commas, <span class="caps">HTML</span> encoding.&quot;</p>



<p>&apos;This is a sentence with apostrophes, <span class="caps">HTML</span> encoding.&apos;</p>

That’s bad. Textile is marking up the HTML as a caps span, but lines are missing.

Any ideas why this might be occurring? Debug trace can be supplied if helpful.

Thank you mucho in advance.

jakob · 2017-10-13 06:49:50

No idea. I checked on txstyle.org and on your Textpattern demo page copying your sample and it works as it should.

Any other differences aside from Litespeed? Is there anything different about PHP implementations, environment variables or configs when running on Litespeed? Does the same site work again when you return it /set it up again running on Apache? Does a fresh install on Litespeed work correctly? Those might at least eliminate some sources of the error…

gaekwad · 2017-10-13 07:26:39

Hi jakob – thank you. This seems to be host-specific, I’ve never encountered it before on other hosting providers.

I’ve set up two factory-fresh Textpatterns (4.6.2 and 4.7.0-dev), and I can reproduce the issue (only 3 of 5 lines displayed) on the new 4.6.2, whereas the 4.7.0-dev works as expected (5 of 5 lines displayed).

Subdomains of the same domain, both subdomains brand new, neither has a history at this host, so no baggage to factor in. Very odd.

Clean 4.6.2 diagnostics (domain reference replaced with EXAMPLE for privacy):

Textpattern version: 4.6.2 (8412e32468526e075446b881d13bd9ce)
Last update: 1970-01-01 00:00:00/2017-10-12 18:51:50
Document root: /home/EXAMPLE/scratch.EXAMPLE.org
$path_to_site: /home/EXAMPLE/scratch.EXAMPLE.org
Textpattern path: /home/EXAMPLE/scratch.EXAMPLE.org/textpattern
Article URL pattern: section_id_title
Temporary directory path: /home/EXAMPLE/scratch.EXAMPLE.org/textpattern/tmp
Site URL: scratch.EXAMPLE.org
PHP version: 5.6.31
GD Graphics Library: bundled (2.1.0 compatible); supported formats: GIF, JPG, PNG.
Server TZ: Europe/London
Server local time: 2017-10-13 08:20:09
Daylight Saving Time enabled?: 0
Automatically adjust Daylight Saving Time setting?: 0
Time zone (GMT offset in seconds): Europe/London (+3600)
MySQL: 10.0.31-MariaDB-cll-lve
Database server time: 2017-10-13 08:20:09
Database server time offset: 0 s
Database server timezone: SYSTEM
Database session timezone: SYSTEM
Locale: en_GB.UTF-8
Server: LiteSpeed
PHP server API: litespeed
RFC 2616 headers: 
Server OS: Linux 3.10.0-614.10.2.lve1.4.50.el7.x86_64
Admin-side theme: hive 4.6.2

Pre-flight check: 
------------------------
Clean URLs are only supported on Apache, use at your own risk.
The following PHP functions (which may be necessary to run Textpattern) are disabled on your server: system_exec, symlink, shell, ini_alter, show_source, highlight_file, define_syslog_variables, posix_uname, posix_getpwuid, apache_child_terminate, posix_kill, posix_mkfifo, posix_setpgid, posix_setsid, posix_setuid, ini_restore, inject_code, syslog, openlog, define_syslog_variables, apache_setenv, eval, phpAds_XmlRpc, phpAds_remoteInfo, phpAds_xmlrpcEncode, phpAds_xmlrpcDecode, xmlrpc_entity_decode
------------------------

.htaccess file contents: 
------------------------
# BEGIN Textpattern
#DirectoryIndex index.php index.html

#Options +FollowSymLinks
#Options -Indexes
#ErrorDocument 403 default

<IfModule mod_rewrite.c>
    RewriteEngine On
    #RewriteBase /relative/web/path/

    RewriteCond %{REQUEST_FILENAME} -f [OR]
    RewriteCond %{REQUEST_FILENAME} -d
    RewriteRule ^(.+) - [PT,L]

    RewriteCond %{REQUEST_URI} !=/favicon.ico
    RewriteRule ^(.*) index.php

    RewriteCond %{HTTP:Authorization}  !^$
    RewriteRule .* - [E=REMOTE_USER:%{HTTP:Authorization}]
</IfModule>

#php_value register_globals 0

<IfModule mod_mime.c>
    AddType image/svg+xml  svg svgz
    AddEncoding gzip       svgz
</IfModule>
# END Textpattern

------------------------

…and 4.7.0-dev diagnostics (again, domain references replaced with EXAMPLE):

Textpattern version: 4.7.0-dev (def121753d361149d8fdcb49e327fc0a)
Last update: 2017-10-12 19:17:48/2017-10-12 19:15:07
Document root: /home/EXAMPLE/scratchdev.EXAMPLE.org
$path_to_site: /home/EXAMPLE/scratchdev.EXAMPLE.org
Textpattern path: /home/EXAMPLE/scratchdev.EXAMPLE.org/textpattern
Article URL pattern: messy
Temporary directory path: /home/EXAMPLE/scratchdev.EXAMPLE.org/textpattern/tmp
Site URL: scratchdev.EXAMPLE.org
PHP version: 5.6.31
GD Graphics Library: bundled (2.1.0 compatible); supported formats: GIF, JPG, PNG.
Server TZ: Europe/London
Server local time: 2017-10-13 08:21:57
Daylight Saving Time enabled?: 0
Automatically adjust Daylight Saving Time setting?: 0
Time zone (GMT offset in seconds):  (+3600)
MySQL: 10.0.31-MariaDB-cll-lve
Database server time: 2017-10-13 08:21:57
Database server time offset: 0 s
Database server timezone: SYSTEM
Database session timezone: SYSTEM
Locale: en_GB.UTF-8
Server: LiteSpeed
PHP server API: litespeed
RFC 2616 headers: 
Server OS: Linux 3.10.0-614.10.2.lve1.4.50.el7.x86_64
Admin-side theme: hive 4.7.0-dev

Pre-flight check: 
------------------------
The following PHP functions (which may be necessary to run Textpattern) are disabled on your server: system_exec, symlink, shell, ini_alter, show_source, highlight_file, define_syslog_variables, posix_uname, posix_getpwuid, apache_child_terminate, posix_kill, posix_mkfifo, posix_setpgid, posix_setsid, posix_setuid, ini_restore, inject_code, syslog, openlog, define_syslog_variables, apache_setenv, eval, phpAds_XmlRpc, phpAds_remoteInfo, phpAds_xmlrpcEncode, phpAds_xmlrpcDecode, xmlrpc_entity_decode
------------------------

There are more-than-usual PHP function disabled, and even commenting all those out (so they load) doesn’t help. The plain text isn’t being parsed to Textile correctly, so a debug HTML dump might not be that useful, though I am happy to supply it.

I’ve also got FTP, PHPMyAdmin and control panel access…and a patient client, which is helpful. His production site issues have been resolved by an upgrade to 4.7.0-dev, so the firefighting is done, but this has really stumped me.

Bloke · 2017-10-13 10:01:39

Utterly stumped. Cannot make it misbehave on my brand new 4.6.2 installation.

What we know:

The hosting setup is identical on the 4.6.2 and 4.7.0-dev installations. Same hosting hardware, same PHP, DB, web server.
The dev DB has been upgraded to 4.7.0-dev schema on the dev installation, but the textpattern table structure is identical in both installations – same collation (utf8mb4_general_ci), same column defs and sizes.
Both DBs running MyISAM.
There’s no difference between Txp’s implementation of vendors\Textpattern\Textile\Parser.php in either version of Textpattern.
The POST payload is identical, as far as I can tell – everything in the Body textarea is sent the same way. Haven’t compared it at a byte level.
The only change in Textile itself is the Parser.php. But Pete tried moving that file to the 4.6.2 install and re-saving the article. It apparently yields the same result as with the 4.6.2 version of Textile: missing output. So the issue is not confined to just Textile itself, although it’s clearly part of the problem, because…
… turning off Textile processing on the Body yields the entire article content in the Body_html column.
PCRE is a recent version (8.40 2017-01-11). If there was a bug / incompatibility introduced in this version which interfered with the older Textile in 4.6.2, it would have been negated when trying the newer Textile, but there was no change in output after saving the article again.
Starting or ending a sentence with a straight double-quote or straight apostrophe prevents the output of that entire line.
Using a single apostrophe anywhere within the line is fine and yields that content with the apostrophe encoded correctly.
Using a double-quote character anywhere within the line does not output the line of text.
I cannot repeat this behaviour on my own server: the 4.6.2 Textile behaves correctly with Pete’s article content.

Theories:

The Textile regex that handles quotes or anchors (quote-some_text-quote-colon-link) in 4.6.2 is broken under certain server environments and it’s interpreting a quoted string without a following colon as some invalid sequence and barfing. But why would the 4.7.0 parser do the same inside a 4.6.2 installation, when it behaved fine on a 4.7.0-dev installation?
The server is mangling/encoding the quote/apostrophe characters during form submission so Textile 4.6.2 doesn’t see them as actual characters and doesn’t know what to do with them and just skips the content. But again, why would the 4.7.0 parser exhibit the same behaviour in a 4.6.2 installation?
Something odd is happening in txp_article.php during form submission. Can’t be restricted to the AJAX save, since the problem shows up on first submission (which is non-AJAX). However, the current 4.7.0-dev has merged save functionality into one AJAX process, regardless of first submission or not, so that might have a bearing on things: maybe submitting via POST vs submitting via AJAX is subtly different on the host environment. Downgrading a 4.7.0-dev to use the 4.6.2 txp_article.php and the 4.6.2 Textile Parser.php (at least on my server, although that works anyway) functioned fine.
Some weird, subtle interaction between the character encoding of the server environments – the connection charset, the DB charset, etc – that’s causing some aspects of Textile to cry.

Short of FTPing up a Parser.php with a bunch of debug statements in it to see what it receives and/or try to identify which part of the process is failing, I’m bewildered.

gaekwad · 2017-10-13 10:37:02

Bloke wrote #307359:

Short of FTPing up a Parser.php with a bunch of debug statements in it to see what it receives and/or try to identify which part of the process is failing, I’m bewildered.

Thank you, Bloke. If such a file exists, I can do that – and I can pass you FTP details if you want to scratch an itch.

Bloke · 2017-10-13 10:56:52

If you are willing to let me poke around in your scratch directories to try and uncover some of the mystery here, I’d love to give it a whirl.

gaekwad · 2017-10-13 10:58:00

Bloke wrote #307362:

If you are willing to let me poke around in your scratch directories to try and uncover some of the mystery here, I’d love to give it a whirl.

I am so stealing that line for Tinder. Email incoming shortly.

Bloke · 2017-10-13 13:16:03

Thanks to Pete’s login and some judicious debug output, I’ve nailed it.

Are you ready?

Either during upgrade to 4.6.2 or during install (assuming you did the latter, right?) there are some glyph characters that have NOT been set in the txp_lang table. That means Textile never gets told what symbols to replace in the regexes that match “double quotes” and “single quotes”. Thus it searches for the quote characters in the article body, finds them and replaces them with… nada. Then it eats the entire sentence because it looks empty. And you get no output for that block of text.

My suspicion is that in 4.6.2 we use the INSERT DELAYED INTO construct when injecting language strings into the DB. Perhaps your version of MySQL is in some mode or doesn’t support this extension so the strings never got inserted, and Textile fell over. All the docs indicate MariaDB does support it, so maybe I’m way off base. But I can’t see any other place in the code where the strings are injected.

Now, in 4.7.0 we’ve designed out the ‘DELAYED’ inserts, so when you install that version, the strings get inserted. Either that, or makss’ improvements to the installation procedure have fixed the issue. Either way, Textile only works when it has all the information it needs. Hence everything was still borked no matter what files we replaced in the 4.6.2, because the DB still had missing strings and the new version of Textile still expects to be fed a complete set of glyphs.

A few schools of thought here: a) it’s a bug in Textile to assume that there are replacement glyphs when there might not be. b) it’s a bug in 4.6.2 for not checking the lang strings get inserted fully. c) it’s client caveat utilitor if they use unsupported databases ;-)

The upshot is that all you have to do to restore normality is to insert the following 4 strings into your txp_lang table:

event: 'common', name: 'txt_quote_double_close', data: '”'
event: 'common', name: 'txt_quote_double_open', data: '“'
event: 'common', name: 'txt_quote_single_close', data: '’'
event: 'common', name: 'txt_quote_single_open', data: '‘'

Sorted.

gaekwad · 2017-10-13 13:18:50

Wow!

jakob · 2017-10-13 15:25:38

gaekwad wrote #307367:

Wow!

+1! Amazing sleuthing…

Textpattern CMS

Textpattern CMS support forum

#1 2017-10-12 11:45:40