Textpattern CMS support forum
You are not logged in. Register | Login | Help
- Topics: Active | Unanswered
Pages: 1
Textile and non-English links
A recent mundane task for an AI bot flagged this scenario:
If a Chinese full stop immediately follows a Textile link, for example:
"付款方式":/chn/faq_payment。Textile may treat the `。` as part of the URL, producing a malformed link with encoded Chinese punctuation.
Manually adding a space fixes it:
"付款方式":/chn/faq_payment 。Can anyone corroborate this please and let me know if it affects other languages and/or punctuation characters? If so, we might be able to file a Textile bug and have it improve the regex link detector to handle multilingual punctuation.
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Hire Txp Builders – finely-crafted code, design and Txp
Online
Re: Textile and non-English links
Stef, as I have said for the past 20 years, inline Textile (links, <strong>, etc) is NOT compatible with Chinese, Japanese, Mongolian, to some extend Korean, and probably problematic with Arabic, Persian, Urdu, Hindi,…
Reason: the Textile parser need a white-space as a separator.
In low-power mode.
Sand space – admin theme for Textpattern
phiw13 on Codeberg
Offline
Re: Textile and non-English links
I didn’t expect it to be a new thing, but don’t recall seeing a bug filed for it. Maybe I should have looked harder, sorry.
Apologies for wasting your time reminding me.
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Hire Txp Builders – finely-crafted code, design and Txp
Online
Re: Textile and non-English links
Many many moons ago I communicated with the original developer™ of Textile about this issue (required white-space as boundary), that went nowhere (I don’t think it was here on the forum, but rather by email). I kinda lost interest in it, found workarounds (for a while MarkDown despite by dislike for it, it behaves less badly). Now mostly a couple of Textpattern short tags instead of using the Textile code (and a little help box on the side of write area, a little plugin thanks to Oleg).
Here is a little sample Japanese text illustrating a few issues: emps.l-c-n.com/notebook/japanese-with-textile
PS – in theory, for links one could use the <txp:link /> family, but lower level authors do not have access to the links panel
Last edited by phiw13 (Today 06:21:14)
In low-power mode.
Sand space – admin theme for Textpattern
phiw13 on Codeberg
Offline
Re: Textile and non-English links
Oooh that’s really helpful, thank you.
Interestingly, if I press and hold the first Wikipedia URL in the Textile block at the bottom, my phone chooses this by default as the URL:
https://en.wikipedia.org/wiki/リンゴに限らず商品_価値_の高い果実を収穫するためにはSo even Android can’t differentiate between the intended link, and requires some whitespace as demarker.
So in order to make sense of this and potentially come up with a fix, are there any language rules that might help the Textile parser differentiate between “end of the marked-up block” and “start of the next block” in the absence of a space or regular punctuation symbol?
Like, for example, the character immediately following the URL (に) is cited as a ‘direction’ indicator. Can Textile use it as a special case, along with a bunch of other characters that the regex can ‘look out for’ alongside space and English punctuation symbols to denote the ‘end’ of whatever came before it? I doubt it, since that Japanese character could appear anywhere else in text, so to specifically look for it and treat it as some delimiter might break other parts of Textile.
Regex, as a tool to parse text, is rather a blunt instrument. Regex101 has a fairly comprehensive set of example rules to catch various situations and Japanese character classes, but some of the rules are incredibly complex and might slow down the parser if implemented. There’s a simpler set available. But that’s all just for the various Japanese representations. Goodness knows how many variants there would be for other languages that don’t use space/punctuation as a delimiter.
What I’m intrigued about is whether the ‘unicode punctuation’ regex helper \p{P} can be of use to improve matters. Textile might already be using it, I haven’t checked.
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Hire Txp Builders – finely-crafted code, design and Txp
Online
Pages: 1