Textpattern CMS support forum
You are not logged in. Register | Login | Help
- Topics: Active | Unanswered
Pages: 1
Textile and non-English links
A recent mundane task for an AI bot flagged this scenario:
If a Chinese full stop immediately follows a Textile link, for example:
"付款方式":/chn/faq_payment。Textile may treat the `。` as part of the URL, producing a malformed link with encoded Chinese punctuation.
Manually adding a space fixes it:
"付款方式":/chn/faq_payment 。Can anyone corroborate this please and let me know if it affects other languages and/or punctuation characters? If so, we might be able to file a Textile bug and have it improve the regex link detector to handle multilingual punctuation.
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Hire Txp Builders – finely-crafted code, design and Txp
Offline
Re: Textile and non-English links
Stef, as I have said for the past 20 years, inline Textile (links, <strong>, etc) is NOT compatible with Chinese, Japanese, Mongolian, to some extend Korean, and probably problematic with Arabic, Persian, Urdu, Hindi,…
Reason: the Textile parser need a white-space as a separator.
In low-power mode.
Sand space – admin theme for Textpattern
phiw13 on Codeberg
Offline
Re: Textile and non-English links
I didn’t expect it to be a new thing, but don’t recall seeing a bug filed for it. Maybe I should have looked harder, sorry.
Apologies for wasting your time reminding me.
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Hire Txp Builders – finely-crafted code, design and Txp
Offline
Re: Textile and non-English links
Many many moons ago I communicated with the original developer™ of Textile about this issue (required white-space as boundary), that went nowhere (I don’t think it was here on the forum, but rather by email). I kinda lost interest in it, found workarounds (for a while MarkDown despite by dislike for it, it behaves less badly). Now mostly a couple of Textpattern short tags instead of using the Textile code (and a little help box on the side of write area, a little plugin thanks to Oleg).
Here is a little sample Japanese text illustrating a few issues: emps.l-c-n.com/notebook/japanese-with-textile
PS – in theory, for links one could use the <txp:link /> family, but lower level authors do not have access to the links panel
Last edited by phiw13 (Today 06:21:14)
In low-power mode.
Sand space – admin theme for Textpattern
phiw13 on Codeberg
Offline
Re: Textile and non-English links
Oooh that’s really helpful, thank you.
Interestingly, if I press and hold the first Wikipedia URL in the Textile block at the bottom, my phone chooses this by default as the URL:
https://en.wikipedia.org/wiki/リンゴに限らず商品_価値_の高い果実を収穫するためにはSo even Android can’t differentiate between the intended link, and requires some whitespace as demarker.
So in order to make sense of this and potentially come up with a fix, are there any language rules that might help the Textile parser differentiate between “end of the marked-up block” and “start of the next block” in the absence of a space or regular punctuation symbol?
Like, for example, the character immediately following the URL (に) is cited as a ‘direction’ indicator. Can Textile use it as a special case, along with a bunch of other characters that the regex can ‘look out for’ alongside space and English punctuation symbols to denote the ‘end’ of whatever came before it? I doubt it, since that Japanese character could appear anywhere else in text, so to specifically look for it and treat it as some delimiter might break other parts of Textile.
Regex, as a tool to parse text, is rather a blunt instrument. Regex101 has a fairly comprehensive set of example rules to catch various situations and Japanese character classes, but some of the rules are incredibly complex and might slow down the parser if implemented. There’s a simpler set available. But that’s all just for the various Japanese representations. Goodness knows how many variants there would be for other languages that don’t use space/punctuation as a delimiter.
What I’m intrigued about is whether the ‘unicode punctuation’ regex helper \p{P} can be of use to improve matters. Textile might already be using it, I haven’t checked.
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Hire Txp Builders – finely-crafted code, design and Txp
Offline
Re: Textile and non-English links
Interestingly, if I press and hold the first Wikipedia URL in the Textile block at the bottom, my phone chooses this by default as the URL
Interesting and suprising. The old iPhone doesn’t recognises that string as a link at all (it is after all in a block of code).
So in order to make sense of this and potentially come up with a fix, are there any language rules that might help the Textile parser differentiate between “end of the marked-up block” and “start of the next block” in the absence of a space or regular punctuation symbol?
As far a I know there are no characters that would could act as a delimiter.
What matters too is looking at the pair: for links a white-space / " / link text / ":link-url / white-space. The white-space at the start of the marked-up block is equally important.
I think what works in Markdown (for Japanese text at least) is the link notation [linktext](link-url). strong and em must use the *and ** , as a pair. But the other MD notation, using _, fails, I don’t remember if the ` works (code). (at least the Markdown variant available through Robert’s Textpattern plugin).
For Textile, if there is a magic regex that skip the white-space delimiter, that would make many inline tags work. The Textile link notation is more complicated.
In low-power mode.
Sand space – admin theme for Textpattern
phiw13 on Codeberg
Offline
Re: Textile and non-English links
Yes the delimiters in markdown are a blessing in that regard, and only one character more than Textile to type. I like them.
I don’t know how good lookbehind is with regex to consider the spacing/punctuation before, as well as after, strings.
I seem to recall there’s a syntax for “helping” things like ° degrees symbols and tm etc to help separate them from the value and keep the two items together. Might even be square brackets, can’t remember offhand.
I wonder if maybe the “solution” is to allow some global well-known delimiter pair like square or curly brackets around links and other inline markup to explicitly demarcate them in situations where they can’t be automatically detected.
That does mean, from a backwards compatibility standpoint, that if anyone has wrapped a link in those brackets they’d be swallowed and they would need to double up the brackets to render the desired characters.
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Hire Txp Builders – finely-crafted code, design and Txp
Offline
Pages: 1