RFC: Textpattern 5 ideas & feature requests

Bloke · 2024-08-21 13:25:37

bici wrote #337689:

a shell script command to update Textpattern via the command line on the server.
eg. php txpcli.php update

Paging @gaekwad. Thoughts?

gaekwad · 2024-08-21 13:50:35

Bloke wrote #337694:

Paging @gaekwad. Thoughts?

I thought I felt my ears burning…

Interesting idea. The auto-installer is a solved problem and battle-tested, but there’s a whole level of other complexity with an upgrade script, including:

Less than 100% visibility on what level of shell access is available
Less than 100% visibility on what apps are available (rsync might be limited, for example)
Less than 100% visibility on ownership permissions
Less than 100% visibility on what assets (files, directories) are to be kept (especially if someone is using /textpattern/some-custom-directory/etc/
Responsibility for backups

It might be easier if we have a much more modular & predictable file + directory structure with v5, but right now it scares me to even try undertaking this on a third-party site.

That said…Wordpress can do the whole thing from the admin-side, so it’s not beyond the realms of possibility. No shell stuff required.

gaekwad · 2024-08-21 13:54:01

This is a v5 preflight thing I’ve been thinking on, could be applied to v4.8 to help us out with upgrading.

A function to export site content to an archive, which can be imported into a v5 site, should we decide that we’re gonna run with scissors and break backwards compatibility to that extent.

Some or all of: files, images, articles, scaffolding (categories & sections), database, and anything else I’ve forgotten.

Edit: I forgot themes.

Last edited by gaekwad (2024-08-21 13:55:55)

Bloke · 2024-08-21 14:29:01

Yes, upgrades are tough not only because we rely on admins uploading files, but also because not all of those files are required or even desired in a live installation. That’s why I asked @bici about what his expectations were.

Is it the intention to run a single command in a live installation, have it know where to get the source files (github release?) and pull them down, unpack them in a temp location, apply only the ones it needs, without trampling anything the user might have customised, then run the upgrade script (if we expose it to cli use, as it will currently only run via the web server, or introduce a parallel script that can run from the cli) to update the database and set the version, then check for updates to plugins/themes and ask for those to be upgraded too?

Or is it simply enough to have the user manually upload the relevant files and have the cli kick off the upgrade? And then rely on the admin to manually update the plugins and themes afterwards.

There are so many facets of an upgrade vs an installation. It’s relatively easy to do it in a semi-controlled web server environment; way more difficult in the wild west of the command line. Not saying it’s impossible, but it would need some carefully considered and refactored code to separate out the various bits.

For example, in some respects it would be easier if the upgrade database commands were bundled into a separate file and could be executed by the web browser upgrade path (via logging into Txp, as we do now) OR some CLI process by directly logging into the DB (credentials taken from config.php) and applying them. But filesystem changes are often interleaved with the DB alterations and need to be done in particular orders, so it’s tricky to segregate them. And sometimes we need to use the power of our safe_*() commands to execute stuff in loops, which is tedious or impossible to do directly in SQL.

Lots to think about. So if we are going to consider this – perhaps alongside broader changes such as boosting our import/export capabilities to include files and images – then we need to define the scope of what “do upgrade” really means.

EDIT: I think I would rather initially put the effort into giving the Diagnostics panel’s “new version is available” the ability to “click here to install it” and have it go through the process of fetching and unpacking source files, then kicking off the /update scripts to apply database and filesystem changes. Although we recommend logging out during upgrades, that’s primarily because logging in again is a more deterministic starting point to trigger upgrade scripts. We’ve relaxed that requirement in recent versions (4.7+ or 4.8+, I forget which) with more robust checks on every admin page load to see if an upgrade is required and kick it off if so.

Perhaps as a by-product of allowing the application to download and unpack itself, that unlocks the ability for a shell script to do likewise.

Last edited by Bloke (2024-08-21 14:35:47)

bici · 2024-08-21 17:26:39

gaekwad wrote #337695:

I thought I felt my ears burning…

Interesting idea. The auto-installer is a solved problem and battle-tested, but there’s a whole level of other complexity with an upgrade script, including:

Less than 100% visibility on what level of shell access is available

Less than 100% visibility on what apps are available (rsync might be limited, for example)

Less than 100% visibility on ownership permissions

Less than 100% visibility on what assets (files, directories) are to be kept (especially if someone is using /textpattern/some-custom-directory/etc/

Responsibility for backups

It might be easier if we have a much more modular & predictable file + directory structure with v5, but right now it scares me to even try undertaking this on a third-party site.

That said…Wordpress can do the whole thing from the admin-side, so it’s not beyond the realms of possibility. No shell stuff required.

It was just a thought, albeit not a well thought out one.
All these years of updating have been fairly pain free and always successful. And the ones time I had to roll back that was also successful due to the manual process and backing up of the DB PRIOR to updating. Please set aside for now. Best to concentrate on all the other goodness that the devs are working on. Hope for the Future.

gaekwad · 2024-08-27 11:48:49

Manage admin themes in a similar way to front-side themes.

phiw13 wrote #337727:

Meanwhile, you can hide it in a custom.css inside the admin theme in use:

This would be straightforward enough with a front-side theme, but admin themes are a different beast.

jakob · 2024-08-27 22:04:55

gaekwad wrote #337734:

Manage admin themes in a similar way to front-side themes.

This would be straightforward enough with a front-side theme, but admin themes are a different beast.

Actually it is as simple as that for the admin themes. You just have to remember not to accidentally overwrite your custom.css when updating textpattern. If /admin-themes and other user-supplied data were to removed from /textpattern folder (as discussed on GitHub), that risk would be much reduced.

phiw13 · 2024-08-27 23:58:23

jakob wrote #337736:

Actually it is as simple as that for the admin themes. You just have to remember not to accidentally overwrite your custom.css when updating textpattern. If /admin-themes and other user-supplied data were to removed from /textpattern folder (as discussed on GitHub), that risk would be much reduced.

Well, not completely as easy as for a front-end theme as you need sFTP access to edit that stylesheet. But then one could argue that modifying the admin-theme in use is something only a more high-level user would be doing / allowed to do.

That said, yes moving admin-themes/ out of /textpattern/ would be a good-thing-to-do. It is already possible for /textpattern/plugins/ as a user configuration.

gaekwad · 2024-09-10 11:27:52

Consider viability of offering language packs as downloadable from UI rather than including all in distribution.

Considerations:

Has benefit of being updatable between versions
Could follow similar approach to RPC without being RPC
Could use self-hosted mirror of Textpacks repo, and / or use Textpack repo directly (with limits)

Bloke · 2024-09-10 20:41:31

gaekwad wrote #337825:

Consider viability of offering language packs as downloadable from UI rather than including all in distribution.

I’d be nervous about doing this precisely because it was a royal pain before. And that was on a server we controlled! Offloading it to somewhere like GitHub as a third party source would be sketchy, imo.

Has benefit of being updatable between versions

Hardly anyone used this feature (might have been because it ddn’t work very well, haha). The only languages this is useful for are the ones that aren’t at 100% on release day and if translators do some translation after a release. And if they do, the packs are all available from crowdin anyway so it’s a cinch to download one and install it by hand. BUT there’s one major annoyance with this approach and it comes from…

Could use self-hosted mirror of Textpacks repo, and / or use Textpack repo directly (with limits)

No matter where we source the strings, they need version controlling and tying to a Txp release. That’s a lot of work.

When someone on a pre-4.7.0 Textpattern asked the (then RPC) server for the latest Textpack, it fetched the latest bundle for that language. That pack included all strings, for past present and future releases, because the packs are a single remote resource: one per language.

So imagine we’re trundling towards Txp 5, which is going to be quite a departure from Txp 4.x. We make a change to a string in en.txt et al. Say, we decide to call Forms Partials instead. A few on-the ball translators also make the changes. Some don’t (yet). Someone running 4.9.0 clicks “get me the latest pack”, the server sends it, and their Presentation menu now has Themes, Sections, Pages, Partials, Styles in it. All the docs still refer to Forms. All the tags do too. But the UI is out of step because the string doesn’t belong there.

When we had the RPC server, we were constantly making new strings that did the same (or similar) to existing strings simply because we couldn’t trample on older string values in case someone grabbed a pack between releases. When we moved to bundling packs with releases, that issue went away, at the expense of a larger byte count overall and the inability to change your language strings between versions (unless you’re careful and can manually merge the changes).

In order for this to ever happen, we need some way of hosting remote strings such that:

Translations are targeted to specific Txp versions, so any extra translation work that someone does between releases is put in the pack specifically for that release and it can be flagged as updateable from the UI.
The Languages panel knows to only query and download the latest strings that match the installed Textpattern version, not earlier, and definitely not future, beta, or unreleased strings.
Translators don’t have to translate the same string multiple times into multiple packs, because their time is precious as it is.

All the above is a lot of extra effort; new processes and procedures for translators to follow, new scaffolding, new scripts to help ease the translation effort, new code to manage it all… and frankly, the benefit to users is only minimal.

I talked to Phil a while ago about tagging strings with releases, so we can maybe periodically ‘release’ updated Textpack bundles mid-Textpattern-release-cycle, but neither of us had the capacity to look into it at the time with any great depth. Not sure how geared up Crowdin is for this. The main stumbling block I foresee is finding a way of letting translators convert strings once, and have the strings pushed to the correct version bundles. That’s most non-trivial.

jakob · 2024-09-12 13:06:29

Another idea, born of a current request:

Expiry dates for files like we have for articles.

A current use case: A simple setup for irregular job openings.
The files are posted as PDF files with the job title in the file title and small notes like due date etc. in the description. The files are assigned to a category and then either set to hidden or removed from the category afterwards. Ideally an expiry date + time could be set after which the file no longer shows.

I’m sure there are other cases where something is made available for download for a specific period of time and then removed again, e.g. “free download for the next week”.

Bloke · 2024-09-12 13:47:00

jakob wrote #337828:

Expiry dates for files like we have for articles.

This is a great idea. We have the concept of a created date already for files. Not sure if that’s honoured or not in the tags (I think it is: <txp:file_download_list> has a time attribute) but if it is, it doesn’t feel too taxing to add expiry, maybe even squeezing it in 4.9.0.

We’d need:

a new expires column in the txp_file table.
some UI furniture to display/alter it in the file edit step.
a multi-edit option to either “Expire now” or (perhaps better) ‘Set Expiry to … [Date box]’ which defaults to ‘now’. In fact, we could do with this on the Articles panel too.
update the file tags to only list and operate on appropriate files within the created-expires window.
bonus points: some sort of control (pref or tag attribute?) that can dictate whether to consider expired files and what to do with them. Articles throw a “410: Gone” because they’re tied to the URL. Files aren’t, but it might still be nice to allow them to be included in the list so maybe they can be styled (e.g “Sold out” if it was related to some kind of event.
anything else…?

Textpattern CMS

Textpattern CMS support forum

#13 2024-08-21 13:25:37

Re: RFC: Textpattern 5 ideas & feature requests

bici wrote #337689:

#14 2024-08-21 13:50:35

Re: RFC: Textpattern 5 ideas & feature requests

Bloke wrote #337694:

#15 2024-08-21 13:54:01

Re: RFC: Textpattern 5 ideas & feature requests

#16 2024-08-21 14:29:01

Re: RFC: Textpattern 5 ideas & feature requests

#17 2024-08-21 17:26:39

Re: RFC: Textpattern 5 ideas & feature requests

gaekwad wrote #337695:

#18 2024-08-27 11:48:49

Re: RFC: Textpattern 5 ideas & feature requests

phiw13 wrote #337727:

#19 2024-08-27 22:04:55

Re: RFC: Textpattern 5 ideas & feature requests

gaekwad wrote #337734:

#20 2024-08-27 23:58:23

Re: RFC: Textpattern 5 ideas & feature requests

jakob wrote #337736:

#21 2024-09-10 11:27:52

Re: RFC: Textpattern 5 ideas & feature requests

#22 2024-09-10 20:41:31

Re: RFC: Textpattern 5 ideas & feature requests

gaekwad wrote #337825:

#23 2024-09-12 13:06:29

Re: RFC: Textpattern 5 ideas & feature requests

#24 2024-09-12 13:47:00

Re: RFC: Textpattern 5 ideas & feature requests

jakob wrote #337828:

Board footer