Queries about the 4.8 plugin regime

gomedia · 2021-06-22 04:37:02

As far as I can tell, when a new plugin is installed both the database and the TXP filesystem are updated.

in the ‘txp_plugin’ table, the data storage seems to mirror previous releases of TXP.
in the new textpattern/plugins directory, each plugin has a self-named subfolder containing my_plugin.php (plugin code), help.textile, textpack.txp, help.html and manifest.json.

And, if I’ve got it right, the website loads the plugin code from file rather than the database.

Some observations:

I had a situation where I was upgrading a live website from 4.7 to 4.8 and the textpattern/plugins directory didn’t get updated correctly. This resulted in the plugin code in txp_plugin being a newer version than its code file. Took me a while to work out what was going on. There were no warnings in Diags or the Plugins tab.
If I delete one of the plugin subfolders from the filesystem, TXP recreates the plugin subfolder (apparently on the next, public or admin, page view) however it only contains the plugin code file e.g. my_plugin.php. Again, there are no warnings in Diags or the Plugins tab about missing files.

So for (1), should/could there be warnings in Diags about inconsistencies between the plugin code in the DB and the equivalent on file?

And for (2), I guess it’s not a big deal as plugins installed prior to 4.8 lack the extra files as well.

etc · 2021-06-22 08:48:17

The only reason a code copy is still stored in db is plugin composer compatibility. So

have you edited and saved your code via plugin composer? It does not update the disk version yet.
once plugin composer is patched, we will remove the db version, storing only a backup copy. Will check re help and meta data, thanks.

Bloke · 2021-06-22 09:22:50

etc wrote #330595:

The only reason a code copy is still stored in db is plugin composer compatibility.

Really? I wasn’t aware of that, sorry. Plugin Composer hasn’t worked for ages.

once plugin composer is patched, we will remove the db version, storing only a backup copy.

Will we? Taking a database dump and moving it to another server ought to be able to recreate the entire system; minus images/files because they’re impractical to store in the database. This includes plugin code.

I don’t want to lose the code in the db. For one thing it allows us to detect if the code on disk has been tampered with. Where are you thinking of storing this ‘backup copy’?

Since plugins reside within the /textpattern dir at the moment, when the filesystem is backed up/copied to another server, plugins come along for the ride. I don’t like that the plugins dir is here. I want people to be able to choose where that dir is and (potentially) move it out of docroot.

Once we figure out the best way to separate plugin code/help/textpack (the “executable”) from plugin support code (e.g. libraries/css/js that a plugin may need and must therefore be in docroot) that can become reality. Then, having the code and metadata in the database will be of more use because it can – in theory – recreate the entire plugins directory on disk from all the data it holds.

I’m not entirely sure how to handle that. We have talked about a public area – something that works similar to the multi-site layout – and it has merit.

Will check re help and meta data, thanks.

Yes please. I spotted that a while ago but figured it was too much hassle to recreate all the relevant disk files from what we hold in the database – since we don’t necessarily know (for old plugins especially) which lang strings they use. For newer plugins that have textpacks bundled with them, they get stored in the textpack column so I think everything can be recreated from there.

One caveat: I know Adi doesn’t do this, and handles all translations separately. So there’s no way to get the .textpack file from his plugin metadata and put it on disk. It would rely on trawling the txp_lang table and guessing which strings belong to which plugin based on prefix/owner/event.

etc · 2021-06-22 09:32:16

Bloke wrote #330597:

Taking a database dump and moving it to another server ought to be able to recreate the entire system; minus images/files because they’re impractical to store in the database. This includes plugin code.

Not really, since plugins can include multiple files/assets, and we don’t know how to pack them together.

Where are you thinking of storing this ‘backup copy’?

There is code_restore field in txp_plugin table. But again, it only stores the ‘main’ php file.

Bloke · 2021-06-22 09:45:16

etc wrote #330598:

Not really, since plugins can include multiple files/assets, and we don’t know how to pack them together.

At the moment, yes.

If we split “core” plugin code (“executable” stuff) and “user” plugin code in the filesystem, then the core plugin stuff itself can be recreated. All other stuff is ‘public-facing data’ and falls into the same band as images and files. If they’re all under a top-level /public or /shared directory, then that’s the front-facing stuff which needs to come along for the ride, by necessity.

Think about testing a new version of Txp. You backup your DB, backup your filesystem. Copy the database to a new server, copy your user filesystem over (images/files and any custom dirs like /css or /js) to the new location and then upload the latest Txp files from the repo.

What’s missing? The textpattern/plugins dir. Today, you need to extract that from your old server and copy that up by hand. It’s partially recreated on login to the new system, but not completely. If the user portion of the plugin was in /public (or /shared or whatever) and the system portion of the code was in DB metadata, the system /plugins dir (wherever it is told to reside by the admin) can be recreated without having to remember to copy it over from inside the core filesystem.

There is code_restore field in txp_plugin table.

Ah, okay. I misunderstood, sorry. I thought you meant eradicating all code from the DB and storing a ‘disk-based backup copy’. Yes, in that case, one single copy of the code in the database is fine. The ‘diff’ can be computed from disk vs this DB copy to determine if one or the other has changed.

Last edited by Bloke (2021-06-22 09:47:35)

gomedia · 2021-06-22 10:27:03

etc wrote #330595:

The only reason a code copy is still stored in db is plugin composer compatibility. So

have you edited and saved your code via plugin composer? It does not update the disk version yet.

once plugin composer is patched, we will remove the db version, storing only a backup copy. Will check re help and meta data, thanks.

Hi Oleg, thanks for the reply.

I don’t use plugin composer.

I think what happened is that my FTP client got it’s knickers in a twist & failed to update some of the plugin files. So I ended up with one version advertised in TXP admin (i.e. in the database that had been restored) but an older version actually being loaded from the filesystem.

etc · 2021-06-22 11:09:42

Bloke wrote #330597:

Really? I wasn’t aware of that, sorry. Plugin Composer hasn’t worked for ages.

I’m wrong, sorry. The db copy serves mainly for migration from old db to new fs storage. Still, it would be nice to update plugin composer, it’s really handy for small plugins distribution. Or even import its features to core?

If the user portion of the plugin was in /public (or /shared or whatever) and the system portion of the code was in DB metadata, the system /plugins dir (wherever it is told to reside by the admin) can be recreated without having to remember to copy it over from inside the core filesystem.

I’m fine with moving plugins to some public subdirectory, but why would we separate the ‘system’ portion from the ‘user’ one? Public-facing data could actually be another php file/library required for plugin functioning.

gomedia wrote #330603:

I ended up with one version advertised in TXP admin (i.e. in the database that had been restored) but an older version actually being loaded from the filesystem.

Yep, a warning would make sense, though this sync problem would be gone if we removed the code from db.

Bloke · 2021-06-22 12:24:01

etc wrote #330604:

it would be nice to update plugin composer

I made some sweeping changes to it in a version I never finished (to do with string management and owner/prefixes), then along came the file storage system and I didn’t have the time to unpick what I’d done already to bake in the file system support, so I kind of forgot about it.

Or even import its features to core?

The distribution side of things would be helpful. Like a save as txt/export feature. But we’re lacking a few key components in core:

The ability to associate lang strings with a plugin (for bundling up) into a release.
The ability to edit metadata like plugin type, and lifecycle/prefs registration.
The ability to write docs.

The last one is potentially easy. Just add an ‘edit’ link alongside the help link. That could open up a single text area exactly like the code edit process with save button.

The second one could become dropdowns in the code edit panel. Or the dropdowns, code and help textareas could all be displayed on one ‘edit’ panel instead of editing the help text separately. But do we really want admins being able to change the plugin lifecycle/prefs registration? The plugin type, maybe(ish) but donkeying with the lifecycle stuff has potential to cause major breakage.

As for bundling strings up for distribution, that’s more difficult. Plugin Composer cheats and allows you to associate a string prefix/owner per plugin, which it then uses to figure out which ones to bundle. It also uses a separate global pref to state which languages you want by default, and a per-plugin pref during export to choose which languages to compile into the exported code.

With no mechanism to actually translate strings (besides smd_babel) this side of things feels incomplete for a core feature. If we can put mechanisms in place to support this, then potentially yes some of the functionality could be migrated. And once support for plugin cache is better integrated in 4.9. (hopefully) then it becomes even more useful.

But at the moment I’m kind of leaning more towards keeping this as a plugin. Open to ideas.

I’m fine with moving plugins to some public subdirectory, but why would we separate the ‘system’ portion from the ‘user’ one? Public-facing data could actually be another php file/library required for plugin functioning.

Yes, exactly. That’s why the current storage system for metadata in the DB is not good enough. We have the ‘data’ column, which is unpacked into the system plugin area, and that’s it. No notion of what the code or data is used for.

We could have a way of saying ‘this part/file/info/ requires public consumption’ so it can be diverted to a suitable area on disk under docroot. Otherwise it’s put in the system plugin area.

Alternatively, could everything be put in one (user-specified) dir for simplicity – and if that happens to be web-accessible then admins strongly encouraged to protect it with .htaccess or Nginx equivalent (warning in Diagnostics and Plugins panel if not).

Then, if the location is not web-accessible due to either physically not being there, or being prevented by .htaccess, is there any way we could fudge it so that a call to a well-known URL endpoint like /plugin-data/abc_plugin/js/somelib.js could check if that asset is flagged as “for public consumption” and if so, go and fetch the (inaccessible) content from the file system and present it to the web server as if it was a real file in that location?

Last edited by Bloke (2021-06-22 12:28:26)

etc · 2021-06-22 12:54:20

Bloke wrote #330605:

But at the moment I’m kind of leaning more towards keeping this as a plugin. Open to ideas.

That’s fine, just it would be great to update it so it saves the fs copy too.

Alternatively, could everything be put in one (user-specified) dir for simplicity – and if that happens to be web-accessible then admins strongly encouraged to protect it with .htaccess or Nginx equivalent (warning in Diagnostics and Plugins panel if not).

Then, if the location is not web-accessible due to either physically not being there, or being prevented by .htaccess, is there any way we could fudge it so that a call to a well-known URL endpoint like /plugin-data/abc_plugin/js/somelib.js could check if that asset is flagged as “for public consumption” and if so, go and fetch the (inaccessible) content from the file system and present it to the web server as if it was a real file in that location?

Let’s face it – even textpattern directory is open to web access by default (you can e.g. access textpattern.js or even existing php files without being logged in). We provide .htaccess-dist for use in plugins directory, but that’s all.

gomedia · 2021-06-22 13:19:35

Sorry! Did I open a can of worms?

Bloke · 2021-06-22 14:09:25

etc wrote #330607:

Let’s face it – even textpattern directory is open to web access by default

True. But we protect all the core (PHP) files with a no-direct-access-here wrapper. Plugins aren’t covered by that.

It’d be really sweet if plugins could benefit from some built-in protection like our core ‘no txpinterface defined’ if you try to access them directly. That would cut down on people filing CVEs because a site hasn’t implemented .htaccess in the /plugins dir, but that’s an aside. I did wonder if it’s possible to prepend those two lines to the plugin file when it’s included but that’s probably too invasive/slow. And doing it on installation might be tricky.

Anyway, yeah, you’re right. Let’s skip my random /public ramblings for plugins. We should just focus on allowing people to move the plugins dir where they want it, the same way they can move the /files dir. Then ditch the code column in the DB (or the code_restore, whichever makes most sense) and fix up the plugin import/install, so we have just one copy of the code in the DB table. Then everything just works and we can diff the file system with the stored DB code to detect changes for the (Modified) column.

We can then start to build the update mechanism for the Plugins panel directly from plugins.textpattern.com, and get more plugins registered there.

etc · 2021-06-22 14:28:52

Bloke wrote #330609:

It’d be really sweet if plugins could benefit from some built-in protection like our core ‘no txpinterface defined’ if you try to access them directly.

I was thinking of renaming them to, say, .inc instead of .php, but this requires a massive code rewrite. But yes, the current protection is fragile.

Textpattern CMS

Textpattern CMS support forum

#1 2021-06-22 04:37:02