Semi-persistent general storage plugin

Bloke · 2008-05-28 10:18:18

To go hand-in-hand with the recent introduction of smd_query, some planned enhancements to smd_if and the excellent new <txp:variable /> tag, I started work a few days ago on a little plugin that might be of benefit. After a discussion with uli yesterday I think I’ve slotted some more pieces of the puzzle into the plugin and wanted to run it by you lot to see if there’s interest before I go any further.

Under the tentative name smd_vars it’s essentially a new table in the database that allows you to store and retrieve arbitrary name-value pairs. So:

You can add/remove/update fixed variables and values on a back-end extensions tab
You can get any value on the public side via <txp:smd_var_get /> that ~~either~~ matches a name ~~or its value~~. Thus with the new tag parser in TXP 4.0.7 you can inject values into other tags. Without a ‘form’ (or container) this just returns the single value but with a form it passes the value back for use elsewhere. If more than one value matches, it could pass each one back to allow you to iterate over them, but I don’t know if that’s useful
The ‘get’ also allows further injection from values available to the page (that includes <txp:variable /> definitions plus all the usual suspects like custom field contents, keywords, or url variables that are subject to variable encoding to minimise malicious use). You specify placeholders in your smd_var and pop the supplied values in when you get them
You can also put any value you like back into a variable via <txp:smd_var_put /> subject to usual variable encoding to lessen potential malicious (mis)use. You can optionally set the variable to expire, which leads to some interesting session-based pages. If the variable exists it is updated, if it doesn’t exist it is created. If you specify delete it is deleted (there is a dedicated tag for deletion as well)
Conditional testing can be done via smd_if under the new parser. There ~~may optionally be~~ is a rudimentary conditional tag added in this plugin but it may not be necessary because <txp:if_variable /> will be your best friend: you could “load” the variable you smd_var_get into the <txp:variable> tag and then test its value/existence later using the core’s built-in variable conditional
Conditional testing on a variable’s status is possible in various ways. smd_var_status is the grand-daddy that allows you to ask for one or more bits of status info to be returned either as text or an equivalent status number. Three are built in right now: 1) available (is the var defined at all); 2) valid (the default: is the variable within its expiry time); 3) Used (does it contain a value or is it empty). These are useful for plugging into smd_if. Next up is smd_var_if_status which is a simple conditional that allows you to take action on one variety of status at a time. Then there are three “convenience” functions for specific conditional testing: smd_var_if_valid, smd_var_if_used, and smd_var_if_available
I haven’t decided if the table benefits from a “type” yet. It probably complicates it more than necessary

A novel use of the system would be form-based querying, of which uli is quite a fan. You could store queries — like stored procedures — in the new table with placeholders for data. When a user submits a request via an HTML form you can grab the relevant “stored procedure” (could use smd_if or other conditionals to decide which one to use based on the passed input if you like), plug the values from the form into the placeholders and throw the lot at smd_query, then iterate over the results via a TXP form. You could even take bits of the results and store them in other smd_vars if you like or put them in local <txp:variables /> for later use.

That’s an extreme example, there are far more benign and mundane uses for it (a state machine?), but before I go too far with development I’d like some feedback on whether anyone thinks this is a good idea. Or is it dangerous? I understand the implications of allowing write access to this table but the good news is it’s a “sandbox” in a sense away from the rest of the database, so as long as any input is sanitized properly it should be fairly safe from injection attacks if you’re careful.

But I’ll take ideas/criticism at this stage to make sure I do everything right to prevent giving you a live grenade with a dodgy pin. So for those of you with advanced knowledge of this kind of stuff, please get in touch and help me out. If the result of feedback means curbing some of the functionality or dropping the idea entirely, so be it.

The floor is now yours. Discuss.

Last edited by Bloke (2008-06-05 18:54:05)

mrdale · 2008-05-28 15:17:31

I always thought that the ability to use tags/variables in css would be useful. Imagine using it for color schemes. ie…

main_color=#fa0
accent_color=#ddd
background_color=#e5e5ff

then use them multiple times in your css and change it once per site, rather than doing the usual hunt and find.

kevinpotts · 2008-05-28 16:33:42

I think this is a great idea. Honestly I would probably not use it for very exciting purposes. The CSS example mrdale cited is reasonable. I guess the closest approximation I do is occassionally store bits and pieces in individual forms, and then call them via <txp:output_form />. Maybe I don’t have a big enough or technically capable imagination.

Last edited by kevinpotts (2008-05-28 16:34:17)

Bloke · 2008-05-28 16:56:41

kevinpotts wrote:

I think this is a great idea. Honestly I would probably not use it for very exciting purposes.

Thanks for the vote. I guess 80% of the time I’d probably not use it for anything fancy, but having essentially a developer-controllable, client-side-updatable prefs table at your disposal — combined with the new tags and parser in 4.0.7 — might make some tasks that little bit easier. Store something on one page, retrieve it a few pages later…

btw, nice TXPQ.com interview. And great contribution to the txp.com home page SEO discussion on wet’s wiki; cool stuff in there.

jm · 2008-05-30 20:28:36

Sounds great Stef! Although it would be a little more complicated, the type field could be beneficial. I think just radio/boolean and text input would be good.

Let me know if you want some help with the tedious admin interface :).

trenc · 2008-05-31 10:08:46

If I figure it out correctly so this would be quasi an mixture of custom_fields, database driven session management?

I think this would be nice for some really needed applications.

Zanza · 2008-05-31 15:06:02

Is this idea usable to have “favourite” selection from registered/author users? I.e., a registered user can set certain articles to be his/her “favourite”. So he/she can return back and find them in a “favourite” page. The time-frame for being “favourite” can be variable. A session, or permanent, or until he/she does something (say, purchase)

Is there any way to have this? I suppose this would need adding some more table to the db, and maybe it’s not the original Bloke’s intention, but maybe this could make he thinking about: it’s an extension of his “semi-persistent” idea. Just need to estabilish some relationship with users/articles. Well, maybe not so easy after all, but I’m thinking aloud… :)

Z-

Bloke · 2008-05-31 15:45:39

Zanza wrote:

Is this idea usable to have “favourite” selection from registered/author users?

You can certainly use the storage as transactional details and then, on completion of a series of things (say, a successful order) delete them all. When you do a <txp:smd_var_put /> an optional attribute will delete the matching variables.

Thinking about what jm said about the type field, this could be a useful scenario for such a feature since, in essence, ‘type’ could be anything you see fit. In the prefs table at the moment there are fixed things like radio, checkbox, text etc, but if I can figure out a generic way of using them (i.e. some built in “types” like the prefs table but the ability to define more, along with their properties) then you could make a ‘transaction’ type if you wish. It doesn’t have to “be” anything, could just be a way of tagging certain records so they can be more easily purged later. Dunno, I’ll have to sleep on it. Ideas welcome.

a registered user can set certain articles to be his/her “favourite”. So he/she can return back and find them in a “favourite” page.

You could use the system to do this, though cookies might be a better mechanism (albeit giving users more control on whether to accept or reject them).

There’s nothing to stop you setting up a system where the ‘name’ is, for example, “favarticle_zanza” (remember you can create these things on the fly so under TXP 4.0.7 you have the full range of TXP tags with which to grab stuff) and the ‘value’ is whatever article they have indicated they prefer. Or of course, “currarticle_zanza” which, if used in your default article form, could store the current article they are viewing. When they next log in you could retrieve that value and show them the article where they left off, which might be a nice feature for people writing long articles to check their progress (I’ve used rvm_privileged to offer “edit article/edit article image” links on the live site to logged-in users). You could also set the variable to expire so if they didn’t log in for, say, 3 days it “forgets” their preference and thus shows the front page instead.

I still think cookies are perhaps the “correct” way to go about this, but there’s absolutely nothing to stop you using smd_vars to do it if you prefer. Subject to local data retention laws, etc (e.g. Germany).

@trenc: yes, if you want to use it like that. In its base form it’s simply a table of name-value pairs so acts like a generic custom field (more, a “preference” I suppose, but that’s semantics) in which you can store and retrieve stuff in real-time. The name and associated value can be based on any data you can grab from tags, user input, the address bar, yahde yahde. In short, if you can get your mitts on it, you can store it. And you can — optionally — specify a duration for that data to be of use.

It’s important to note that records are not “deleted” when they expire; they simply return nothing when smd_var_get is used. To delete variables you can either call <txp:smd_var_put name="blah" delete="1" /> (because you may very well wish to store ‘nothing’ in a variable) or you use the “purge” feature in the admin interface to delete out-of-date variables. I may well offer an extra attribute to smd_var_get — or a dedicated tag/conditional — that returns the ‘status’ of a record in case you wish to take action based on whether the variable holds ‘current’ or ‘expired’ data for example, or if the variable exists at all. Haven’t thought that through yet, so any feedback welcome.

@jm: I’m definitely considering the ‘type’ field as I mentioned above. If you have any input on what you think might be useful to store in it or what you might like to do with data of different types then please let me know. Use-cases always help cement ideas in my head!

And thanks for the offer of the admin side help. It’s causing me a bit of bother so I may well be e-mailing you if I can’t figure it out… :-)

Last edited by Bloke (2008-05-31 15:50:02)

uli · 2008-05-31 20:24:08

Hi Stef,

the last days I was thinking about a possible scenario, which could be a true relief for tracking down plugin conflicts. Its visible elements might consist of a simple pair of buttons and some (multi-)selects on a TXP dashboard page:

From a ~~select menu~~ multi select (in case of plugin+library twins), you choose the plugin you want to test against, which of course has to remain turned on. After selection is made, you klick the “Go!” button, and an smd_query reads all the status values of the txp_plugin table. In the following step these values are stored in the smd_var table (or maybe vars are sufficient?). Subsequently the original plugin stati would be switched to zero, except for those of the necessary plugins like in this case at least smd_query and smd_vars plus the manually chosen one. You’re now all set to check for suspects on the plugins page (maybe a message could say so).

When you have found the culprit, you go back to your dashboard, choose it from a multi select (which got its information from the previously saved names), and press another Go! to run a further query, which would restore the previously saved settings, but leaving the culprit turned off. (Oh yeah: clean up behind you and delete unnecessary data.)

Last edited by uli (2008-05-31 22:21:32)

wet · 2008-06-01 00:14:46

uli wrote:

the last days I was thinking about a possible scenario, which could be a true relief for tracking down plugin conflicts. Its visible elements might consist of a simple pair of buttons and some (multi-)selects on a TXP dashboard page

A solution with a slightly narrower feature set: wet_plugout.

uli · 2008-06-01 03:19:14

What about if one of these smd_var|query|totally_new things would be capable of doing some maths?

Do some date calculations and find out which article was the most popular one in the period of August ‘til Oktober? Add some px to an image’s width in order to set the total width of a window? Retrieve the entries of several db fields and make a bar chart of it on the fly?

Even if some of you know other ways of realizing exactly the mentioned or can tell why a window shouldn’t be resized or why it’d need a little more than just maths in order to find out the most popular article: there may be other appliances that wait for nothing more than a simple mathematical implementation.

Just a little grain …

Bloke · 2008-06-01 07:01:29

uli wrote:

Hi uli, thanks for letting your mind freewheel a bit. Always good to get some random ideas that might spark obscure things in people’s heads. However…

Do some date calculations and find out which article was the most popular one in the period of August ‘til Oktober?

Date calculations are of course possible with smd_query, though I think it would require diving into the logs table to find out how many accesses each article has had. A neater solution might be rvm_counter to auto incremement a counter. I’ve never used it but it might well be able to count accesses to each article *shrug*. And smd_if can compare stuff to other stuff using >, <, <=, >=, eq and not.

Add some px to an image’s width in order to set the total width of a window?

Not quite sure I follow. You can auto-resize images with lam_dynamic_image but getting the visitor’s browser window size is a client-side problem so is really only in javascript’s territory after the page/DOM has loaded.

Retrieve the entries of several db fields and make a bar chart of it on the fly?

Hehehe, specific-mungous :-) You’re really diving into the realms of custom code instead of a plugin there; a chart plugin that could take tabulated data and draw stuff would be a real challenge! Currently of course, smd_query could get the data but you’d more than likely have to drop into PHP to use the gd library to make a bar chart, so you might as well just code the query in PHP too instead of having another plugin floating around.

I can’t see anything in those ideas that smd_vars can help with (but I have only just woken up so may have missed something). If you think of any ways in which smd_vars could help you or others, or can improve the plugin based on the sketchy details in this thread, feel free to help shape this beast into something people can use. Certainly, mentioning this thread to trenc for use in yab_shop has helped formulate a few more feature ideas in my mind, thank you.

Textpattern CMS

Textpattern CMS support forum

#1 2008-05-28 10:18:18

Semi-persistent general storage plugin

#2 2008-05-28 15:17:31

Re: Semi-persistent general storage plugin

#3 2008-05-28 16:33:42

Re: Semi-persistent general storage plugin

#4 2008-05-28 16:56:41

Re: Semi-persistent general storage plugin

#5 2008-05-30 20:28:36

Re: Semi-persistent general storage plugin

#6 2008-05-31 10:08:46

Re: Semi-persistent general storage plugin

#7 2008-05-31 15:06:02

Re: Semi-persistent general storage plugin

#8 2008-05-31 15:45:39

Re: Semi-persistent general storage plugin

#9 2008-05-31 20:24:08

Re: Semi-persistent general storage plugin

#10 2008-06-01 00:14:46

Re: Semi-persistent general storage plugin

#11 2008-06-01 03:19:14

Re: Semi-persistent general storage plugin

#12 2008-06-01 07:01:29

Re: Semi-persistent general storage plugin

Board footer