Situation report for Textpattern 4.6 and beyond

DragonBard · 2014-07-01 03:54:00

You are probably wondering how to handle select boxes and radio buttons with the above setup. Simple, this is where the hierarchy comes in. The implementer defines a “ComboBox” type that provides the base wrapping code, it calls on a sub-type to provide the items. The sub-type handles wrapping the individual items. IF you care about controlling the order, then you will have to provide some way to control that, most likely an additional table for maximum flexibility.

DragonBard · 2014-07-01 04:10:09

DragonBard wrote #281776:

4. You can get what you are after with far fewer tables. Don’t fall for storing data as it’s “true” data-type. Just store it as a reasonable length string, with appropriate linkages to where used. Use a hierarchical (aka self-referential) table to store the custom field types in their registry. The registry should consist of an ID, name, possibly a label, and help tip. Here is the fun part. Add two to four large fields, possibly text. The purpose of the fields is to provide the code to handle the admin side and published side view of the custom fields (two fields if no JavaScript, four fields if you will allow JavaScript). Let the implementer worry about how to interpret, verify, format, spindle, mutilate, and fold their own data. In other words, use the exact same idea as textpattern forms, in a smaller scale. The domain you should be implementing is “custom field”. You are currently implementing a custom SQL table editor.

5. You should be able to implement core custom fields with about 4 tables. Registry, Hierarchy, FieldValues, and FieldUse. Of course, modifying it to fit textpattern and adding security will grow the solution set. I’m only just beginning in using textpattern, but I built a similar custom field implementation back in 2001 in classic ASP, and it was plenty fast, even with loading the display handler code from the database for every page view. Cache the handlers and it should be plenty fast. That was a project for work, which died due to lack of interest.

By now, my suggestions may be too late. Take them in the spirit of cooperation they are intended and do with them as you will.

DragonBard.

Thinking on this some more, the code should be in a separate table so that the implementer can provide different code for different contexts, say “default”, “article”, “image”, “article_listing”, etc. This should take care of the desire to have one custom field definition apply to multiple textpattern contexts.

By letting the implementer specify the code, this allows the implementer to define formatted fields, such as telephone number, with whatever formatting, verification, sanitization, and storage semantics they desire, without textpattern having to guess it all ahead. All you are really after here is specifying what textpattern will replace the <txp:custom_Field /> tag with. Keep the data storage simple, and let the custom field implementer worry about storage and formatting. The custom field implementer shouldn’t expect the moon, so don’t promise it to them. That’s what plugins are for.

DragonBard.

Bloke · 2014-07-01 09:03:29

DragonBard wrote #281776:

Relations are multi-dimension, not 2D.

Point taken, but it’s ends up being semantics. The relational model was invented to take advantage of disk-based storage over tape, but retained the notion of storing data in contiguous blocks to minimise head movement. SQL grew around this and thus pushes us to think in terms of rows and columns because you have to:

Define the data structure (indexed, hierarchical, network, relational, columnar, …)
Put data into the structure
Use the relationships defined by the structure to retrieve data

My point was simply that this structured data storage approach does my head in!

You are falling into the SQL trap of believing the data type is the domain.

Maybe. I’m a novice at SQL. Where were you six months ago when I was thinking this through? ;-)

if you are doing large mailings you might want to separate the city, state, and postal code for presorting purposes, but otherwise don’t bother.

Sure, it’s OK to modge it into one field for rudimentary applications, but most of the time I happen to need to perform tax calculations, or offer delivery options based on location. City, Postcode, Country are all important separate entities in those cases as you say, so I tend to apply that notion across the board to other systems. My reasoning is that, invariably, the question comes up further down the line “how can I bill my customers for…” and I don’t like having to re-engineer things.

You can get what you are after with far fewer tables.

No doubt.

Don’t fall for storing data as it’s “true” data-type.

Noted. I’m kind of abstracting it away from the user, but essentially it’s not using a true representation, just a ‘Best Fit’. People don’t care if their Yes/No radio button data is stored as a tinyint, int, varchar, or text, but they’ll damn well care when they have a million records and it takes fifteen seconds to extract the information. That (and space concerns) are my motivation behind using an appropriate data representation for the ‘type’ of data stored, nothing more.

The registry should consist of an ID, name, possibly a label, and help tip

Yes, with the exception that we’d forego the literal label and tool tip in lieu of names that can be referenced for l10n purposes. We’re also looking at date stamping fields with at least a ‘created’ date. Preferably an expiry too if I can figure it out: just one small implementation detail to iron out there at present.

Here is the fun part. Add two to four large fields, possibly text.

If using Text, the table would not be able to be cached by the database engine, so every request would be to disk. Not my definition of ‘fun’.

You are probably wondering how to handle select boxes and radio buttons with the above setup. Simple, this is where the hierarchy comes in… IF you care about controlling the order, then you will have to provide some way to control that, most likely an additional table for maximum flexibility.

Order is important. As are default values for all fields.

By letting the implementer specify the code, this allows the implementer to define formatted fields, such as telephone number, with whatever formatting, verification, sanitization, and storage semantics they desire

I like this idea, and it did cross my mind as a potential nice-to-have. To that end, at the moment all fields are wide open (because as you say we can’t second guess anyone), but my aim is that plugins will be able to step in and add data constraints to each field if desired. Textpattern offers very little validation (in terms of data, at least. It checks rudimentary things like valid categories, section names, dates, and such like, but doesn’t validate custom field content, image names, description, etc) so I’d probably keep it that way and leave specifics up to plugins.

At the moment, plugins can override the built-in storage mechanism mapping and augment them by adding new widget types. Txp handles the back-end of managing the physical tables on your behalf.

Thank you for taking the time to post. I can see where you’re coming from and they were my initial thoughts too. It’s just that the more I thought about it, the uglier it became to stuff everything into a varchar and hope it scales well.

MySQL is not the fastest database in the world and needs all the help it can get from well-designed implementation “hacks” to help it do its job of delivering data as fast as possible. Sure, it adds application code complexity but my goal is to bridge the gap between having to burden users with the nitty gritty of understanding data types at the database level while still delivering performance that scales into the millions of rows, wherever possible.

I hope I’m on the right track…

DragonBard · 2014-07-01 13:49:05

Bloke wrote #281789:

Maybe. I’m a novice at SQL. Where were you six months ago when I was thinking this through? ;-)

I was working on an Exchange 2010 migration. I hadn’t even heard of Textpattern at that point. Now that the migration is done, I decided to maybe do some recreational computing. I used to do database and web app development work, but my true talents are in Systems and Network administration, so it has been years since I last played with anything database or web.

I’ve never liked MySQL, and never understood why so many think it is so great. As you say, it does not scale well and seems to introduce problems to every project that uses it. I’ve also never like PHP, which is why I like Textpattern so much. I can do nearly anything I want without ever having to touch PHP (unlike WordPress). I very much like the <txt… /> tags and how they work. If I got too involved in development, you would end up with Textpattern rewritten as a .NET app that maintains textile and the txp tags, but not much else.

My main point for this discussion is that it is far too easy to get mired in attempting to map your problem onto SQL datatypes when what you need to do is map SQL datatypes onto your problem. Also, the domain of your data often will not match SQL datatypes at all. Normalization does NOT imply deconstruction of your data into base datatypes. Normalization is all about the domain of your data. One must look beyond datatypes to how the data will be used and manipulated.

I really like Textpattern. I want to see it advance. I’m also highly opinionated and don’t mind too much being ignored.

Thank you for considering my input.

DragonBard.

gaekwad · 2014-07-01 13:52:53

DragonBard wrote #281808:

I really like Textpattern. I want to see it advance. I’m also highly opinionated and don’t mind too much being ignored.

Welcome aboard, sir/miss.

Bloke · 2014-07-01 14:30:51

DragonBard wrote #281808:

it is far too easy to get mired in attempting to map your problem onto SQL datatypes when what you need to do is map SQL datatypes onto your problem.

Amen to that! I’d love to be able to use varchar for all data as it’d make everything so much simpler, but MySQL has kinda forced me to think differently, for better or worse. btw, I’ve no idea how this’ll work when we move away from sole MySQL support to using something like PDO.

Also, the domain of your data often will not match SQL datatypes at all.

Yeah, that’s a pain too. Storing complex objects seems natural when stuff is related (your address example, for instance), but the SQL paradigm is one-field-to-one-piece-of-data which forces you to either explode your thinking, use a large/blob field type, or implode your object to a serialized text form. One creates additional work, the next is resource-expensive, the other is frowned upon by database stalwarts as being “wrong”. No-win.

One must look beyond datatypes to how the data will be used and manipulated.

Yes. But since MySQL treats datatypes differently under the hood, we must pander to its storage anomalies or risk creating a lumbering behemoth. Striking the balance between the user experience and doing things for the sake of making it ultra flexible is where I’m trying to position this implementation. That is, adopt some sensible convention over configuration to ease the burden on both sides.

I really like Textpattern. I want to see it advance. I’m also highly opinionated and don’t mind too much being ignored.

Glad you like it. We do :-)

And please keep throwing ideas around and challenge our thinking. Even though you’re not a PHP fan, feel free to chuck patches and stuff this way. When we move to GitHub we’re hoping that process will be even simpler. I sincerely hope you don’t feel you’re being ignored, either now or in future. I don’t profess to hold the answers (heaven forbid that day!), I’m just trying to bring something to fruition that I hope addresses a long-standing feature request and add something which (currently, afaik) no other CMS can do to any wholly satisfactory degree. They’re all either far too limiting or way too complex to configure.

Believe me, I need all the help I can get…

candyman · 2014-09-22 08:25:47

I’ve discovered gaekwad’s TXP demo site and tryed out the 4.6 dev.
Apparently itss similar to current 4.5 with a more clean/user friendly admin interface.
Anyway it seems almost finished and I don’t see any major improvements (media management and so on) from TXP 4.5.x.

So, if can I ask, what is the release problem?

Must we wait for some brand new features or are there problems or other kind (plugin compatibility and so on)?

Just to know, TXP 4.5.7 is already excellent for my needs.

Last edited by candyman (2014-09-22 08:30:02)

philwareham · 2014-09-22 08:42:47

There is no release problem, we are just working on several different pieces of the system, none of which are ready for release as yet.

For example my admin side UI changes are being worked on independently of the core, and will only be rolled in once I’m happy with them, and they are stable enough.

In terms of codebase changes version 4.6 probably has the most extensive revisions in Textpattern’s history, even if some of those are not immediately apparent to end users at this point of in time.

sacripant · 2015-02-13 12:47:54

Hello core team !
Some news about Textpattern 4.6 ?

dos · 2015-02-14 11:15:40

Yeah, would be great to get some insights into the developing process. A new blog post might be a good idea. Even if there are no big news, it’s always nice to hear from the developers.

CeBe · 2015-04-28 21:01:39

I just spent a long time searching the forum on how to contribute. It’s not very clear – at least for me.

There’s a Github repository. Ok.
I forked it and cloned my fork. Ok.
Then I tried the readme file, it describes installation for the end-user. Then the contributing file, which gives short instructions.

I didn’t find any practical information, such as

who can contribute (anyone, I guess)
and how:
- is it as simple as “clone, chose an issue, make your changes, submit a pull request”?
- are there issues to address first?
- if I’m about to work on one of them, do I have to inform someone (who? how?) or even ask for permission?

A simple but explicit guide is missing and my feeling is that it can prevent people from daring to contribute.

Last edited by CeBe (2015-04-29 06:24:14)

wet · 2015-04-29 03:53:59

CeBe wrote #290276:

I didn’t find any practical information

Thanks for the pointers, I’ve added few bits there. Is there anything else we could do to lower the entry barrier or make the project more inviting?

CeBe · 2015-04-29 06:39:24

wet wrote #290277:

I’ve added few bits there.

Yes :) Thank you! (Of course I forked before cloning, I edited my previous message)
Help with translations is a good point.

So here comes another question: is it really necessary to create a new branch? Fork+pull request seems sufficient and easier to manage, or do I miss something?

philwareham · 2015-04-29 08:00:09

You can just fork and pull if patch is fairly minor. A new branch is suitable for more extensive changes.

CeBe · 2015-05-01 10:11:25

philwareham wrote #290283:

You can just fork and pull if patch is fairly minor. A new branch is suitable for more extensive changes.

I may be picky (sorry…): what criteria determine what is “fairly minor”, or “more extensive”?

A good practice may be to:

look into the issue if it is identified as minor or not, or even if it is clearly mentioned to create a new branch
if not (or if in doubt), create locally a new branch, and merge (or rebase?) if it finally happens to be a minor change.

I don’t feel very comfortable yet with git, I hope I haven’t written foolishness.

Textpattern CMS support forum

#91 2014-07-01 03:54:00

Re: Situation report for Textpattern 4.6 and beyond

#92 2014-07-01 04:10:09

Re: Situation report for Textpattern 4.6 and beyond

DragonBard wrote #281776:

#93 2014-07-01 09:03:29

Re: Situation report for Textpattern 4.6 and beyond

DragonBard wrote #281776:

#94 2014-07-01 13:49:05

Re: Situation report for Textpattern 4.6 and beyond

Bloke wrote #281789:

#95 2014-07-01 13:52:53

Re: Situation report for Textpattern 4.6 and beyond

DragonBard wrote #281808:

#96 2014-07-01 14:30:51

Re: Situation report for Textpattern 4.6 and beyond

DragonBard wrote #281808:

#97 2014-09-22 08:25:47

Re: Situation report for Textpattern 4.6 and beyond

#98 2014-09-22 08:42:47

Re: Situation report for Textpattern 4.6 and beyond

#99 2015-02-13 12:47:54

Re: Situation report for Textpattern 4.6 and beyond

#100 2015-02-14 11:15:40

Re: Situation report for Textpattern 4.6 and beyond

#101 2015-04-28 21:01:39

Re: Situation report for Textpattern 4.6 and beyond

#102 2015-04-29 03:53:59

Re: Situation report for Textpattern 4.6 and beyond

CeBe wrote #290276:

#103 2015-04-29 06:39:24

Re: Situation report for Textpattern 4.6 and beyond

wet wrote #290277:

#104 2015-04-29 08:00:09

Re: Situation report for Textpattern 4.6 and beyond

#105 2015-05-01 10:11:25

Re: Situation report for Textpattern 4.6 and beyond

philwareham wrote #290283:

Board footer