Textpattern CMS support forum
There’s been a GitHub issue about unit testing open for a while now. While doing this on the admin-side and catching Ajax stuff requires more thought than just conventional unit testing (as outlined in the comments) there’s nothing to stop us doing tag unit testing on a fresh install right now.
All it really needs(!) is someone to come up with a single article or Form (or series thereof) that contains every tag in it, in as many combinations as possible. This includes:
- Single tags.
- Container tags.
- Tag nesting.
- Core attributes (
- Custom fields – this will unfortunately require changes to a stock install as CFs will need to be assigned to an article to test).
- Shortcodes and short-tags.
- CSS and other content (files, images, links) that will also require known-content to be present in the DB.
The notion is:
- Install latest dev/beta/whatever.
- Load the test article(s) and Form(s).
- View the test article(s) on an empty Page template (so we don’t get any unexpected page furniture output).
- Grab the HTML produced.
- Compare it to a “known good” hunk of HTML obtained from a previous, verified run of that same article.
If the two chunks of HTML match, no regressions have been introduced. The new article and its output then become the new baseline. Next time we change any core tags, we run the test again and compare them to check we haven’t introduced any bugs.
If we add new tags then we need to generate a new baseline (write more test cases in our big article) once we’ve verified that the output is as we expect.
Steve (netcarver) and I did this via a YAML file when we were testing Textile. We had a set of 80+ Textile tests that were run through a simple bootstrap script that loaded up a Textile instance, called
->textileThis() on all the tests and captured the output.
It then compared the results to our gold master HTML and flagged any instances, colour coded green, amber or red based on the severity of the violation, and what the differences were between what we got and what we expected. It also displayed a little table at the bottom with the overall results of the number of tests that passed and failed.
Armed with this, we could be sure that any changes to the parser, surrounding functions, or tag code didn’t inadvertently break anything else.
I’d love to do something similar for Textpattern tags.
This, I understand, is an uphill task for one person and the barrage of tags we have. But as a community, surely we could come up with a series of tests and collate them somewhere, then compile them into a single resource and build a tiny test suite around it. Anyone could then download that periodically – perhaps it also includes a database with the necessary tests and content in it – and verify that everything works as it should.
Let’s say this was desirable (and I think it is), how would we proceed? Where should we collect these tests and package them up?
Perhaps it would be nice to put them in a dedicated directory in the main dev GitHub repo that was excluded when we made a release? Or in the
.github directory where some of our other tools reside? Or a separate repo entirely just for testing, with pre-packaged content? Open to ideas on the best way to proceed.
The point is, having this resource available would be a huge benefit to verify our core wranglings. The fact it only tests the public-facing code is fine right now – we can worry about the admin-side later.
It doesn’t even have to be a full Txp installation. We could build a simple command-line wrapper that loads up a parser and runs the tests, then does a diff against the gold master. Having it run in a browsers is a nice-to-have because we could prettify it.
Maybe it could also run in “benchmark mode” to also output the time it took, or repeat the tests N times, which would be handy for optimizations, as we could use it to detect where the bottlenecks were.
I think we should start doing this. Step 1 is to figure out how to build it or to set up a test environment with known content that is accessible to everyone. Ideas welcome.
Then I’d like to call upon everyone in the community to please submit tag tests – as many as you can think of – that we can add to a huge list that can be used to test and verify the operation of Txp. Everything from the simplest tags with no attributes to more complex tags that stretch the parser. If each tag test could be submitted with its expected HTML output, we could marry the two and start to compile an ultimate tag testing suite.
Whaddya reckon? Thanks in advance for any input.
Re: Tag-based testing
One way to do this and keep it compartmentalised would be to introduce a Form for each test/tag. Each Form could have its tags first and its expected output in a second block in the same Form.
If we put all such Forms in a single group (say,
Section or something) and confined shortcodes to other form types (e.g.
Article) then we could simply write a Page template to load up all ‘Section’ Forms and run through them one by one, executing the first half and comparing it to the known-good HTML in the second half of each Form, spitting out the results of the comparison.
Would that work?
EDIT: if the test suite was a separate repo, people could make pull requests to add or amend tests and expected output, which could then be compiled into the final repo using a CI hook ready for anyone to download and run… maybe?
Last edited by Bloke (2018-04-02 09:26:34)