The future of Textpattern docs; intentions & plans; RFC

gaekwad · 2025-08-27 11:41:19

I’m sharing some plans for Textpattern user documentation (henceforth ‘docs’) which cover the repository (henceforth ‘repo’) and the wider infrastructure including website hosting.

Disclaimer

As a preflight note: some of the tone within this post may come across as coarse / blunt.

Please rest assured my intentions are positive, and for the avoidance of any doubt I’m opening up the discussion to anyone who wishes to get involved – veterans, newbies, weekend warriors etc – to add commentary, advice, opinions, and ask for clarification on anything.

BLUF:

In the short term, our docs will continue to be self-hosted on Jekyll (i.e., no changes).
In the medium term, our docs will be migrated to & hosted on a third-party hosting service, likely GitBook or Read The Docs (i.e., this is a change).
In the medium term, our docs will made available as formatted portable files (e.g. EPUB, PDF) for offline use (i.e., this is a change).
In the long term, our docs will be migrated back to self-hosting on Textpattern, with automated builds (i.e., this is a change).
For all the above cases, our docs will continue to be collaboratively editable in a version control system, likely git atop GitHub (i.e., no changes).

Let’s break those down and untangle some snakes…

Jekyll-hosted docs

Our user docs are currently generated from a GitHub repository to a regular schedule. Approved changes & additions made in that repository are automatically reflected in the docs website, typically within 30 minutes (typically much quicker). I have built a server which has a primary purpose of generating these docs to a schedule.

That works fairly well with minimal muss or fuss, until it doesn’t, and then it involves hours of my time to resurrect & restore it to a point where it sort of works fairly well, but I’m not 100% sure what’s going to break next, or which layer of abstraction is at fault. Is it Jekyll? Is it a plugin? Is it Ruby? Is it something else? There goes half a day of head scratching & testing, along with an unhealthy amount of cortisol & spicy language for me.

Every single time, I get frustrated how opaque things are with Jekyll, its plugins, and Ruby. This is primarily because our docs site is the only site I maintain with Jekyll, and as a result I’m not proficient enough for my liking. We use Jekyll because prior to that we used GitHub Pages, and that runs on…Jekyll. It made sense for us to have as an efficient workflow as possible when moving away from GitHub Pages, and so we went with Jekyll.

Native search has been broken for [macOS/iOS]/Safari users for as long as I can recall. I have no idea how to fix that. As much as it pains me, a workaround is to use another macOS browser. The pushback from you fine people has been minimal, and so there’s even less reason to find a Jekyll-native search solution. This is very much a double-edged sword since issues can lay dormant without resolution for some time, and that can breed mild contempt.

I want to move away from Jekyll for multiple reasons, some of which are outlined above and others are outlined below:

We can use generous open source allowances on third-party platforms ^{where search just works}.
We can decommission our docs server and move the donated server credit to other servers & services.
I can reallocate server admin time & energy from the docs server to other servers & services.

We are fortunate enough to have an annual open source grant from DigitalOcean. There is an application process each spring where I apply for a renewal, and we make sure that we’re ticking the right boxes. To ensure we don’t blow the allowance budget, our servers & services are carefully managed across three continents with some redundancy. Our docs server represents about 20% of total annual budget allowance. That’s a considerable amount that can be reallocated if we switch to a fit-for-purpose hosted service.

We inevitably go over the ‘zero cost’ allowance we get from DigitalOcean each year. It’s not a considerable amount, but it’s vital we make best use of all our finances across the projects to maintain best value for stakeholders (that’s you, the users). We have a core of monthly donors who generously support our project at Open Collective and Github Sponsors and as a team we’re very careful to manage those funds for the long term. We don’t know from one year to the next whether our DigitalOcean credit will renew, and since a sponsor can opt in or out at any point we have to be super careful in our spending. The upshot is that we’re familiar with making a small amount of donation go a long way, so it’s second nature.

Adjacent to this reallocation of server allowance is tooling a server for our developers to have better insight into what happens with Textpattern behind the scenes. We can make better use of PHP & database tools, run analysis on our codebase, all sorts of things that benefit all Textpattern users in the long term. I am much more inclined to spend a few days a month building & maintaining tools that we can all benefit from versus unpredictable annoyances like Jekyll.

This leads nicely into…

GitBook or Read The Docs

These are third-party services that offer generous allowances for open source projects. I have mellowed over the last 5-10 years and I’m much more willing to let appropriate services do some of the heavy lifting if they pass muster.

Both services pull files from GitHub and build browser-ready files. Both services have working search functions (how novel!). Both services will let us continue with our current GitHub workflow for editing & creating docs, albeit with some front-loaded work involved with formatting for the final web pages. Both services will be appropriate for us in the medium term.

When those docs are up & running without a Jekyll in sight (in site?), we can proceed to the next item of business…

Portable, pretty compiled docs

When we have sensible docs that are nicely formatted for a website, it’s not a stretch to repurpose that source material into offline formats like EPUB or PDF. There are workflows that can automatically compile docs to an EPUB & PDF, and these would be updated on every approved change or addition to the source material. We can achieve this with a predetermined task with a computer doing the heavy lifting, rather than someone picking through a desktop publishing package.

Now we have docs going to a third-party service for hosting, and those portable downloads available for anyone to grab. The source material is being used for multiple purposes, and we have trusted systems doing their thing.

Let’s move to the elephant in the room…

Bring Textpattern docs to a self-hosted Textpattern

Textpattern has a method for importing its default content from pre-formatted XML files. Textpattern has a scriptable automatic installer. Our demo sites use the automatic installer over & over each day, and there is nary a problem.

There’s no reason why we can’t format our docs source material in such a way that it can be converted to XML and scooped up by the automatic installer. We can add another workflow to the repo to take the source material, run it through the Pandoc machine, and we have Textpattern-ready files to throw into a Textpattern installer.

We have docs with categories, tags, search will work, and there’s nary a Jekyll in site…sight. Marvellous!

It sounds so simple! Well, it may be. We have a lot to learn, much testing, and we need to make sure it’s ticking along nicely before we flip switches.

Schedules & timescales

The most recent Textpattern production release landed in early 2022. It’s time for a new one, and Textpattern 4.9.0 is coming very soon. At the time of writing, there are plans to include support for the upcoming PHP 8.5 (ETA: November 2025) as well as formalising support for PHP 8.2 to 8.4.

A lot has changed in Textpattern 4.9.0, and it’s not even out yet. It’s likely that Textpattern 4.9 will be the final minor branch of Textpattern. There will be patch releases (e.g. Textpattern 4.9.1) but focus will be on the ‘big’ release of Textpattern 5.

Given the time since Textpattern 4.8.8, the common sense approach is to get Textpattern 4.9.0 ready for production release as soon as we’re happy it’s ‘ready’. There doesn’t need to be another hurdle to get Textpattern 4.9.0 out of the door. Moving our docs to a Textpattern instance now will delay the release of 4.9.0, and I won’t let my docs proclivities be a factor in that. It’s nearly time for Textpattern 4.9.0 to flex its muscles.

tl;dr

We’re sticking with Jekyll docs for the meantime.
Then we’re moving docs to GitBook or Read The Docs when we figure it out.
Then we’re obliterating the Jekyll server.
Then we’re going to make pretty downloads for bedtime reading.
Then we’re going to figure out Textpattern docs running out of Textpattern.

Thanks for reading.

Questions, requests for clarification, advice, feedback and other business are all warmly welcomed.

Edits: clumsy typos.

Last edited by gaekwad (2025-08-27 12:14:04)

Bloke · 2025-08-27 12:12:23

That’s a fantastic roadmap. Totally onboard.

I haven’t forgotten about the docs playground you generously set up for me last month. Just got sideswiped with a couple of clients that require a lot of development effort.

Both will contribute code back to the community (when I’ve formalised it a bit more, and made it less project-specific to the point I’m happy to share the progress via git) so this effort will, I hope be very worthwhile for Textpattern. At the moment I’ve only tested the code paths one of the clients needs, so there’s a lot of stuff in there that could fall apart under more varied use cases. Before I’m happy it’s even remotely workable, I’ll need to do a lot more development and testing. But I promise it will be worth the effort when I eventually push the public first commit, labelled, “v1 is here. I’m going to bed.”

In the meantime, if there’s anything we need to push this forward to Stage 2, I’m happy to put time into it because docs are a project sticking point; not just for the amount of Pete’s precious time they consume to maintain when things stop working, but because we can be smarter about building a lot of them (tags, specifically) from the code itself if we’re cunning, and from git through some judicious pipelining.

Oleg has already made us a route in through a clever reflection hook into the attribute generation. The trouble is that not all tags use all of them, or use some in specific ways. We perhaps need to find a way to document those in the code so they can be sucked into the docs in a more routine fashion, cognisant of version so we can document “now/latest/stable” and “upcoming” docs more easily.

4.9.0 has been coming “real soon now” for far too long and, as Pete says, we can’t keep putting it off by doing other stuff for the benefit of “later”.

Stake in the ground time. And this roadmap is an important part of that. Thank you.

gaekwad · 2025-08-27 12:19:05

Bloke wrote #340305:

I haven’t forgotten about the docs playground you generously set up for me last month. Just got sideswiped with a couple of clients that require a lot of development effort.

Not a problem, Bloke. For clarity, that repo playground is as much for me as it is for you (or anyone else, for that matter). It’s how we’ll learn the ropes for GitBook and / or Read The Docs, so don’t consider it a sole tenant situation for you at all.

In the meantime, if there’s anything we need to push this forward to Stage 2, I’m happy to put time into it because docs are a project sticking point

For now, no – thank you. I’ll get my teeth into the CI stuff when time & headspace permits, and when that’s more clear we can nail down the route to Stage 2.

4.9.0 has been coming “real soon now” for far too long and, as Pete says, we can’t keep putting it off by doing other stuff for the benefit of “later”.

You & Oleg do the CMS wizardry, I’ll wrangle the user manual for now. Fair?

Bloke · 2025-08-27 13:10:51

Works for me. Thank.you.

jakob · 2025-08-27 13:40:50

Erm, just an annoying question in-between (sorry if I missed the explanation above): what was the reasoning behind the intermediary step with Gitbook / Read the Docs? Is it not to tear away too much of the devs’ attention away from the CMS in getting the importer working?
I wonder/fear that we may settle for the mid-term variant where the energy might be better invested in the long-term option. Then again, maybe Gitbook or Read the Docs have some advantages over txp. I don’t know enough about them.

Edit: In the past I dreamed of having the docs in some portable form that I could take with me. I still love the idea of having portable docs, but must admit mobile internet coverage has improved along with higher-volume data plans to the point that it is no longer so essential. That said, I’d still like the idea, though I’m not sure a PDF or EPUB is the most useful form. How feasible is it to make it available as an offline standalone browser? In Safari (and maybe other browsers) you can make site-specific browsers using the “Add to dock” function. You end up with a single-site browser as a separate app, which avoids the docs getting lost among the many other open browser tabs. There are other mac apps like Fluid and Unite that will do that, and probably similar things on Windows too.

gaekwad · 2025-08-27 13:51:49

jakob wrote #340308:

what was the reasoning behind the intermediary step with Gitbook / Read the Docs? Is it not to tear away too much of the devs’ attention away from the CMS in getting the importer working?

Yes. Well, that and some other considerations:

We need to figure out the Pandoc-izing from Markdown to Textpattern importable format (XML).
- That import format may well change for Textpattern 5 (JSON? YAML? Other?) if the import system gets overhauled.
Textpattern 5 may well be some years away yet.
- When we figure out the Pandoc wrappers for XML and/or whatever else format, it should be a case of making different formats and away we go.

Edit: all the figuring out stuff takes time to grok, and it’s something that non-developers (hi!) can do. I’m perhaps oversimplifying or idealising a bit, but Bloke’s and etc’s time seems better spent on the code side.

I wonder/fear that we may settle for the mid-term variant where the energy might be better invested in the long-term option.

That’s valid & very fair. I’m looking at GitBook or RTD as better than Jekyll, and Textpattern as best for managing the content…and using the third-party route as an intermediate fix for the Jekyll search issues & other stuff brings a $$ value along with some headspace to figure out the Textpattern-native route.

Then again, maybe Gitbook or Read the Docs have some advantages over txp. I don’t know enough about them.

I’m in the same boat – I know some stuff about them both, but if a viable better route is available for a reasonable amount of time & effort, then the best route is less of a leap after that, especially if the ground work has been done.

From an intermediate perspective, there looks to be some YAML to tack onto our docs repo, join the dots, and the system starts working.

gitbook.com/docs/getting-started/git-sync/enabling-github-sync
docs.readthedocs.com/platform/stable/reference/git-integration.html

Last edited by gaekwad (2025-08-27 13:55:36)

jakob · 2025-08-27 14:09:34

Thanks. Maybe we can put our heads together on that in a few weeks’ time? I’ve used pandoc before and found it to be generally pretty good, but the YAML stuff at the top needs working out.

Sorry, I added an edit to my post above, but you were quicker with your reply.

gaekwad · 2025-08-27 14:12:13

jakob wrote #340311:

Maybe we can put our heads together on that in a few weeks’ time? I’ve used pandoc before and found it to be generally pretty good, but the YAML stuff at the top needs working out.

Wonderful. Thank you. I’ll drop you a line second half of September, see how we’re placed.

gaekwad · 2025-08-27 14:14:09

jakob wrote #340308:

Edit: In the past I dreamed of having the docs in some portable form that I could take with me. I still love the idea of having portable docs, but must admit mobile internet coverage has improved along with higher-volume data plans to the point that it is no longer so essential. That said, I’d still like the idea, though I’m not sure a PDF or EPUB is the most useful form. How feasible is it to make it available as an offline standalone browser? In Safari (and maybe other browsers) you can make site-specific browsers using the “Add to dock” function. You end up with a single-site browser as a separate app, which avoids the docs getting lost among the many other open browser tabs. There are other mac apps like Fluid and Unite that will do that, and probably similar things on Windows too.

Let’s find out. I’m not married to EPUB and PDF specifically, if there’s a way we can make a useful thing (Jupyter notebook?) and there’s a market for it, let’s see how viable it would be.

planeth · Yesterday 15:24:03

jakob wrote #340308:

Edit: In the past I dreamed of having the docs in some portable form that I could take with me. I still love the idea of having portable docs, but must admit mobile internet coverage has improved along with higher-volume data plans to the point that it is no longer so essential. That said, I’d still like the idea, though I’m not sure a PDF or EPUB is the most useful form. How feasible is it to make it available as an offline standalone browser? In Safari (and maybe other browsers) you can make site-specific browsers using the “Add to dock” function. You end up with a single-site browser as a separate app, which avoids the docs getting lost among the many other open browser tabs. There are other mac apps like Fluid and Unite that will do that, and probably similar things on Windows too.

As of ‘offline standalone browser’, do you mean Progressive Web App ?
We could totally have a website for docs, for which all the page are cached for offline use by a service worker.
Add a webmanifest at the root and it will make it ‘installable’ for a majority of browsers. Firefox being the outlier here.
If you need me to expand more on that, just ring me !

phiw13 · Today 01:26:17

jakob wrote #340308:

Edit: […] How feasible is it to make it available as an offline standalone browser? In Safari (and maybe other browsers) you can make site-specific browsers using the “Add to dock” function. You end up with a single-site browser as a separate app, which avoids the docs getting lost among the many other open browser tabs. There are other mac apps like Fluid and Unite that will do that, and probably similar things on Windows too.

Safari’s “Add to Dock” works quite well for the docs site, I’ve used earlier this summer when doing some work in limited network conditions¹. I think Chromium browsers can do something similar. Or maybe it is just limited to PWA’s which is much more limited as it requires some special configurations for links etc.

Firefox seems again to have dropped of limited support for PWA’s.

–^–

¹ Limited availability of extensions though :-( maybe I have to break the bank and update to the bloated StopTheMadness Pro

phiw13 · Today 01:32:06

In the short run, moving to Read The Docs seems a good option.Some people I worked with did just that a few years ago, away from Jeckyl and poor search ( they had many more issues, + they moved away from GH). They now self-host the raw data (on a Gitea instance I think it is called).

Longer term using Textpattern to display the documentation would be great and a good example of the possibilities.

Textpattern CMS support forum

#1 2025-08-27 11:41:19