Textpattern servers - situation report, December 9th 2021

gaekwad · 2021-12-09 13:50:03

There was some unexpected forum downtime a few days ago. This downtime also affected our pre-alpha Textpattern plugins and Textpattern themes sites, along with the Textpattern design patterns site.

What’s happening right now

This is the current status:

All our sites are back online (at least as far as my testing has shown).
They are now all running from a single server, not two as before. There is just about enough capacity / resource for us to run on one server for a short time. Soak tests have been successfully run, but this is not a sustainable long term.
New servers are being commissioned and will be deployed in the coming weeks.
All backups are intact, no data was lost. A restore from backup was undertaken, and everything checked out just fine.

What’s happening in the near future

Thanks to our supporters at DigitalOcean we have an allocation of VPS server credit to spend with them. This means we can deploy servers in any of their datacenters, along with extra storage volumes and other useful stuff. Thanks to me growing up in a household with absolutely no spare money, I am adept at extracting as much value from our allowance as possible, while maintaining a level of service and availability that I think is appropriate.

At a high level — and this is subject to some minor changes — this is what the next chapter of Textpattern servers will look like, and I hope we can achieve this over winter 2021/2022:

3x new production servers (tarzan, tomato and turtle) to house our main site, forum, ancillary sites, plus self-hosted docs (i.e. we’re moving away from GitHub Pages for the docs). One of the servers will have a status page for our services, so anyone can see what’s working (and by extension, what’s broken). In the event of an incident, we (or I, since it’s usually me that breaks stuff…) can post updates there instead of spamming Twitter (which feels a bit amateur and low-rent, honestly).
A fourth server will be deployed with some kind of live chat / conferencing facility. I’ve talked about this before, and it’s on my list to do, but this will happen.

…and why did stuff break in the first place?

At the time, I alluded to an error that I made. For completeness, I’ll explain what happened…mostly so you can avoid making the same mistake and have a better understanding of how things run behind the scenes.

We have an email inbox that receives various types of emails, and the dev team have access to it. We get a lot of drivel arriving in the form of spammy web design services (invariably someone offering Wordpress, Drupal or Magento services), offers for backlink articles (we have a high-value .com domain since we don’t currently deploy nofollow and marketeers love that), and other junk. The important stuff we’re interested in — security issue reports, for example — are found and dealt with. There are occasions when the dev team need to contact people outside of the core team, especially when a member of our community has specialisations in a given area. Until now, it’s been a mishmash of emails, and this needs to be more formally handled, more so when you consider we’ve had an notable uptick in the volume of security reports in the past year. The vast majority of reports are covered under our Textpattern security considerations and user privileges article, but each report is handled appropriately. Until now, that’s meant a hand-written reply to each reporter, and we invariably don’t hear back from them after that.

We are going to deploy a ticketing system to handle these emails, so we can spot trends, reply with boilerplate text when we need to, and figure out how best to architect both our .com site and the docs / knowledge base sites. It’s also an excellent and sustainable way to extend access to our enquiries to trusted members of our community with their areas of expertise.

You may think this is me seeding the idea that our email inbox would benefit from a moderator / triage person or two…and you’d be right. We don’t get a lot of emails, but when a legit researcher takes the time and energy to provide a report on an issue they’ve found, it’s common courtesy on our part to work with them effectively, and in a timely way. In contrast, the drive-by “beg bounty” hunters can be dealt with using minimal resources, which leaves our sanity / energy for other things that benefit the Textpattern user base.

Long-winded story short: I set up a neat ticketing system on my dev box, it worked fine, then deployed to production…and in the process, it blew up Nginx (our web server), broke PHP (our scripting language), and tornado (the forum server) is in a sorry mess. I have rescued the sites, transferred them to trapeze and we’re back on the air. I will read last rites to tornado today, since it’s very much in scorched earth territory, and it will be nuked this evening when I am 100% sure all useful resources have been extracted.

Textpattern 4.8.8 will have support for PHP 8.1. Given that most of our sites run Textpattern, and we run the latest release, I am holding off from deploying to the new servers until we’re happy Textpattern 4.8.8 has passed beta tests. This has the benefit of streamlining the PHP setup on each server, since right now we’re running dual-version PHP:

FluxBB runs on PHP 7.4.
Textpattern 4.8.7 runs on PHP 8.0.
The exception to the point above is textpattern.org which runs on PHP 7.4 due to legacy plugins.

So, we need two versions of PHP at the moment. We can retire textpattern.org when the new plugins site is up and running, and we’ll figure out how to make FluxBB behave with modern PHP (see github.com/textpattern/textpattern-forum/issues/449 for the gory details), and then we can drop PHP 7.4 from our server(s). The benefit of running single-version PHP is that system resources can be allocated to one version, not two. Memory allocation, CPU cycles, that sort of thing — better to focus on one thing and open the taps than split between two and balance them.

I think that’s all for now. Thanks for reading, please leave any questions below. Sorry for breaking stuff!

colak · 2021-12-09 20:03:02

gaekwad wrote #332154:

So, we need two versions of PHP at the moment. We can retire textpattern.org when the new plugins site is up and running…

Are we going to keep the domain and maybe do a redirect to the .com one?

gaekwad · 2021-12-09 22:15:06

colak wrote #332156:

Are we going to keep the domain and maybe do a redirect to the .com one?

Definitely.

colak · 2021-12-10 04:05:44

gaekwad wrote #332158:

Definitely.

phew! panic gone:)

gaekwad · 2021-12-10 10:04:18

colak wrote #332160:

phew! panic gone:)

We don’t do panic at Textpattern Towers, please don’t worry!

Textpattern CMS

Textpattern CMS support forum

#1 2021-12-09 13:50:03