Textpattern project web server issues - August 12 2022

gaekwad · 2022-08-12 18:10:31

There has been some unexpected downtime during recent days. I am confident that I have figured out what is going on, and I am reasonably sure I know why.

There is a chance that some hardware is failing on our web server, and if that is the case then our sites will be migrated to a new server. The timescale for this is the next 72 hours.

This is a pre-warning that we may be moving servers. I will update this thread if this becomes necessary.

jakob · 2022-08-12 19:36:53

Thanks for the info. I guessed you must be working behind the scenes. Best of success with a migration if it becomes necessary. FWIW I only noticed it with the forum. The main site was reachable.

gaekwad · 2022-08-13 08:06:32

jakob wrote #333774:

I guessed you must be working behind the scenes. Best of success with a migration if it becomes necessary. FWIW I only noticed it with the forum. The main site was reachable.

Frustratingly, nothing has changed in the config for some time, so no work was underway. I have a small fleet of servers with an identical build process, and it’s only this server exhibiting the issue. I thought it was an Nginx + PCRE conflict and tweaked the PHP config accordingly, but that didn’t help as it went south again this morning.

The issue manifests as an Nginx segfault with this error:

ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: called for pid 58551, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: called for pid 58553, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: called for pid 58555, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: called for pid 58559, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: called for pid 58561, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: called for pid 58563, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS

Each of those PIDs is spawning a new Python process and it takes the server out with a redlined CPU. A reboot fixes it, but that’s clearly not sustainable. My gut feeling says it’s hardware failure on the horizon, but I need half a day to migrate to a new server and I don’t have that today (Saturday). If all goes to plan, I will have time tomorrow.

To be continued…

jakob · 2022-08-13 16:21:26

Thank you for all you are doing, whenever you get to it. I fear that few of us can help you with it, but if there is something, just say.

gaekwad · 2022-08-14 11:05:46

FYI: I’ve rolled us back to Nginx 1.23.0 from Nginx 1.23.1 as that seems to coincide with the start of these issues.

If we make it past 07:00UTC tomorrow without a panic, then we’re closer to finding the root cause.

Pat64 · 2022-08-15 03:02:11

@gaekwad:

I notice from the demo versions of Textpattern, SVG images aren’t showing (both on the front and within the administration interface).
This is due to a strict CSP declaration. Maybe you could add a new one for image resources in data encoding format (i.e. data:image/svg+xml):

img-src 'self' data:

Last edited by Pat64 (2022-08-15 03:03:13)

gaekwad · 2022-08-15 07:20:41

gaekwad wrote #333778:

If we make it past 07:00UTC tomorrow without a panic, then we’re closer to finding the root cause.

Same time of day, same problem…so it wasn’t the patch release of Nginx that causes it.

There’s a 3rd party module that I think might be the culprit, so that’s disabled for now. I am recompiling Nginx without that module. I will update this thread when I have further findings.

gaekwad · 2022-08-15 07:27:24

Pat64 wrote #333780:

Maybe you could add a new one for image resources in data encoding format (i.e. data:image/svg+xml):

img-src 'self' data:

Done!

Pat64 · 2022-08-15 10:29:56

Perfect!

gaekwad · 2022-08-16 09:55:19

Now resolved.

Nginx 1.23 includes changes that affect third party modules, and there was a conflict between two or more of them. I found the culprit, I’ve removed it, and we’re back on the air.

Thanks for flying Air Textpattern.

Dragondz · 2022-08-16 13:28:08

Thanks Pete for your work.

Textpattern CMS

Textpattern CMS support forum

#1 2022-08-12 18:10:31