Go to main content

Textpattern CMS support forum

You are not logged in. Register | Login | Help

#1 2022-08-12 18:10:31

gaekwad
Server grease monkey
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 4,134
GitHub

Textpattern project web server issues - August 12 2022

There has been some unexpected downtime during recent days. I am confident that I have figured out what is going on, and I am reasonably sure I know why.

There is a chance that some hardware is failing on our web server, and if that is the case then our sites will be migrated to a new server. The timescale for this is the next 72 hours.

This is a pre-warning that we may be moving servers. I will update this thread if this becomes necessary.

Offline

#2 2022-08-12 19:36:53

jakob
Admin
From: Germany
Registered: 2005-01-20
Posts: 4,578
Website

Re: Textpattern project web server issues - August 12 2022

Thanks for the info. I guessed you must be working behind the scenes. Best of success with a migration if it becomes necessary. FWIW I only noticed it with the forum. The main site was reachable.


TXP Builders – finely-crafted code, design and txp

Offline

#3 2022-08-13 08:06:32

gaekwad
Server grease monkey
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 4,134
GitHub

Re: Textpattern project web server issues - August 12 2022

jakob wrote #333774:

I guessed you must be working behind the scenes. Best of success with a migration if it becomes necessary. FWIW I only noticed it with the forum. The main site was reachable.

Frustratingly, nothing has changed in the config for some time, so no work was underway. I have a small fleet of servers with an identical build process, and it’s only this server exhibiting the issue. I thought it was an Nginx + PCRE conflict and tweaked the PHP config accordingly, but that didn’t help as it went south again this morning.

The issue manifests as an Nginx segfault with this error:

ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: called for pid 58551, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: called for pid 58553, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: called for pid 58555, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: called for pid 58559, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: called for pid 58561, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: called for pid 58563, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS

Each of those PIDs is spawning a new Python process and it takes the server out with a redlined CPU. A reboot fixes it, but that’s clearly not sustainable. My gut feeling says it’s hardware failure on the horizon, but I need half a day to migrate to a new server and I don’t have that today (Saturday). If all goes to plan, I will have time tomorrow.

To be continued…

Offline

#4 2022-08-13 16:21:26

jakob
Admin
From: Germany
Registered: 2005-01-20
Posts: 4,578
Website

Re: Textpattern project web server issues - August 12 2022

Thank you for all you are doing, whenever you get to it. I fear that few of us can help you with it, but if there is something, just say.


TXP Builders – finely-crafted code, design and txp

Offline

#5 2022-08-14 11:05:46

gaekwad
Server grease monkey
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 4,134
GitHub

Re: Textpattern project web server issues - August 12 2022

FYI: I’ve rolled us back to Nginx 1.23.0 from Nginx 1.23.1 as that seems to coincide with the start of these issues.

If we make it past 07:00UTC tomorrow without a panic, then we’re closer to finding the root cause.

Offline

#6 2022-08-15 03:02:11

Pat64
Plugin Author
From: France
Registered: 2005-12-12
Posts: 1,595
GitHub Twitter

Re: Textpattern project web server issues - August 12 2022

@gaekwad:

I notice from the demo versions of Textpattern, SVG images aren’t showing (both on the front and within the administration interface).
This is due to a strict CSP declaration. Maybe you could add a new one for image resources in data encoding format (i.e. data:image/svg+xml):

img-src 'self' data:

Last edited by Pat64 (2022-08-15 03:03:13)


Patrick.

Github | CodePen | Codier | Simplr theme | Wait Me: a maintenance theme | [\a mi.ni.ma]: a “Low Tech” simple Blog theme.

Offline

#7 2022-08-15 07:20:41

gaekwad
Server grease monkey
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 4,134
GitHub

Re: Textpattern project web server issues - August 12 2022

gaekwad wrote #333778:

If we make it past 07:00UTC tomorrow without a panic, then we’re closer to finding the root cause.

Same time of day, same problem…so it wasn’t the patch release of Nginx that causes it.

There’s a 3rd party module that I think might be the culprit, so that’s disabled for now. I am recompiling Nginx without that module. I will update this thread when I have further findings.

Offline

#8 2022-08-15 07:27:24

gaekwad
Server grease monkey
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 4,134
GitHub

Re: Textpattern project web server issues - August 12 2022

Pat64 wrote #333780:

Maybe you could add a new one for image resources in data encoding format (i.e. data:image/svg+xml):

img-src 'self' data:

Done!

Offline

#9 2022-08-15 10:29:56

Pat64
Plugin Author
From: France
Registered: 2005-12-12
Posts: 1,595
GitHub Twitter

Re: Textpattern project web server issues - August 12 2022

Perfect!


Patrick.

Github | CodePen | Codier | Simplr theme | Wait Me: a maintenance theme | [\a mi.ni.ma]: a “Low Tech” simple Blog theme.

Offline

#10 2022-08-16 09:55:19

gaekwad
Server grease monkey
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 4,134
GitHub

Re: Textpattern project web server issues - August 12 2022

Now resolved.

Nginx 1.23 includes changes that affect third party modules, and there was a conflict between two or more of them. I found the culprit, I’ve removed it, and we’re back on the air.

Thanks for flying Air Textpattern.

Offline

#11 2022-08-16 13:28:08

Dragondz
Moderator
From: Algérie
Registered: 2005-06-12
Posts: 1,529
Website GitHub Twitter

Re: Textpattern project web server issues - August 12 2022

Thanks Pete for your work.

Offline

Board footer

Powered by FluxBB