Textpattern CMS support forum
You are not logged in. Register | Login | Help
- Topics: Active | Unanswered
Textpattern project web server issues - August 12 2022
There has been some unexpected downtime during recent days. I am confident that I have figured out what is going on, and I am reasonably sure I know why.
There is a chance that some hardware is failing on our web server, and if that is the case then our sites will be migrated to a new server. The timescale for this is the next 72 hours.
This is a pre-warning that we may be moving servers. I will update this thread if this becomes necessary.
Offline
Re: Textpattern project web server issues - August 12 2022
Thanks for the info. I guessed you must be working behind the scenes. Best of success with a migration if it becomes necessary. FWIW I only noticed it with the forum. The main site was reachable.
TXP Builders – finely-crafted code, design and txp
Online
Re: Textpattern project web server issues - August 12 2022
jakob wrote #333774:
I guessed you must be working behind the scenes. Best of success with a migration if it becomes necessary. FWIW I only noticed it with the forum. The main site was reachable.
Frustratingly, nothing has changed in the config for some time, so no work was underway. I have a small fleet of servers with an identical build process, and it’s only this server exhibiting the issue. I thought it was an Nginx + PCRE conflict and tweaked the PHP config accordingly, but that didn’t help as it went south again this morning.
The issue manifests as an Nginx segfault with this error:
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: called for pid 58551, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: called for pid 58553, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: called for pid 58555, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: called for pid 58559, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: called for pid 58561, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: called for pid 58563, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
Each of those PIDs is spawning a new Python process and it takes the server out with a redlined CPU. A reboot fixes it, but that’s clearly not sustainable. My gut feeling says it’s hardware failure on the horizon, but I need half a day to migrate to a new server and I don’t have that today (Saturday). If all goes to plan, I will have time tomorrow.
To be continued…
Offline
Re: Textpattern project web server issues - August 12 2022
Thank you for all you are doing, whenever you get to it. I fear that few of us can help you with it, but if there is something, just say.
TXP Builders – finely-crafted code, design and txp
Online
Re: Textpattern project web server issues - August 12 2022
FYI: I’ve rolled us back to Nginx 1.23.0 from Nginx 1.23.1 as that seems to coincide with the start of these issues.
If we make it past 07:00UTC tomorrow without a panic, then we’re closer to finding the root cause.
Offline
Re: Textpattern project web server issues - August 12 2022
@gaekwad:
I notice from the demo versions of Textpattern, SVG images aren’t showing (both on the front and within the administration interface).
This is due to a strict CSP declaration. Maybe you could add a new one for image resources in data encoding format (i.e. data:image/svg+xml
):
img-src 'self' data:
Last edited by Pat64 (2022-08-15 03:03:13)
Patrick.
Github | CodePen | Codier | Simplr theme | Wait Me: a maintenance theme | [\a mi.ni.ma]: a “Low Tech” simple Blog theme.
Offline
Re: Textpattern project web server issues - August 12 2022
gaekwad wrote #333778:
If we make it past 07:00UTC tomorrow without a panic, then we’re closer to finding the root cause.
Same time of day, same problem…so it wasn’t the patch release of Nginx that causes it.
There’s a 3rd party module that I think might be the culprit, so that’s disabled for now. I am recompiling Nginx without that module. I will update this thread when I have further findings.
Offline
Re: Textpattern project web server issues - August 12 2022
Pat64 wrote #333780:
Maybe you could add a new one for image resources in data encoding format (i.e.
data:image/svg+xml
):
img-src 'self' data:
Done!
Offline
Re: Textpattern project web server issues - August 12 2022
Perfect!
Patrick.
Github | CodePen | Codier | Simplr theme | Wait Me: a maintenance theme | [\a mi.ni.ma]: a “Low Tech” simple Blog theme.
Offline
Re: Textpattern project web server issues - August 12 2022
Now resolved.
Nginx 1.23 includes changes that affect third party modules, and there was a conflict between two or more of them. I found the culprit, I’ve removed it, and we’re back on the air.
Thanks for flying Air Textpattern.
Offline
Re: Textpattern project web server issues - August 12 2022
Thanks Pete for your work.
Offline