Email notifications fail ?

phiw13 · 2021-03-16 09:49:51

Since the day after the forum + server upgrade I have not received any email notifications for topic I follow. I should have received three at least, one for an older topic I follow, two for a recent topic I subscribed to check the state of notification more than anything. None arrived and are not in my spam folder.

Paradoxically, at first email notifications did arrive (e.g. for this thread).

PS – email address used is my iCloud one (but filed here as the good old [at]me[dot]com. That still works, afaict.)

Last edited by phiw13 (2021-03-16 12:57:53)

gaekwad · 2021-03-16 11:02:08

I’ll look into it – thanks for the heads-up!

gaekwad · 2021-03-16 11:06:45

Postfix is stuck, investigating.

=> There is 1 zombie process.

Yeah, that’ll be it. Give me 5 mins.

Last edited by gaekwad (2021-03-16 11:09:47)

phiw13 · 2021-03-16 11:31:49

Drip, drip, drip the emails start arriving.

The zombie has been banned?

Thanks for looking into it.

gaekwad · 2021-03-16 11:48:36

phiw13 wrote #329287:

The zombie has been banned?

Not quite. I was wrong. Is fixed, but here’s the backstory (for transparency, because we all learn that way)…

tl;dr: I changed the server hostname stuff but not the Postfix sender, and Apple (rightly) rejected the sender.

Lots has changed on the Textpattern platform behind-the-scenes in the last few months, modernising and streamlining our web presence scaffolding, ultimately to make things more performance and efficient, but with a slightly selfish angle of making life easier for me to manage.

Our DNS used to be handled on GoDaddy, which is nothing short of hellish for proper management. Honestly, it nearly made me cry. We’ve since switched DNS to Cloudflare, and it’s much faster, way easier to manage and just better all round. There is no filtering going on with CF at the moment, just (fast) DNS management.

So, DNS. Before the recent migration we had lots of sites under the textpattern.com domain. Each had its own subdomain, and each subdomain needs various DNS entries to make it all work:

an A record for binding to an IPv4 address
an AAAA record for binding to an IPv6 address
a CAA record for TLS issuance on wildcard domains
another CAA record for TLS issuance on non-wildcard domains
sometimes a TXT record for SPF
sometimes another TXT record for DKIM

At one stage we had nearly twenty subdomains, each with up to 6x DNS entries. On GoDaddy. Oof. It was not fun.

So, long story short, I moved our hostnames over to the textpattern.net domains, so it’s air gapped from the massive .com DNS block. I changed the hostnames, all the TLS, all the build scripts, the reverse DNS, it all worked fine.

I did not change the Postfix sender address. It resolved to a now non-existent hostname.

My fault entirely. Sorry.

I’ve just processed ~7 mails stuck in the queue, all to phiw13, so they should be flowing freely now.

Anyone else: please check your spam. Emails from about 11:00UTC today should flow freely, but I’ll keep an eye on the mail queue.

For further credit, and my own edification, here’s how I processed the email queue: take the email that’s stuck, halt it from sending, export to a text file, change the sender in the text file, send the text file as a new email, delete the halted email from the queue, check it actually sent, start the process again.

Like this:

QID=1A2B3C4D5E6 \
&& sudo postsuper -h $QID \
&& sudo postcat -qbh $QID > $QID.eml \
&& sudo sed -i s/tribune\.textpattern\.com/tribune\.textpattern\.net/g $QID.eml \
&& sudo sendmail -f www-data@tribune.textpattern.net phiw13@example.com < $QID.eml \
&& sudo postsuper -d $QID \
&& sudo mailq

Fixed.

Last edited by gaekwad (2021-03-16 12:59:46)

phiw13 · 2021-03-16 12:57:33

gaekwad wrote #329288:

Not quite. I was wrong. Is fixed, but here’s the backstory (for transparency, because we all learn that way)…

tl;dr: I changed the server hostname stuff but not the Postfix sender, and Apple (rightly) rejected the sender.

[…]

Thank you for the whole back story. And yes ~ 7emails is what landed in the inbox tonight, the first one being from this post (to give you some different view / time stamp when things went wrong). Anyway, marking this as solved.

phiw13 · 2021-03-18 05:36:37

Maybe not fixed after all ? This reply has not reached my inbox, even though it appears I and still subscribed to that thread.

Or not yet reached my inbox ?

(I logged in here ~one hour before bici’s reply.)

gaekwad · 2021-03-18 08:10:55

phiw13 wrote #329329:

Maybe not fixed after all ?

Weird. 11 emails in the queue, same issue as before, even with the corrections made.

I will process them now and monitor over the day. Thanks for the report.

Edit: OK, queue cleared. There may be some backscatter from the changes I made and the delays with email servers talking back and forth, I’ll triple check all the settings.

Last edited by gaekwad (2021-03-18 08:17:29)

phiw13 · 2021-03-18 08:43:33

gaekwad wrote #329333

OK, 6 messages received, including yours above and the most recent Destry post .

If it helps debugging I can send you the headers for those 6 messages, in case there is something or other interesting (most is completely out of my field).

gaekwad · 2021-03-18 09:05:19

phiw13 wrote #329334:

If it helps debugging I can send you the headers for those 6 messages, in case there is something or other interesting (most is completely out of my field).

Thanks. I think the problem will fizzle out in a day or three.

High level: the sending email server (our end) spits an email over to your email provider’s server(s), and it runs various checks. The failing check appears to be email being received from tribune.textpattern.com (which doesn’t exist) instead of tribune.textpattern.net (which does exist).

Postfix, the mail transfer agent, is set to use tribune.textpattern.net as the underlying hostname, which it does. Evidently there may be a memory effect in place here since the old .com hostname is sometimes used. Vast majority of email is being delivered just fine. I’ve got some time later today, I’ll check the DNS records to make sure that’s all sensible, there may be some SPF voodoo that I can weaponise and force it into working.

Last edited by gaekwad (2021-03-18 09:07:10)

phiw13 · 2021-03-18 10:01:30

gaekwad wrote #329335:

Thanks. I think the problem will fizzle out in a day or three.

So it sounds like only a small subset of mail is affected, and somewhere a relay-server is caching something that will soon be cleared. Ok,then patiently waiting for things to settle down

BTW, I was more worried about other users potentially affected who may not notice something is going wrong.

gaekwad · 2021-03-18 10:24:46

phiw13 wrote #329336:

So it sounds like only a small subset of mail is affected, and somewhere a relay-server is caching something that will soon be cleared. Ok,then patiently waiting for things to settle down

BTW, I was more worried about other users potentially affected who may not notice something is going wrong.

On the surface, it’s mostly an Apple iCloud issue with emails not landing, and it appears to be mostly you and sometimes bici affected. Right now, the mail queue is empty (i.e. all emails have ‘sent’), and when I use the Submit button here, an email will be queued for you (and any other subscribers), which brings the queue up to…a few…then it should drop to zero again. Later on I expect to see a few emails with responses from other email servers (hi, Apple) with an error of sorts. We’ll sort it, nothing you need to do.

Textpattern CMS

Textpattern CMS support forum

#1 2021-03-16 09:49:51