Attempted delivery

Look dude, we gave it a red hot go ok? We even drove down your street!

Inspired by a recent delivery experience by our favourite government owned postal service culminating in what I can only assume was a glance down our driveway followed by a "fuck that". Thankfully parcel lockers are a thing or I might have had to "tut" really loudly in the queue when going to retrieve my parcel (Which is now running this blog by the way, amazing what a cardboard box can do these days).

Funnily enough this post isn't about snail mail, though there is a tool I use regularly called "s-nail" which is a replacement for mailx that can send emails from the command line of a Linux system, I'm sure you can guess what inspired the name. s-nail follows the fantastic convention and design of what email used to be, essentially deliver mail locally to a user. Why would you need to do that externally?
Ok fine here is a config file you can use:

set smtp-use-starttls
set smtp=smtp://smtp.gmail.com:587
set smtp-auth=login
# Change 'xxxxxx' with your username
set smtp-auth-user=xxxxxx@gmail.com
# Change 'xxxxxxxxxxxx' with your password
set smtp-auth-password=xxxxxxxxxxxx

If the config above appears in a file called .mailrc in the calling user's home directory i.e. /home/username/.mailrc then it will be interpreted and actioned by the s-nail command (Commonly aliased by default to mail and mailx).

Cool man, digression much?

Sometimes I like to give background on the tools I use, sue me. To non-Linux or non-Unix OS users (There is a difference, do... not... get... me... started) these tools probably seem a bit arcane and silly in the world of GUI this and Import-Module that. However there is a lot to be said for simple text and string parsing over objects and binary encoded junk. "What's in this file and what does it do? I dunno, look up the file extension on Google and take a stab" is beyond ridiculous for about 324872398 reasons that I won't get into for fear of letting my inner zealot show (I mean I probably will but in a different post).

Now that we have a solid understanding of command line mail utilities we can begin ✨THE STORY✨.

Lists lists and more lists

At my org we have a mailing list server, Mailman 3 to be exact. Mailing list software can be gnarly to maintain at best and downright diabolical at worst. Mailman 3 is no exception though it does have a nice Web UI and is well built (Unlike its predecessor) so I can forgive its idiosyncrasies because at least it's not a total nightmare to manage. What is a nightmare to manage is the users. Users don't like change and they really don't like when other users complain to them. So when you implement a new Mailman server and migrate away from Mailman 2 as well as handing over management of each list to a new owner, shit can hit the proverbial really fast, in both a technical sense and a people sense.

After migrating, from a technical perspective nothing went wrong, when it came to people though...

This email is super important!!!

To begin with, our email infrastructure from an on-premises perspective is quite complex, I detailed this in a previous post which is now gone for good because I do not believe in backups. Ok I do but I don't test them...
That aside, essentially we have about 7 moving parts when it comes to the Mailman server(s) ability to send email, if you take them out of the mix you're left with 5 other moving parts.
As you can imagine, troubleshooting failures for these can be quite time consuming and tedious, churning over log files and running message traces left and right just to find someone's cat picture chain mail that is SUPER URGENT PRIORITY ONE is not my idea of fun.

One such spam totally legitimate email sent to a mailing list on the Mailman 3 server was causing issues for certain recipients.
To give you an idea of how much troubleshooting was required, here is a high level overview of the Mailman mail flow:

The plan is to remove the Mailman 2 server at some point when all lists are migrated off, this will significantly simplify the mailing list server and mail flow setup.

So the email was sent and worked its way through the flowchart to the internal recipient...

A wild email appeared

...except it didn't.
The ticket mentioned that a member of the mailing list didn't receive the email.
And then another.
And then another.
And another.

This seemed too weird to be coincidence. I started troubleshooting logically at the Mailman end. The first place to look was the Mailman 3 archives for the list, if the email was there then the old Mailman 2 server did deliver the message to Mailman 3 which 100% processed it successfully. The email was there.

So far so good.

Next was to check Postoffice to ensure it managed to lookup the internal user and forward the email to the right place. It could and did.

The Exchange hybrid servers (Seriously MS, why does putting exchange into a hybrid state not allow for full migration later?) logs looked fine and showed that they forwarded the mail to the user's target address in Exchange online. Well they showed about as much as Exchange logs have ever shown, email in -> email out... helpful.

Checking message trace in Exchange online showed the email arrived and was delivered.

I went back to the Mailman 3 server and used our old friend s-nail to send an email as both myself and the mailing list name. Just to make sure there wasn't some weird happenings with the server itself when mail was sent as the list address.

s-nail -v -s "TEST" -r me@org.tld user@org.tld < test_message.txt
s-nail -v -s "TEST" -r list@list.org.tld user@org.tld < test_message.txt

Both of the test messages arrived...

What on Earth?

I went back to the client that submitted the ticket:

Hi there,

It looks like the messages from the mailing list were successfully delivered to the affected recipients inbox, could you perhaps check with the members to make sure they don't have any client side rules in Outlook?

Thanks

Meanwhile I decided to check server side rules and forwards on a selection of the mailboxes:

Get-Mailbox {user} | select UserPrincipalName,ForwardingSmtpAddress,DeliverToMailboxAndForward
Get-InboxRule –Mailbox {user}

Both of which returned either nothing or unrelated rules.
This was becoming very strange.

After a little time (During which I decided to forget about this issue in the hopes it would go away) the client got back to me to say the members had no rules in Outlook besides the server side ones I had already found.

Completely lost on what to do next I started just staring at bash prompts and Exchange online mailbox panels.
Then I noticed something.

Trash talk

That little link.

Maybe I should check that?

BANG! Every single "missing" mailing list email. Deleted and then purged from the deleted items folder...

The users lied?!?

After going back to the client and explaining what I had found, they went back to their list members to enquire why they had been deleting these emails.

The answer?

"Oh there were so many coming through I just got annoyed with them and started to delete them..."

The lists had become some spammy and postings so frequent that users had just started binning the posts out of hand. Then when asked if they received the communications they said no.

Conclusion

An old adage we are all familiar with and tend to parrot out after the fact:

Start with the obvious

We won't though will we?