I don't do whitelisting. Instead, I always reach out and offer to help the other party correct their SPF record.

It happens often enough that I wrote a script in Racket that will generate the email for me and paste it into the clipboard [1]. The email tells them exactly what they need to change, and links to docs from their current email provider (so they don't have to trust me about edits to their DNS).

[1]: https://gist.github.com/otherjoel/6b8bf02f6db6e0c47ba6bca72e...

And at the same time, I regularly get Spam/Phishing with perfect SPF, DKIM, DMARC, etc. The domains and IPs they use might get blocked within a day, but of course, these people have no problem getting others.

And although I have set up my MTA perfectly, my mail gets refused by MS/t-online/etc., because I don't have enough "sender reputation". In e-mail, we have an oligopoly of a few big mail providers, and in the end, they decide which mail gets delivered and which isn't, and to me it looks like they give a rat's ass about SPF and DKIM, and probably rightfully so, because most spammers are probably better at configuring MTAs than your average mail admin.

Note that unless T-Online has additional requirements here this doesn't need to be your home address but only a valid mail address through which you can be reached.

(1) any kind of journalistic content on your site

(2) any kind of financial gain from showing ads or making ads

(3) organizing any kind of group of people active on German territory

(4) running a business website

There might be more, but those are the ones I remember from reading the paragraphs a while ago.

And these are, of course, vague, which means that even something like "my favorite restaurants in Berlin" could be considered an ad, or any kind of comment on politics might be considered a form of journalism.

I dislike these rules, because they basically kill German blogging scene. Not so many people want to run a blog and have every idiot on the Internet know their personal address. And few bloggers want to rent a digital office or actual office, that will send mail to them (an indirection). The German law in this respect is terrible and working against a free Internet and against freedom of voicing your opinion. It works greatly in favor for tech giants, because people resort to putting their blogging on Facebook, Instagram and other disservices. It is very anti-decentralization.

It doesn't have to be your personal address though.

I'm somewhat mixed about the details of the law, but requiring businesses to make it clear who you are dealing with makes sense to me.

I like the idea, but I would think sending a technical email (with industry-specific acronyms that you don't spell out!) to a business that has no in-house IT would just be ignored in most cases.

Overall I'm pleased with how well this approach works. When people realize that their email is getting stuck in spam filters because of a problem on their end, they're usually motivated to get it fixed. Sometimes it gets sent to an owner who had barely enough tech mojo to stand up a gmail account at a custom domain, and even then the instructions are usually simple enough for them to follow.

> If you do not have access to your company’s DNS records, please forward this email to someone in an IT role.

If you don't even know what DNS records are, I'd imagine you'd assume you don't have access to them and so forward them to the IT person as suggested. But sure, maybe he could also add ", or don't know what they are" to this line.

I think it's a good idea (and said so twice!). I was curious what the reception was like.

Sorry if my comment about acronyms was too much. It is a pet peeve of mine to see acronyms not spelled out, especially technical ones in a document intended for non-technical people. I didn't intend it to derail the conversation. Obviously it was taken in a way more critical way than I had intended -- my fault.

I do

    (load "spf-fail.rkt")

    (require spf-fail)

    (require "spf-fail.rkt")

Another method would be to open the file in DrRacket (or VSCode or Emacs or whatever editor you have set up with a Racket plugin/lang server) and just "run" it in the REPL.

There are comments at the top that explain how to use it once it's loaded:

    ;; Generate a form email to let someone know their SPF records are misconfigured for their current email provider.
    ;;
    ;; Run (fill-report "domain.com" "1.2.3.4") where the 2nd arg is the sending email server's IP address.
    ;; It will copy the completed report to the clipboard for you.
    ;;
    ;; Only works on Windows for now.

    raco canned spf domain.com 1.2.3.4

Or at least create a simplified common abstraction layer.

It’s the most inherently user-hostile thing I’ve ever encountered - and I’m only just now starting to understand it, even though I’m almost 20 years into dealing with it.

I'm not sure I like it. Like, what if the notification of notice was incorrect? You lose your license anyway?

Including a fair, common sense path to forgiveness severely limits legal risk for users, and is one of the things I like about the Blue Oak license.

> Copyright

> Each contributor licenses you to do everything with this software that would otherwise infringe that contributor's copyright in it.

This sounds like the license specifically allows you to infringe on the contributor's copyright.

I've had a couple of other-company-IT-admins tell me that my MX is jacked because I use hosted SPF via proofpoint, and when they look up my SPF it looks like this:

"v=spf1 include:%{ir}.%{v}.%{d}.spf.has.pphosted.com ~all"

A surprising number of mail admins don't understand SPF macros.

Interestingly all Proofpoint customers too.

I’ve seen it more common to isolate services to subdomains and specify subdomain SPF records rather than use macros. This is my preferred approach.

I’m not hating on the macros. They’re just seemingly very rarely used. I know they’re on the table but I haven’t found a compelling use case in my own deployments.

- with DNS, the simplest as possible SPF record.

- Without DNS, (aka pure IPv[46] SMTP), your have implicit SPF: instead of querying the DNS and parsing the SPF record, you parse the mail header to check "reply-to/from/etc" fields (the appropriate fields) for the sending SMTP IPv[46] address, that to perform spam scoring.

Things like, insisting we need to include their SPF record in ours, even going so far as to scan the SPF record for the include, only to find out they use their own domain in the envelope address (which is what I wanted them to do in the first place).

Or not distinguishing at all between envelope and header addresses and using our domain in both. Which of course means they're not tracking delayed bounces.

It really becomes an issue with larger orgs where everybody wants to use the main domain for brand purposes and subdomains are just totally frowned upon for whatever reason. If you just leave my SPF alone and rely on DKIM, it means you can still pass DMARC and track bounces properly. Hell I'd be fine with making subdomains for the envelope address that lists your infrastructure in the MX records but again, eyes really start to glaze over when you say "envelope address."

Basically what I really want is a guide that boils down to: if you're not their primary email provider, then don't touch your client's SPF record.

I still configure my default apex DMARC records (in most scenarios) to enforce strict alignment on both SPF and DKIM, but I’ve been relaxing that on a case by case basis or overriding the apex DMARC policy at a subdomain level and only using DKIM where supported.

I do use DMARC and SPF in the fashion you described. In my environments I typically need to take every measure to ensure only authorized services/servers/senders are sending, via authorized hosts and IPs, and this often changes based on subdomain, so that’s why I use strict alignment. Personally I strive to keep various services strictly separated by (sub)domain.

I attended his tutorial and talk at BSDCan[3] this year and both were excellent. I highly recommend buying the book when it comes out (or supporting the Kickstarter), it will go through all the gory details of setting up and running a mail server, and best practices, including a ton of material on SPF/DKIM/DMARC.

(P.S. I have no affiliation with the author or the book in any way.)

[1] - https://mwl.io/

[2] - https://www.kickstarter.com/projects/mwlucas/run-your-own-ma...

[3] - https://www.bsdcan.org/2024/

I don't even run a mailserver, I'm just hoping it will take a bunch of the guides that have been floating about on the web, consolidate the sharp edges, and make sure its up to date.

I also hope it has some discussion on troubleshooting. Like dealing with blacklists and what not, folks always talk about that, but I've never see it documented what is actually done to resolve these problems (Like who do you send an email to, how do you even find out who to send an email to, etc.)

I like the quality of this SPF/DKIM/DMARC guide. This is the industry I started out in, and I actually wrote the guides for SendGrid, Amazon SES, and a few other email products. I don't mind saying that author has done a better job at this than me.

That being said, I see SPF/DKIM/DMARC guides like this pop up with some regularity, but users continue to have the same level of comprehension as before. I think the nature of this problem is not one that lends itself to being solved by guides. It's the sort of problem that a user is really only faced with once, which means that they're not getting the repetitions in to warrant any long-term comprehension.

I'm naturally biased here, but if you're onboarding users with SPF/DKIM/DMARC to your application, it's good if you can just get them setup with automation.

Interesting. Not lot of startups out there talking about Domain Connect on their front page. Seems like a decent amount of overlap with my project TakingNames.io[0]. Feel free to reach out if you want to talk shop.

[0]: https://takingnames.io/blog/introducing-takingnames-io

I've run 5 or 6 different mail servers over the past 10 years. Originally before O365 I was an exchange admin, then postfix, iRed, mailcow, mail gun, you name it. Hosted on every cloud provider, even in our colo with part of a private /24 allocation with good reputation (built since 1997, gawdamn). Every sort of header combination, tls setup, and no blacklists. Always 100% alignment, including strict rejection policy (best results even over quarantine).

Does not matter, if you're sending from custom domain not handled through a big name, expect the spam box with Gmail. Yahoo and Outlook are fine, but Gmail is the bane.

I've spent maybe 100 hours of my own over this last year and know what I realized? Nobody cares about email anymore, except for automated account management stuff (login, PW reset). Businesses pay the $3 /mo / seat for fastmail and don't think twice.

But the current trend is toward social chat (discord or Whatsapp) and most the people who own an iPhone just use their apple ID email for everything.

Although I am a fervent supporter of open protocols and believe email (with pgp signing) is an awesome long form communication format... Face it, it's going the way of the fax machine.

However, email has basically evolved into the way you communicate with “systems” and I’m kind of happy about it. Communication with companies outside your network, e-commerce accounts/purchases, communication with government systems, schools, banking, airlines, concerts/events, restaurants, etc. Hell, even RSS is now basically in email — newsletters are growing fast as a medium, not shrinking.

You just book a hotel in Nairobi? It’ll be in your email. No other communication method even comes close for this use case.

Social/chat apps will never unseat this because they’re social. Like nightclubs, the trendy ones come and go. Come back when you’ve set up an interoperable network of virtually every person on earth. Then we’ll talk about email being dead.

Email is the primary way that I stay in touch with my extended family and friends. At least for us, it's very much alive.

I’ve had to relax my SPF record to include the entire mail pool of my ISP to be able to send to anything hosted by Microsoft. I tried to liaise with them directly, and through Linode, but they refused to exclude the IP from their opaque blocklist. Their proposed solution was to change the IP of the VPS, but that’s just agreeing to play whack-a-mole with a bad faith actor.

There should be a path to greater transparency and accountability from the SMTP cartels, but I’m at a loss as to how that can manifest.

I've run tiny smtp systems for 25 years or so. It can be done. I am based in the UK but at least one of my domains is a .net jobbie, so nominally American. That one still works fine and it is my (ltd) company domain, so all good. The MX records etc have moved around a bit but always very carefully.

It all starts around the IP address you are using. Is it "tainted"? is it in a tainted block? If it is then you need to either go elsewhere or clean it up and that takes a bit of time. By clean it up I mean apply for removal from the usual suspect's blocklists - Spamcop (lol), Spamhaus and all the rest that you can find.

Now setup PTR records. That has to be done by your ISP. If they can't do it for you, then find a new ISP. If you can't get PTR records to match A records then you may have to give up. One of the first checks an anti spam system will do is reverse look up an incoming IP address and compare it. Also that should match the HELO/EHLO announced by the SMTP MTA:

SMTP connection from IP address 12.13.14.15 HELO (my name is) smtp.example.co.uk

Receiver will check: smtp.example.co.uk == 12.13.14.15 AND 15.14.13.12.in-addr.arpa == smtp.example.co.uk.

Everyone gets their knickers in a twist about SPF, DKIM and DMARC but if you do not get the prior basics of IP -> A -> HELO -> PTR sorted out first then you will fail sooner or later. I also recommend that you ensure your MX records (receiving) match up too with your sending records. It means you can use mx is SPF, for example.

If you have multiple internet connections and IPs then be absolutely certain that your inbound and outbound IPs for SMTP match up.

Sorted all that? Cool, now proceed to SPF.

Most people fail at the PTR stage. If your ISP will not do PTR for you then you are probably screwed for self hosted SMTP. If you cannot change ISP to one that will, then you are really screwed. Sorry. In that case you will have to engage a service that will route SMTP on your behalf. It won't cost much but you won't own it and you will have to pay someone to do it. Soz.

What really gobbles my bobble is BIMI. Even without the paid-for certificate ($1500 is absurd), you can set it up to show your logo, and works on some providers (like yahoo). But careful, you have BIMI without the cert set up? Gmail spam-cans it.

Same with pgp, if you include your signature a lot of providers will immediately increase it's spam rating, usually high enough to land in spam (+7 pts usually), even though I doubt any spammer or scammer is inviting you to encrypted chats.

Email is broken because we all signed up for Gmail and didn't know better at the time.

SMTP and SIP are often held aloft as fucked up. My Dad's home telephony runs off a RPi and a Yealink DECT station and a dynamic DNS.

The modern internet might look a bit fucked up if you only look at the X/Facebook/webby wankery stuff but the real internet is functioning quite happily.

Sending and receiving as a wildcard alias is fantastic, and Fastmail allowing that was the reason I finally moved. I held out on that feature for a long time. (You can also do this in Exchange Online if you want to run a convoluted and officially unsupported configuration.)

I only twice had any kind of RBL issues, and one was a motivating factor in moving. I also got tired of worrying about patching and CVEs. I do that enough for my work.

> On February 20th 2024, the Hague Appellate court instructed dmarcian to block dmarcian.com visitors who originate from Dutch Internet Protocol addresses.

Apparently they had some trademark dispute with their local representative. The story on their website seems to be missing some key details to understand the whole picture.

But such emails can very well be (and in my experience usually are) spam. My spam filter will take the lack of implementation of these as a strong signal that the email may be spam, but does not think that just because these are implemented, the email isn't spam. As it should.

Just wanted to call this out because the article strongly implies that implementing these will let your email bypass spam filters.

I would also say a guide like this is helpful whenever you are using a custom domain with an email service, and therefore need to set these records yourself. Okay, you might not need to have an in-depth knowledge of these concepts, but it's certainly helpful to understand why your email provider is telling you to set all these weird DNS records.

Why don't they send DMARC reports?

Every little podunk email provider seems to be able to send me DMARC reports, as well as Microsoft, Yahoo!, Gmail, Xfinity, and AOL. Every morning I get a pile of them. But never a peep from Apple.

They like to reject our email for inscrutable reasons. We don't send bulk email. We only send email (almost entirely transactional) to about 5,000 paying customers who have accounts on our website.

Two days ago I was however delighted to receive a reply to an email I had sent to them on May 1 asking for assistance and clarification on why they block us. They told me that the website of one of my wife's employers (not our website or domain), featured in her email signature, was hosting malicious software. It isn't on any blocklists and I can't find any evidence of this currently being the case. Maybe it did at one point and they cleaned it up? I suspect Apple blocks and never revisits if the reason is still valid.

I noticed that cloudflare's email forwarder uses an ARC record and it works a treat.

My current delivery settings are strict so that only my server can delivery our email. I would think I could implement ARC in less strict manner and tighten it up as it becomes more common.

Does that seem reasonable? Any better ideas.

Unfortunately, the mail server of the forwarding domain doesn't seem to support ARC, so Gmail frequently throws away everything that doesn't have a DMARC header, since without DMARC the only other option is SPF, which doesn't work for forwarding.

Also it's kinda messed up how much of the small business sector is relying on AOL webmail to operate.

I remember Mail-in-a-Box being popular at one point, wondering if that's still the case.

Mail-in-a-box still works, though last time I checked they were on quite an old Ubuntu LTS release. There are a few pre-packaged docker containers too (i.e. docker-mailserver) which seem to be popular. I myself use Mailcow, but that's pretty heavy for "just" a mail server.

There's also a mail server that I can't for the life of me remember the name of, which packaged a whole bunch of stuff into one single binary you can run rather than use the classic "every part of the email delivery chain is a separate process" approach. I think it was written in Go?

maddy or mox?

I have email redirected from other domains into my (Gmail) inbox. For it to arrive, I use SRS, so the email is properly aligned and always makes it into my inbox. The problem is that some of that email is malicious. I have a choice of dropping those mails, and I never see a trace of it in my inbox, or forwarding them with SRS, and they look to Gmail like 100% perfectly good mails sent from my own domain (but still potentially malicious). It's annoying.

https://support.google.com/mail/answer/175365?hl=en

I don't know why Google want to force forwarders to do spam filtering.

Another option would be to have another inbox on the domain and have Gmail fetch with POP/IMAP, but many domain registrars don't have that service. Or is that what most people do?

For personal accounts I don't think you can convince Google to accept mail from incorrectly configured senders, you would need to use a different service if you must receive it. For business accounts they are less strict, although Google still randomly drops mail and I wouldn't personally use them for that reason and others (I think they have good suggestions for how mail should be set up, though). I'm not trying to sell you anything and haven't even tried them yet (and have some questions I haven't asked yet) but purelymail.com is one that seems promising to me in general and is inexpensive with no extra fee for custom domains (if you aren't attached to the way Google handles mail; I'm guessing the archive search and general interface will be more clunky). I'm guessing you can configure SpamAssasin to not strictly reject mail from incorrectly configured senders as long as they aren't on spam lists (they say they reject those immediately), but I'm not sure.

Thinking about it more I'm guessing Google requires forwarders to check spam due to not wanting customers to configure trusted ARC forwarders like Microsoft does and so not forwarding spam makes it easier to convince them the forwarder isn't a spammer trying to blame others for the spam (they might also compare forwarded spam with what they receive directly). If a forwarder uses ARC and can convince them they aren't spamming then Google should trust their SPF assessment if you just mean senders who only have SPF configured. But if you want to receive email from old timers who insist that email should have no way to detect spam there may be no way to do that with Google (or a number of other providers).

Its nothing to do with using domains other than gmail, its that you said you are actively relaying all mail into your gmail account and rewriting the sender as yourself, but then its annoying that spam gets marked as valid mail from your domain. Thats nobodies problem other than your own setup, and there are loads of other ways you could do it. But the way you chose to set it up is 'annoying'.

Sorry to insult you, but I feel that warrants letting you know you caused your own problem!

You could have ignored my question, you could have explained why I'm doing it wrong, hell you could have posted a random setup guide from Google and downvoted me. Instead you post this attack with no content, justification, or insight. You are being an ass. You are allowed to be, but I'm allowed to call it out.

And if you think I set my email up in a sub-optimal way, just so that I could ask for advice on HN, with the goal of pissing you off... you are delusional.

Give me a mail server that can use LE for certificates and I'll gladly give DKIM and DMARC a try...

--- SPF --- RFC5321.MailFrom domain: example2.com Auth Result: PASS DMARC Alignment: PASS

--- DKIM --- Domain: example2.com Algorithm: rsa-sha256 Auth Result: PASS DMARC Alignment: PASS

--- DMARC --- RFC5322.From domain: example2.com Policy (p=): none SPF: PASS DKIM: PASS DMARC Result: PASS

Start with something like `p=quarantine; pct=25` to have 25% of reported DMARC policy failures be marked for quarantine, review the reports after a week and then ramp it up to 50%, 75%, 100% every few days. Then if your domain is not having a significant percentage marked for quarantine in your DMARC reports after a week or two, switch to `p=reject; pct=100%` and continue to monitor the reports to make sure everything is good.

DMARC is not bulletproof to people using your domain as spam though because even with a reject policy, if SPF fails but DKIM passes, DMARC will pass or vice versa. It helps curb abuse and takes 15 minutes of effort to set up once but still is not enough to kill spam.