The problems I noticed were, it doesn't matter what the SPF and DKIM look like. If Google or Microsoft refuse to relay your email based on secret internal factors then you're out of business.
Best, and often practically only, way to avoid this problem is to buy your email services from Google Microsoft duopoly.
It wouldn't shock me if an email services monopoly/duopoly would prefer email spam's only workaround to be signing up for their services, instead of fixing the root of the spam problem.
Workaround (?) Buy their services for 1 (one) year and then move to something good (?)
Microsoft seems to be the most common culprit.
Agree.
I don’t understand why Microsoft are so bad at this?
They have access to a large percentage of all email traffic to train domain reputation, and spam detection models on yet seem to be notorious for both false positives and false negatives.
Google’s spam filters are far more accurate.
I think they cannot be bothered. Their customers are more tied in than Google's and are not going to switch because a small proportion of emails are rejected.
More specifically they seem to rely IP reputation rather than domain reputation, and IPs do change hands, especially for smaller servers.
Yes, and they do that routinely.
This attitude is just FUD.
The issue here generally boils down to the defining difference between a generalist Admin and a Messaging Admin. The generalist can follow instructions, and nearly all the instructions out there stop at the point where SPF/DKIM/DMARC are successfully implemented. A generalist worth their salt will then fill in the gaps if they can', and knows this isn't where you stop when you want mail deliverability. There's a higher bar.
If you follow instructions written by non-professionals blindly you don't ever reach the point where you get to quality work.
Google, Microsoft, and the other large ESPs don't refuse to relay your email based on secret internal factors. This is what the non-professional people say to falsely justify why they can't do something.
Google and Microsoft publish the internal factors they use in the form of whitepapers at the industry working group. Its not ready made, and there are a lot of them, and they may not release their specific implementation details, but the metrics are there and often are based in weighted form (reputation-based systems).
If you follow them correctly, and set up the appropriate reporting accounts, and maintain those accounts, you won't have these problems. You generally only have problems once you've violated guidelines continuously, which happens when you rely on, or are unable to discern between qualified and unqualified help.
The factors are published at https://www.m3aawg.org/
Every professional that specializes in email or messaging that I know of is well aware of this.
People don't have the same vitriol when it comes to comparing Generalist Admins to DBAs, and this is the same with any specialized niche.
If you need email and messaging to work in a complex environment, you hire a person that specializes in it.
MS flat out refuses to unblock our IP on their "outlook protection" racket despite many attempts through their self-service website.
Our IP is the same for the last ... ~5 years now? Is it because we did not buy a /24? is it because we are so small they have no real reputation data? who knows!
In my experience, MS and others give you a chance to correct the issue and if you don't do so in a timely manner (which requires qualified help), or it isn't corrected, then you get blackholed thereafter, and will remain that way up to years afterwards.
There is very little interaction from them, they assume you'll be professional enough to read the published literature and act accordingly.
The literature is a way of adding cost to those that would send spam, it also adds cost in other ways.
If I had to guess without knowing more, assuming you've correctly configured yourself locally (which may not actually be the case), I'd say it could be because of your ISP.
In recent years, with the depletion and exhaustion of IPv4 address space, many ISPs have moved towards CGNAT, where multiple customers share the same IP transparently. The ISP may do this without you knowing, but you'd have to have constructive knowledge in some fine print.
Subsequently by extension, they share the same reputation characteristics for that portion as others on the same network. Residential IP blocks get heavily punished or outright blocked on both sides.
There isn't this problem with IPv6 (no CGNAT and its complications).
I've seen this a few times now; even when the business purchased the business tier service for a static IP. In the client's case there was fine print that mattered that they didn't read in their service/purchase agreement.
The telltale sign that this might be your problem usually requires discussions with your ISP, but if you can't get to a qualified person on the line (from the backend/T2 team) you can run a test.
Have your networking guys check the traffic outbound and inbound (from public facing node) with a connection/packets that uses decrementing TTLs with either ICMP or TCP packets to get a path that aggregates each hop. Tracert or equivalent.
See if it is appearing to be routed through bogon network address space before it hits the wider network.
There are reserved addressing for CGNAT, and if the traffic is being routed across those address ranges this may be a large portion of your problem. This is just one of many things someone that specializes in messaging knows a thing or two about.
Graduated vendor responses occur with messaging, when you have little sound reputation at the start, getting everything right matters. Commercial places warm their domains and IP addresses up slowly over the span of a month. If you send to a provider like gmail, you need to click open those emails as mail that never gets read affects reputation per the whitepaper (m3aawg).
If you don't follow the practices the industry publishes, they don't relay the traffic.
> who knows!
I should know because I've worked in this area for quite a long time. It really is not black magick, and it is a specialized niche for a reason.
>The literature is a way of adding cost to those that would send spam, it also adds cost in other ways.
It is an oversimplified way of evaluating of consequences of overwhelming control the two monopolists have over access of small providers to email services. And it leads to wrong conclusions at least in respect to the range of its influence. Yes, it makes difficult life for the small amateur spammers as strongly as for beginner administrators and service providers.
However while for the determined spammers to hire experts isn't a problem for small entrepreneurs and for non-profit personal activities it is a blocking barrier.
Of course, they can use the services of established providers with all the limitations and other disadvantages of such solutions or accept slavery joining millions of users and firms accepting full and unlimited control of MS and Google (to their undisguised satisfaction).
About the consequences of sudden and totally unexpected interruption of email services without giving reasons we all can read often enough.
You are categorically mistaken and lack a true understanding of these things.
You are warned before you are outright banned. It shows up in the logs if you actually set that up properly.
It only appears like they cut you off because you ignore the things professionals pay attention to. Allowing an amateur to create and impose a problem and loss for other business is beyond stupid.
If you lack the expertise and context, you have no business dictating how things ought to be, and rabble rousing is vile.
>You are categorically mistaken and lack a true understanding of these things.
>If you lack the expertise and context, you have no business dictating how things ought to be, and rabble rousing is vile.
Your response seems to be typical for persons who are right because they are right - no args related to the content you respond to and ad personam args instead.
Thanks for the details! (It's not a residential IP, it's a VM at Hetzner.)
>If you don't follow the practices the industry publishes, they don't relay the traffic.
They are sending us email, we forward it, Gmail throttles it because it looks like spam, and then they don't accept the bounce for example :)
> I should know because I've worked in this area for quite a long time. It really is not black magick, and it is a specialized niche for a reason.
It's not black magic, it's abuse of market power.
If its Hetzner, they have a bad mail reputation as a result of failing to address issues with their shared resources in a timely manner.
They had at one point a persistent downgrade in mail reputation to the point where it was almost impossible to keep a working mail server with them that would be accepted by any major ESP.
They weren't particularly receptive to addressing support issues where their systems were breaking guidelines/RFCs impacting reputation, at least when I spoke with them about one of my servers a year or so back (which I promptly migrated to another provider).
From what I understand, there were egregious issues. Some of the rumors included source address validation not being done allowing DDOS and spoofing originating from these shared servers on the network block, issues with published PTR records, and a few other things. All of which heavily contribute to mail deliver-ability issues.
> They are sending us email, we forward it, Gmail throttles it because it looks like spam and then they don't accept the bounce for example.
If you are acting as a relay and forwarding mail from Google, to another Google recipient, you need to follow the mandatory guidelines.
https://support.google.com/a/answer/81126?hl=en
Naive relaying or forwarding can/will clobber headers, modifying the from header will also set off reputation issues. If you forward you need to be using ARC headers. The milter is a total pain to set up, validate, and get working.
High volume sending also has stringent requirements. You can read all about it at that link.
I'm confused. How could CGNAT affect a static IP?
The ISP had set it up so Egress traffic on the static IP was shared and included other residential traffic, and Ingress may have been mirrored or segmented by MAC.
It was unclear, and the ISP wasn't giving us much, it took months to track down and some really clever networking tests. The Network Engineer really came through there in collecting the info we needed to have a discussion with the ISP. I mention it to save others the headache, and labor involved.
Going IPv6 native corrected a whole host of issues.
> The IP doesn't change, so technically it's static. We never said it was exclusive.
That's a pretty wild take. Was there no alternative ISP?
Not in that particular locality, the only alternative was cellular with a cradlepoint at 10x spend for 1/2 the bandwidth, and connectivity issues in bad weather.
It was buried in the fine print related to IPv4 exhaustion.
The CFO I was working with was as flabbergasted as I after we'd found out, but I've seen it a few times now, even when there is another option because of a duopoly in an area.