For me, as someone with their own mail server, these technologies mostly serve to inform me that Russian IP addresses are still trying to send email in the name of my domain for some stupid reason.
It makes sense that people whose business is sending email know how to set up email correctly. I'm mostly surprised at how many legitimate sysadmins struggle with getting the basics correct. Surely those dozens of DMARC emails you get that your sendgrid email has been refused because of a bad SPF signature should set in motion some kind of plan to ask if maybe marketing is using them legitimately?
Automated signatures are of limited value but I rarely see rejections based on SPF and DKIM that are a mistake. Things are probably worse for big organizations but as a small email server, technical rejections are usually the right call. The only exception is mailing lists, but the dozens of people who still use those can usually figure out how to add an exception for them.
The problems I noticed were, it doesn't matter what the SPF and DKIM look like. If Google or Microsoft refuse to relay your email based on secret internal factors then you're out of business.
Best, and often practically only, way to avoid this problem is to buy your email services from Google Microsoft duopoly.
It wouldn't shock me if an email services monopoly/duopoly would prefer email spam's only workaround to be signing up for their services, instead of fixing the root of the spam problem.
Workaround (?) Buy their services for 1 (one) year and then move to something good (?)
Microsoft seems to be the most common culprit.
Agree.
I don’t understand why Microsoft are so bad at this?
They have access to a large percentage of all email traffic to train domain reputation, and spam detection models on yet seem to be notorious for both false positives and false negatives.
Google’s spam filters are far more accurate.
I think they cannot be bothered. Their customers are more tied in than Google's and are not going to switch because a small proportion of emails are rejected.
More specifically they seem to rely IP reputation rather than domain reputation, and IPs do change hands, especially for smaller servers.
Yes, and they do that routinely.
This attitude is just FUD.
The issue here generally boils down to the defining difference between a generalist Admin and a Messaging Admin. The generalist can follow instructions, and nearly all the instructions out there stop at the point where SPF/DKIM/DMARC are successfully implemented. A generalist worth their salt will then fill in the gaps if they can', and knows this isn't where you stop when you want mail deliverability. There's a higher bar.
If you follow instructions written by non-professionals blindly you don't ever reach the point where you get to quality work.
Google, Microsoft, and the other large ESPs don't refuse to relay your email based on secret internal factors. This is what the non-professional people say to falsely justify why they can't do something.
Google and Microsoft publish the internal factors they use in the form of whitepapers at the industry working group. Its not ready made, and there are a lot of them, and they may not release their specific implementation details, but the metrics are there and often are based in weighted form (reputation-based systems).
If you follow them correctly, and set up the appropriate reporting accounts, and maintain those accounts, you won't have these problems. You generally only have problems once you've violated guidelines continuously, which happens when you rely on, or are unable to discern between qualified and unqualified help.
The factors are published at https://www.m3aawg.org/
Every professional that specializes in email or messaging that I know of is well aware of this.
People don't have the same vitriol when it comes to comparing Generalist Admins to DBAs, and this is the same with any specialized niche.
If you need email and messaging to work in a complex environment, you hire a person that specializes in it.
MS flat out refuses to unblock our IP on their "outlook protection" racket despite many attempts through their self-service website.
Our IP is the same for the last ... ~5 years now? Is it because we did not buy a /24? is it because we are so small they have no real reputation data? who knows!
In my experience, MS and others give you a chance to correct the issue and if you don't do so in a timely manner (which requires qualified help), or it isn't corrected, then you get blackholed thereafter, and will remain that way up to years afterwards.
There is very little interaction from them, they assume you'll be professional enough to read the published literature and act accordingly.
The literature is a way of adding cost to those that would send spam, it also adds cost in other ways.
If I had to guess without knowing more, assuming you've correctly configured yourself locally (which may not actually be the case), I'd say it could be because of your ISP.
In recent years, with the depletion and exhaustion of IPv4 address space, many ISPs have moved towards CGNAT, where multiple customers share the same IP transparently. The ISP may do this without you knowing, but you'd have to have constructive knowledge in some fine print.
Subsequently by extension, they share the same reputation characteristics for that portion as others on the same network. Residential IP blocks get heavily punished or outright blocked on both sides.
There isn't this problem with IPv6 (no CGNAT and its complications).
I've seen this a few times now; even when the business purchased the business tier service for a static IP. In the client's case there was fine print that mattered that they didn't read in their service/purchase agreement.
The telltale sign that this might be your problem usually requires discussions with your ISP, but if you can't get to a qualified person on the line (from the backend/T2 team) you can run a test.
Have your networking guys check the traffic outbound and inbound (from public facing node) with a connection/packets that uses decrementing TTLs with either ICMP or TCP packets to get a path that aggregates each hop. Tracert or equivalent.
See if it is appearing to be routed through bogon network address space before it hits the wider network.
There are reserved addressing for CGNAT, and if the traffic is being routed across those address ranges this may be a large portion of your problem. This is just one of many things someone that specializes in messaging knows a thing or two about.
Graduated vendor responses occur with messaging, when you have little sound reputation at the start, getting everything right matters. Commercial places warm their domains and IP addresses up slowly over the span of a month. If you send to a provider like gmail, you need to click open those emails as mail that never gets read affects reputation per the whitepaper (m3aawg).
If you don't follow the practices the industry publishes, they don't relay the traffic.
> who knows!
I should know because I've worked in this area for quite a long time. It really is not black magick, and it is a specialized niche for a reason.
>The literature is a way of adding cost to those that would send spam, it also adds cost in other ways.
It is an oversimplified way of evaluating of consequences of overwhelming control the two monopolists have over access of small providers to email services. And it leads to wrong conclusions at least in respect to the range of its influence. Yes, it makes difficult life for the small amateur spammers as strongly as for beginner administrators and service providers.
However while for the determined spammers to hire experts isn't a problem for small entrepreneurs and for non-profit personal activities it is a blocking barrier.
Of course, they can use the services of established providers with all the limitations and other disadvantages of such solutions or accept slavery joining millions of users and firms accepting full and unlimited control of MS and Google (to their undisguised satisfaction).
About the consequences of sudden and totally unexpected interruption of email services without giving reasons we all can read often enough.
You are categorically mistaken and lack a true understanding of these things.
You are warned before you are outright banned. It shows up in the logs if you actually set that up properly.
It only appears like they cut you off because you ignore the things professionals pay attention to. Allowing an amateur to create and impose a problem and loss for other business is beyond stupid.
If you lack the expertise and context, you have no business dictating how things ought to be, and rabble rousing is vile.
>You are categorically mistaken and lack a true understanding of these things.
>If you lack the expertise and context, you have no business dictating how things ought to be, and rabble rousing is vile.
Your response seems to be typical for persons who are right because they are right - no args related to the content you respond to and ad personam args instead.
Thanks for the details! (It's not a residential IP, it's a VM at Hetzner.)
>If you don't follow the practices the industry publishes, they don't relay the traffic.
They are sending us email, we forward it, Gmail throttles it because it looks like spam, and then they don't accept the bounce for example :)
> I should know because I've worked in this area for quite a long time. It really is not black magick, and it is a specialized niche for a reason.
It's not black magic, it's abuse of market power.
I'm confused. How could CGNAT affect a static IP?
The ISP had set it up so Egress traffic on the static IP was shared and included other residential traffic, and Ingress may have been mirrored or segmented by MAC.
It was unclear, and the ISP wasn't giving us much, it took months to track down and some really clever networking tests. The Network Engineer really came through there in collecting the info we needed to have a discussion with the ISP. I mention it to save others the headache, and labor involved.
Going IPv6 native corrected a whole host of issues.
> The IP doesn't change, so technically it's static. We never said it was exclusive.
That's a pretty wild take. Was there no alternative ISP?
> Russian IP addresses are still trying to send email in the name of my domain for some stupid reason
For what it's worth, I've started seeing cybersecurity insurers requiring riders and extra payments if you don't block Russian IPs.
But there are big problems with mapping from IPs to countries. My IPv6 is detected as Russian, though it is London-located tunnel exit point and I'm in the Netherlands.
If your HE tunnelbroker account's country is set to Russia, you'll show up as from Russia for Google since HE publishes a geofeed of ip range -> user account country for them.[1] You should be able to change it on the settings page.[2]
If that's not it, you an see which database maps your IPv6 range to Russia and contact them to ask them to change it.[3]
Of course, if you have accounts with a Russian addresses, then things will revert.
[1] https://tunnelbroker.net/export/google
If it is a tunnel, then it might have been used by someone else before.
Those "London oblast" jokes don't come from nowhere.
Sounds like an issue with an outdated locally hosted IP2 Location database.
Google thinks it is in Russia too. And Cloudflare thinks the same.
If it's Hurricane Electrics tunnel I've had similiar issues, I think they use Russian blocks for their IPv6 tunnel since the abuse potential is so high and they don't want to deal with it so they just bundle all their shit with Russia and move on.
maybe you actually have a MITM proxy stealing all of your traffic and keystrokes
Sounds more like IP isn't a reliable factor to determine location. Not that this would be bad though.
Ive got a server hosting a number of things, amd monitoring setup for a lot of stats. Got tired of seeing blips because various countries were beating on my server, not a DoS, but enough requests to notice, and sometimes generate an alert. I blocked 7 countries, in full, and the impact was fantastic. No more 2gb of logs generated every day by countries that have no business accessing my server.
Unless you own a global business, i see no reason to even allow other countries access. The potential for attacks is too great, especially from some very specific countries.
I'm the CTO of a US-based insurance company. Apart from some reinsurers in London and Bermuda, and a couple contractors in Canada, we don't do business outside the US. We've blocked all countries except those, and it has cut down massively on the folks attacking us.
Lots of companies do this on their websites now using cloud flare or something similar. It’s practical. Still it’s frustrating as a user when you’re traveling over in Europe and can’t access your accounts to pay bills or whatnot.
Next time I travel overseas I'll have a VPN ready.
My bank had some technical problem that prevented access from overseas last time I traveled and I couldn't access my account (which was extremely inconvenient).
Most banks that will work with. For what ever reason the bank I now use knows most vpn providers and completely blocks all traffic from them so using a vpn is not an option either. The “vpn” I’ll have to use is tunneling back to my home ip. It’s actually quite frustrating.
Commercial VPNs are often blocked too. I found a p2p vpn to my home network + ssh socks5 proxy to work well.
Have you considered the additional cost of making it harder for your customers to do business with you, as well as the limited visibility that you set up for attacks that may become multi-stage in nature later?
You never see or collect the information by blocking everything at the outset.
In a world where you can proxy past these blocks fairly trivially, that's information you don't have for attribution later.
Defense in depth, or layered defenses are a best approach, but not if they blind you equally.
As someone who has whitelisted only US IP address space for my employer and blocked everything else I can attest that is DRASTICALLY reduces hostile traffic to us. I have an RDP honeypot that was blocking dozens of IPs every day before the whitelist and now it blocks 1 or 2 a day.
Kinda similar, but when I looked at the finances, I was surprised by how much money we're getting from places like the Cayman Islands, Switzerland, and the Emirates.
> I blocked 7 countries
Russia, China, Nigeria, Romania, North Korea, Iran and Belarus [1]?
[1] https://www.ox.ac.uk/news/2024-04-10-world-first-cybercrime-...
How/why did you pick these 7?
Using your link: Ukraine, USA, UK, Brazil, & India all rank higher than Iran and Belarus. US & Ukraine rank higher than Nigeria and Romania.
We (a US org) block all countries listed on the OFAC list
https://ofac.treasury.gov/sanctions-programs-and-country-inf...
Those countries likely have a higher chance of real traffic as well. If I’m doing business in Nigeria then obviously I can’t block it even if it ranks high on the threat level.
Yes, obviously you don't block the countries you plan to do business with. I got that much.
It probably makes sense to leave the US out of the list, assuming the CableNinja is in North America.
The rest seems pretty arbitrarily chosen, though. JumpCrisscross gave no additional context to why they left out Ukraine, Brazil, India, UK, when picking countries from the list they linked. They have higher cybercrime index ranks.
Whether they have a higher chance of "real" traffic is highly dependent on the business in question.
I'm sure there is some amount of thought behind the choice, beyond just using the index, which is why I'm asking.
Let me throw out a guess: Ukraine is a wartime ally, Brazil is the seventh largest country in the world, India is the first, and the UK speaks English and has a lot of connections to USA.
They're each also popular IT outsourcing destinations who aren't sanctioned. You may not do business in Ukraine or Brazil, but chances are one of your customers or contractors do, and blocking those IPs isn't usually in the first or second swipe. (If you're blocking the UK and India, you're probably blocking all foreign IPs.)
Romania!? I did a double-take, as it is a member of the European Union. I would think if their cyber-reputation was so terrible, there would be pressure from inside the EU to fix it.
They’re a small economy with lots of hostile traffic, so while in the EU and not sanctioned like the rest of the bunch, I’ve commonly seen them on the chopping block.
Pretty close tbh. Sub romania for brazil, and nigeria, for... i dont remember right now
just close the tcp sockets and you wont even notice them trying to connect and failing
do you also log everyone who looks at your house? it's a self inflicted problem
At least in the case of VPS my experience has been 99% failed ssh attempts. I just use nftables to rate limit those to 2 failed attempts per minute. Log size is quite modest and can easily filter out failed attempts when viewing.
Ok but you can’t block someone else using their own IPs to send email.
If you set DMARC to report, you’ll get notices from remote email systems when they receive noncompliant emails with your domain in the Envelope From field. Those reports are where you’ll see Russian IP addresses show up when they are trying to spoof your emails.
But there is no way to block them because neither the senders nor receivers are on your infrastructure. The best you can do is set a reject DMARC policy and hope everyone follows it.
As a non-email guy, I can tell you that if a system that boils down to having an (optionally certified?) key requires much more than just putting it into a folder with a domain name and running a service, it’s badly designed and has unnecessary complexity. Which will result into abusers having more expertise than legitimate users. The fact that you can “get” DMARC SPF DKIM wrong, while it’s basically a hard requirement for operation, is just screaming something important to the email software.
As a generalist admin, would you say the same about DBA operations or would you say that's just not my specialty?
The reasoning you provide doesn't differentiate, and speaks more of frustration which naturally comes with any area you aren't steeped in, or knowledgeable about.
Frustration doesn't come naturally. It comes with shitty software design.
"I don't know" is not a problem, you learn and you know, no frustration.
The problem is "I spent N hours/days on a thing that everyone does and which is a 99.99% of use cases and boils down to just having a keyfile in a proper(?) location and this knowledge doesn't translate effing nowhere".
would you say the same about DBA operations or would you say that's just not my specialty
It depends on the absurdity of the complexity of setting something up, not on operations themselves. Getting some results is absurdly complex -- not naturally complex and not necessarily very complex, just much more complex than the nature of the result itself.
For example, that's how you were supposed to install openvpn before angristan scripts: https://www.digitalocean.com/community/tutorials/how-to-set-... . To save someone a click, it's 50 pages "installation tutorial" with around 50 commands and a dozen of config files. And guess what, it uses "easyrsa" package to "set up RSA PKI easily". So it's not how openvpn meant to be installed, but an "easy" way.
You are mistaken. Your reasoning is flawed because the heuristics you use are flawed, and the consequences of the heuristics are the reason you are frustrated.
There are critical tools that you clearly have not learned, and likely were never taught. Tools that have been around since the time of the Greeks.
This is evident in your use of poorly defined language running you indirectly in a circular path (trauma/torture loop).
There is irreducible complexity in software. Domain knowledge is needed to use complex software for purpose.
The script you say makes assumptive choices for you. What will you do now that RSA has practically become broken at small key sizes, and instead you need to use a different algorithm?
Do you know how to transition this without starting from scratch, or have you become corrupted by dependency, on someone who provided that for you that did have that knowledge? Are you helpless to do anything but wait.
If you want to correct the underlying reason for your troubles, I'd suggest going over the associated material covered in a Trivium based curricula.
It will require unlearning bad heuristics and re-learning good heuristics. It requires a lot of effort and constant attention until you've got your thought processes fixed and these provide the basics for rational thought.
You should have been taught these things in school.
Logic (Aristotle), Philosophy (metaphysical objectivity, identity and its requirements), Argumentation, Descartes Method, and Kant with regards to A priori knowledge, reasoning, and argumentation.
Small things with an outsized bigger impact.
If you can't understand what is written in the whitepapers, you have no hope of following the conformant requirements.
Software reduces to practice the requirements of business logic, which is described in those whitepapers.
Sometimes its irreducible, and you have to approximate, and they won't hand this ready-made to people that aren't willing to put the time cost and professional skill needed to do so correctly.
You have to offer tribute, in the form of expertise, and time to benefit from these systems. As you have to do for any other specialized career.
To summarize. His complaint is common tasks involving commonly used software that are fairly simple but the software remains obtuse for some reason.
Your response is that he ought to read the standards and implement things himself. That the frustration is due to a skill issue, not to deficiencies in the software.
Or do I misunderstand?
I feel like the only thing missing here is the recommendation to do it all in assembler. To "build character" or something.
I suppose that technically you're correct, in the sense that if he were more skilled he likely wouldn't be as frustrated. Such an observation hardly invalidates the complaint about poorly designed software though.
There's nothing wrong with someone who wants to roll their own but most people most of the time want an out of the box solution. It's inevitable given the level of complexity involved in the modern tech stack. Building all of it from scratch by yourself simply isn't realistic.
You read that right for the most part except you missed important nuance.
You have to understand the tasks themselves are not and cannot ever be simple because of the adversarial nature imposed by bad actors.
You can point at a small component piece and say that's a simple task, but taken in the full real working context its not at all simple because there were other requirements that were ignored when viewed in isolation that are crucial to continued function in a useful way.
The frustration is due to a skill issue, anyone that could set a system up without issue would, and there would be no frustration if that were the case.
Importantly also, this isn't a software problem, its a problem that cannot ever be completely solved by software. There are problems that computation simply cannot solve directly. This is one of them. Its touched on in Automata theory under the Limits of Computation.
Anytime you have two different underlying states whose structure is identical when examined (a single state that cannot be differentiated) it falls into one of these type of problems. Reputation systems are a form of approximation for hidden state systems used to differentiate in such cases by skewing it so those that those who abuse the system are limited and quarantined, whereas those that don't can use the system. The hidden states are required to make these systems work and retain usefulness.
The alternative is no communication at all because resources a limited, and the SNR doesn't allow differentiation putting that cost on every reader who will stop using such systems because it makes them useless and the cost is unreasonable.
The requirements and cost that result from implementation of the whitepapers requirements keep the systems useful. Not everyone should be running their own server largely because they aren't appropriately qualified to fulfill their responsibilities and obligations in doing so, and as a result of that lack of expertise cause issues for other businesses imposing cost when they are allowed to do so.
The alternative, having no requirements is having no messaging at all. You literally can't have it both ways.
The complexity involved is why Messaging and Email Systems are their own subspecialty within IT.
> Building all of it from scratch by yourself simply isn't realistic
You don't build it all from scratch. You configure the software someone else built from scratch appropriately to meet the implicit requirements to interoperate or you don't, and the consequence of failure is mail doesn't get accepted for those recipients at that provider.
As I said, non-professionals writing tutorials making it seem like this is simple, and people blaming their own ignorance on others; is where all the hardship is coming from.
It isn't simple at all, if it were an average child could do it.
I can tell you from experience, nearly every single postfix stack that I've walked onto the job and seen at a small business, lacked critical functionality in their configuration with only a single exception in a decade. That's thousands of instances that required standing up new infrastructure correctly, and they didn't have issues after that.
In nearly all of those cases a non-professional got hired, lied about their experience, and then set them up for failure and they got what they paid for, but didn't know it at the time.
It could have been set up correctly if the people were qualified, but they weren't and it wasn't, and its an ongoing cost because requirements change over time, when they change you change, or your system stops working.
Things like auditing and logging, rate limiting, alerting, migration, features like list-unsubscribe, and many other requirements... etc.
Most cases, people stop the configuration at the point where an email technically goes out and they call it a day, up until calamity strikes because they didn't pay attention to important things.
There are people who pay for the advice and are told months in advance if you keep doing this you'll wake up one day unexpectedly and find no email can go out, and they don't stop. They have to learn it the hard way.
Imposter syndrome is a thing in the industry, but there are also a lot of imposters pretending to be professionals as well.
Sure, I won't disagree that there exist plenty of unqualified people doing things wrong while pretending that they know what they're doing. That seems obvious enough in the general sense.
I'll also agree that there are systems which exist that for whatever reason can't realistically be simplified.
However, on what basis do you claim that email - or rather email anti-abuse - qualifies as such?
> The alternative, having no requirements is having no messaging at all. You literally can't have it both ways.
You seem to be implying that the usefulness of the system derives from or otherwise depends on the difficulty of configuring it. However it doesn't seem to me that you've provided evidence of that. On the contrary, isn't the entire point of a reputation system that it avoids such gatekeeping by depending on historical behavior rather than some arbitrary barrier to entry?
I would make my own claim. That there exist software implementations that are far more complex than they realistically need to be, often because the thing being implemented has evolved over time and the resources or motivation or whatever needed to re-engineer and rewrite the implementation aren't available.
I would also claim that sometimes software has shitty UX for no better reason than the person developing it doesn't understand the needs of (some subset of) the people using it.
When configuring a network node to exchange messages in a really quite primitive protocol requires professional expertise to do correctly I'd say that's a clear indication that something is very wrong somewhere in the stack. Where exactly is certainly up for debate but a well behaved entity should not find it difficult to self host such basic functionality.
Communication as a whole, not just email. The failures to address this, point to an inherent limitation of the systems we've built for computation. You'll have to revisit automata theory, and have some knowledge of why CPUs are able to do work at the lowest levels of abstraction.
Boiling it down, it comes down to system properties that are preserved, and Von-Neumann Architecture acts as a DFA. Computers act on a single state at any one time, moving only ever one edge on a abstract state graph at each operation.
People generally are considered NFAs that can operate on multiple states and decompose states, and have a wider range of problems in the types of problems we can solve.
This is abstract but the gist is, the computer follows an abstract rail of decisions that is really quite dumb, but necessarily so, and it doesn't halt or runaway except with bugs, because we preserve properties limiting the math to areas where it cannot have the problems except outside the working environment (i.e. power loss, hardware failure etc).
There's a reduction to an abstract algebra system inherent in the architecture by preserving certain properties in the design. You first run across this paradigm in first year EE (Systems and Signals) and a course is available on OCW if you haven't taken that, detailed knowledge is not needed though unless you plan on designing these hardware systems.
Any time you have an underlying state that is both true and false given the same state (the message), and in adversarial environments the property requirements for computation are broken. This can naturally occurs in any communication system, and the hoops we have to jump through that we add on in the form of requirements is defining a way to differentiate that hidden state indirectly by the presence of the requirements which good actors follow more closely than bad actors. This is decomposing the state in structure from an NFA type problem to a series of DFA type problems as I'm sure you might recall from your Compiler Design Courses (if you've taken them), or learned from the Dragon Book.
Any message sent must be sent in an identical structure. Any bad actor will adapt to ensure their messages get sent through flooding and raising the noise floor. Any good actor will adapt in a number of ways sometimes by no longer using a system that doesn't provide benefit. You can only operate on the same state.
If you can only process and interact with the message structure itself. No computation system will ever be able to skew what is sent or received so that only the legitimate messages are sent, and the illegitimate ones aren't. Everything goes through the same point. With everything going through, the noise floor is so high nothing gets through, and communication is the sharing of meaning/signal between two parties, people adapt and abandon the system for systems that work.
The core issue is a fundamental computer science issue.
When a computer hardware system first boots up, the bringup stage in hardware sets up the constraints needed to do work. Ask yourself what about the design of computers today prevents the classic unsolved computer science problems and you'll find this staring back. Halting and Decidability (usually).
There are impossible to solve problems, because we've proven that math is incomplete, which impacts on decideability.
Computers work on specific principles, and when you don't understand or know how those work you can easily jump to magical conclusions that simply do not work or have a basis in reality.
A very simple example of this same problem demonstrates this. You are given two spreadsheets without distinct (unique) names. You have 10,000 rows of employees, and you have a list to deactivate 400 people's accounts in an hour, the list of people to be deactivated is by name. You have a script to do all that's necessary for that for individual accounts given a specific account, but some of those people's names are identical to others, and they are different people. The first match you happen to see is the CEO.
How do you solve this?
If you pass the names to automation blindly, you'll deactivate people's accounts that should not be deactivated and you get fired. If you don't in the time period alotted, your fired. How do you solve this?
The only possible way to solve this given the constraints is you ask for a list that includes a unique identifier for the people that need to be deactivated, and a matching list to work from and then the automation can work.
If you just did it blindly, the computer would do it blindly. It has no way to know otherwise. The function is a deactivation so it would deactivate every item passed to it, ending in... you are fired.
There is no other way that does not result in you being fired. Fuzzy matching doesn't work because without the identifier you know that one of those two or three needs to be deactivated but you don't know which, and getting it wrong ends in you being fired. This type of problem is called decidability.
You get the same types of this subtle problem all over automation in different forms. Like in Linux with ldd's output, which is why it fails silently when passed to any automation. The overloaded null state means two different things, and its undecidable when it flattens, and if you examine it carefully it breaks regular expressions. Why? That property isn't preserved.
You are used to dealing with the top of the stack where these properties are preserved, unless you or others break them with a bug.
You’re just arrogant here imo, and I regret the time spent on elaborating. Your comment is straight from the 25 years ago when it was normal to read toxic lectures to lame noobs on forums and create software that has no last mile connection to reality. I’m glad that that era is long over except for a few remnants. Have a regular day.
You have willfully blinded yourself to opportunities that if taken to heart could have prevented yourself a world of suffering.
You mistake the environment you are in, and where it is going, which will threaten your ability to survive at some point as you are helplessly dependent on an environment that will cease to exist in the near future.
This was neither toxic, nor arrogant, just the facts and advice provided in good will and faith, something that is vanishing along with tolerance, and those facts should frighten you because they have detrimental outcomes as a consequence for you.
You didn't want to hear it because of indoctrination, and an inability to to comprehend. As a result, you have only yourself to blame for the choices you've made and what predictably comes next. Struggle and frustration.
Those that can't help themselves won't be helped by others. Those that cannot learn and adapt doom themselves by their own choices. Darwin's fitness.
A time is coming where the blind in their unpredictable and crazy behavior may be given a final mercy that can't be taken back, for the good of all because these people are a detriment to all if left alone. Historically, this is well known and it wasn't until modern times that we had the resources to care for such illness in seeming perpetuity.
Until things change, you've made it clear the only path for you is to struggle on needlessly, without any help, and let it distort you in a spiral of madness until you succumb to your self-fulfilling prophecy and break moreso than you already have.
Slapping goodwill and advice down falsely believing its toxic, when in fact its just unpleasant/harsh truths you weren't strong enough or willing to face speaks greatly to the character and outcomes you will face.
There are people who happen to know more than you do, about a great many things; because you were given a poor foundation purposefully. Its not arrogance to want to give people the opportunities that an education they should have been given as a child provides. The alternative is delusional adult children running amok destroying the pillars of their own survival.
You tread forward down the path those malevolent people laid for you, deceived, and never straying; biting any hand that offers help. Its sad because its preventable and needless.
I'll pray you revisit this when you get tired of the madness you put yourself through.
In most organizations there is no point in a sysadmin to spend the effort in understanding how to set it up correctly as Marketing has got more authority on email. Marketing will simply demand changes to the config that they do not understand and there is nothing you can do to stop it as they will have the CEO on their side.
> Marketing will simply demand changes to the config that they do not understand and there is nothing you can do to stop it as they will have the CEO on their side.
Marketing should get their own (sub)domain for sending their missives, that way the primary corporate domain's reputation is not harmed.
Unless you want to run the risk of outgoing e-mails from Finance / Accounts Receivable to be sent to other companies' Junk folder.
It's amusing to see this advice in this thread contrasted with the recent Troy Hunt phishing attack thread where folks are complaining about companies like Microsoft having dozens of varying domain names.
> […] about companies like Microsoft having dozens of varying domain names.
There's a difference between one and dozens, and even between one dozen and dozens.
Most companies are not of Microsoft's size either: just having news.example.com would probably be sufficient for a lot places.
This is email marketing 101, HN'ers are massively overstating how many domains are getting blacklisted because of "marketing".
Orgs like that will hire consultants like me when they can't figure out why their stuff isn't landing in the inbox. Then 3 months later their webdev will somehow delete the entire zone when adding their A record.
You mean like the time I had a salesperson demanding that we turn off Cloudflare across our entire domain because he'd read some random article somewhere saying we should?
The goal of sales isn't to block upto a 1/3 of world wide traffic. Turning off Cloudfare means more traffic and more sales are not blocked. Did you even read the article or did you dismiss it because it came from 'sales'.
Sales: "look, I turned this off and sales went way up"
Security: "We had to cancel every single one of those sales because they came from stolen credit cards. It's costing us more to deal with that then we are earning"
Accounting: "We're measuring a pretty big loss because security cancelled legitimate purchases together with fake ones and now clients are leaving."
Which is another reason to strictly enforce SPF and DKIM, in my book. Let marketing break those policies, that way I don't need to bother with reading your company's spam!
Marketing decides on DKIM and SPF ?
The problem I personally ran into as a one person IT department was that the VP of marketing had more power over me, as a manager, and that meant more to my supervisor (the CEO) than me fighting to do things as correctly as possible. I was seen as a roadblock or speed bump. So, they may not decide on DKIM and SPF, but if marketing isn’t happy then their negativity could cause push back that forces changes that may technically not be good for the company.
I’ve abandoned that role and have gone back to an IC role and I’m much happier for it.
As long as you're not breaking the law / hurting people, does the struggle really matter? The best way I've been able to make people listen to me is by just presenting them with options and results.
If you do it this hacky way - we run this risk and this bad thing can happen etc. After a few times they see the consequence of their decisions people start paying attention to you. Do it a few more and now the company will have an "institutional knowledge" that you are usually right, and even if the manager leave, you still end up like the go-to guy on how to ship.
And sometimes the marketing people might end up being correct! I've once actually battled to "do the correct thing" (way back in the day it was a ruby on rails modeling I think) and the product owner was like - just do it this hacky way I don't care ... I did it the hacky way and you know what - it was the right call - we never changed it again and the business knowledge we got from it was actually valuable.
In the end, for me personally, I give people respect for their roles and the benefit of the doubt that they're in the position for the right reasons. But when I don't get that kind of thing in return then it just pisses me off. What I realized along the way is that I don't want to be in charge of things like this, it's simply not for me, at the very least it isn't on that team. Maybe that will change with the right people but the whole thing soured me on management in general and I will avoid it like the plague.
I'm pretty bitter about it all still, but it's a combination of a lot of things beyond this particular bit I shared. All I can say is I'm glad I am no longer in that role, it was slowly killing me.
The biggest problem there is that it's a statistical gamble, and often times the damage isn't apparent for months or years later, which is plenty far enough removed from the decision that the manager isn't going to remember let alone realize "he told me so." And you reporting "I told you so" even in very easy, factual, and respectful professional language will typicall not be well received. There's also a decent chance that when the thing breaks or you get breached, you'll be blamed for it, or at least be on the defensive.
Now that said, I've worked with a lot of IT/engineering people who are pretty obstructionist to normal business operations and sometimes need to be told, "yeah, we're accepting the risk, move forward with the plan." Sometimes it's for good reasons, other times it's just our normal humanity asserting itself in different ways. It's a hard problem for sure.
Indirectly, yes. Since they don't understand the details, management just "wants it to work". So too many email admins just give up and make their sending policies as permissive as they can to account for whatever new service marketing is using at the time.
DMARC is required for BIMI, and marketing wants that logo to show up in the Gmail app next to your mail
even worse when you have even less control than that, if you run some type of hosting and are trying to convince non-technical clients (or even worse, non technical clients who think they are technical) to “please just add this record exactly as it says here to your domain” and they’re somehow unable to for months and months
> "please just add this record exactly as it says here to your domain" and they’re somehow unable to for months and months
I ran into this helping a friend whose biz emails to gmail recipients were getting dropped; the IT dept of the umbrella corp wouldn't respond. Same to me when I sent the correct DMARC, SPF etc.
(My friend's biz was his own but it shared some resources with a larger corp.)
I eventually realized that the (wrong) DMARC reporting domain wasn't even registered. I did what you'd expect and I soon had DMARC reports for subsidiaries of the umbrella corp. My friend passed that up to the CEO and suddenly IT was responsive.
In the end, it turned out that IT was deliberately blocking his biz emails to his biz family members. After 10 years they suddenly decided that email to family+gmail was risky and that they were going to gaslight my friend about it. Because reasons.
That’s a wild story, thanks for sharing - I find interfacing with external IT teams extraordinarily frustrating. I suspect it’s because businesses often don’t manage their IT teams well or have a good process to expedite business -> IT requests that really should be super easy and provide a lot of tangible value for the amount of comparative effort involved.
I’ve run into outright malicious stuff internally like this, but never externally - I would probably go apoplectic if I was your friend
to be fair here: for a lot of companies, if the mass mailing stops, the money-flow stops then that's no good for anyone... so the CEO will probably err on the side of money, presumably.
Why would properly configuring SPF, DKIM, and DMARC stop the mass mailing, though?
As someone who set these up, I can tell you, the answer is rather simple:
- spammers have 1 system to set up in order to spam. They get it right.
- company admins have dozens of projects, of which this is a tiny one, with zero ROI to the bottom line (if people don't consider how critical security is). So they delay.
- companies often have dozens of systems integrated, when I set up DMARC/DKIM the first time for my company, a bunch of email tools broke, we had to do a bunch of leg work, took us a month end-to-end. The value was recognized when we almost lost 20k to a "ceo emails you" scam. But until then it wasn't a priority.
- we didn't even have a full IT, i just stepped in because I cared enough.
- my current company has a dedicated security team. These holes are plugged VERY quickly.
> that Russian IP addresses are still trying to send email in the name of my domain for some stupid reason
You can set your policy to reject, that will deter the Russians from using your domain.
I used to have my policy set to reject, but then I found out some part of an Enterprise Outlook mail filtering chain was rewriting the mail I sent before checking the DKIM signature. I can't fix stupid, especially for other parties, so I changed the policy to quarantine instead.
I doubt Russian spammers will care about the difference to be honest. If they accept that their email will be delivered to spam folders, why would they care that the email gets silently dropped? In neither case anyone is going to fall for them.
I am just having this problem. Actually getting SPF, DKIM and DMARC right and having a domain with a 0 spam score will still land you in the spam directory. It turns out, you need to have a "reputation"? before your email gets accepted into gmail. My head was spinning as to how that reputation will be built if your email just goes straight to spam.
But sure, Linkedin emails are definitively not spam and their dark-patterns at adding you at n+1 emailing list doesn't get them banned from the big (or any?) provider.
It's easy, you just have to have a regular, decently sized volume of non-spam emails, and suddenly your email stops being marked as spam!
The logic isn't even that bad. SPF and DKIM serve to prove to the email who the sender is. That doesn't mean much if the sender is a spammer. Verifying identity claims is only the first part in checking email for spam, the harder part is checking if that identity is someone you trust.
When you email Outlook or Google, you're better sending more than a few every single day, and the recipient better manually drag those emails from their spam folders to their inbox, or they're all being learned as spam.
And you have to build up the volume gradually. In the industry this is called "warming up IP addresses". See for example https://help.elasticemail.com/en/articles/2788598-how-to-war... or https://docs.aws.amazon.com/ses/latest/dg/dedicated-ip-warmi...
which goes to the original title. spammers are better that this stuff then regular businesses.
> you just have to have a regular, decently sized volume of non-spam emails
But if you have a regular decent size of emails coming from your domain, that is more likely to be spam than if you have a small number of intermittent emails coming from a domain.
so my personal domain just needs to send newsletters to millions of people, or ... how exactly? what's decent size? how frequently?
> It's easy, you just have to have a regular, decently sized volume of non-spam emails, and suddenly your email stops being marked as spam!
The domain is new and didn't send a single email until I tested it.
Edit: The domain is actually a bit old but was parked/inactive for a while, though the email was used only for receiving.
Yup, that'll get you stuck in spam limbo alright. Good luck climbing out if it if you're initiating conversations with anyone on Gmail or Outlook (or, even worse, corporate Outlook).
Those email services will usually have no trouble with replies to emails sent from their service, so if you get someone to email you first you'll save them the trouble of dragging your email from their spam folder to their inbox.
With Outlook, in my university the problem was that when we send emails they disappeared mid air, no bounce, no spam folder. The solution was that they must write an email from the Outlook address, after that we are added to a secret good list and we can write them.
I've had to figure out a problem with reaching university Outlook servers where the Outlook server didn't like the (spec compliant) way my email server was writing the From address and rewrote it halfway through the spam filtering chain.
Then it checked the DKIM signature on the message it REWROTE ON ITS OWN and decided that the signature didn't match, and rejected my email.
Corporate email stacks are hell.
This has almost always been the issue when working with clients on why our emails never appear. In the end they have some weird middleware that rewrites things and then the next level of the stack sees the middleware as the sender of the email for our domain which fails as it's not an approved sender by SPF and dkim signatures don't validate.
A fresh, plain setup on office 365 doesn't fail, but however their security department reconfigured things causes it to fail.
I've never been on the configuration side of M365 email like that, only basic cheap tier stuff and only briefly. I can't say what they're doing, but the same settings sending to practically any other email provider or even other 365 tenants works perfectly fine.
Do you have a write-up of this, anywhere? I'd appreciate the details (what format did it reject? what did it change it to? what version of Exchange?).
Not OP but I've also seen this. I think some of the servers in question were "outlook-protection" in their domain names. Some kind of managed service middleware in the stack to do additional scanning.
I second the request. A few years ago we switched to Google Workplace, but it would be nice to know. I would like to forward it to the sysadmins just in case we go back to our own server.
I worked on this for a while, at a time and in a market where most of our recipients had @hotmail addresses. I discovered that mass email sending was akin to a "pay-to-win" game.
We had/opted to acquire the services of a company "expert in email deliverability" (Return Path), who somehow provided detailed metrics of how our IPs were scored by MSFT. I always wondered why MSFT didn't provide those scores by themselves, and how a 3rd. party could have access to them.
Re. your comment... slow ramp-up is the only way, with constant monitoring of deliverability and consequent adjusting of recipients (i.e. removing those who do not open or hard-bounce). I did also wonder if paying that company perhaps gave us a headstart when adding new IPs...
It's almost like all those bad actors (linkedin) are owned and controlled by the big players (microsoft) that benefit from email being only commodity they can provide.
I think the domain rep is worth less than IP rep. I had occasional issues sending issues when I self hosted on a VPS. When I moved my domain to Fastmail I haven’t ever had my emails go to spam.
Most home and VPS IP ranges have negative rep.
As a tip, go to a VPS that's had a history of being very selective of allowing SMTP traffic but still allows after some kind of review. Cheap providers that never did any blocking probably have bad reputations for their entire address range.
I've been successfully using VPSes to send emails for 20 years.
I am sending from SES. Interestingly, I didn't have a problem getting the email delivered to inbox in fastmail despite having an aggressive "protection level".
This isn't a problem for personal emails, as after a request or two friends will unspam you. Google blackholes emails, breaking all mail logic (no bounce), so I assure you the SPAM folder is a good gmail sign.
I would imagine that on the corporate side, your employees could do the same. Beyond that, if you're sending spammy stuff, have unsubscribe headers and links in emails.
If it's a new domain, then your problem isn't reputation exactly, it's having a newly-registered domain. Buying a new domain, setting up the SPF, DKIM, and MARC, and then immediately spamming from it until it's banned everywhere a week later is standard spammer MO.
I've been self-hosting mail for me and my family for about 20 years and don't send nearly enough mail to have a "reputation" with anybody. Still, I don't have any problems with deliverability of mail.
> Actually getting SPF, DKIM and DMARC right and having a domain with a 0 spam score will still land you in the spam directory.
This little bit of wisdom gets passed around all the time, but it's actually not true. You can send email from a brand new domain to Google and Microsoft and whoever just fine. What you can't do is send email from a brand new domain, and a brand new email server--or an email server on a VPS, or an email server on a residential IP. Residential IP blocks are almost completely blocked, because of unsecured devices being used to send spam, and VPS blocks have the same problem. You can get around this by using a mail relay, or building your domains reputation on a server that already has a good reputation.
> or an email server on a VPS, or an email server on a residential IP
So what options are left for a self-hoster. Colo?
Find a host that cares about their reputation. It can be hard to know who responds to abuse reports, but if they mention it in their TOS that's a positive. Also, many hosts block outbound port 25 by default now; that's a positive sign as well.
The more effort you have to put in to use them to send mail, the more likely spammers don't use them, and the more likely their ip space has a positive or at least non-negative reputation for sending mail.
Get a business-grade connection from your ISP. Make sure they give you a static IP from their business side, and check its reputation before you set up email. If it has a bad reputation, make the ISP give you a different one.
SPF/DKIM is really about mail server reputation. So it mostly benefits larger servers like the ones run by Google, Microsoft and Yahoo. Unfortunately, that means that attempts by those larger providers to combat spam using such reputation will naturally hurt smaller providers. So the actual effects of SPF/DKIM are on the whole negative.
The root problem is that we don't actually need to keep track of email server reputation. No one says to themselves "Huh, this is from a Gmail address, it must be legit". We really want to keep track of sender reputation. We need to be able to treat anonymous email differently than email from people we actually know. That implies that we have some work to do on the problem of identity. As it is, there is not even a way for a known email sender to securely introduce an unknown email sender. You know, the way that regular human people normally are able to transfer identities from one to the other.
>SPF/DKIM is really about mail server reputation. So it mostly benefits larger servers like the ones run by Google, Microsoft and Yahoo. Unfortunately, that means that attempts by those larger providers to combat span using such reputation will naturally hurt smaller providers. So the actual effects of SPF/DKIM are on the whole negative.
That paragraph is incorrect. SPF/DKIM is not about reputation. The main purpose is preventing domain impersonation from unauthorized senders. E.g. mail servers will reject fake emails from "upofadown@microsoft.com" because you don't control any email servers that's whitelisted in microsoft.com DNS TXT records.
E.g. I was able to register a brand new .com address and then successfully send to gmail and MS Outlook accounts within minutes because I had proper SPF/DKIM in the DNS records for that new domain. That new domain had zero reputation and yet Gmail accepted it because SPF/DKIM was configured correctly -- and -- the underlying ip address of the server it came from had a good reputation.
If SPF/DKIM was truly about "reputation", it would mean I'd have to wait days or months for reputation history to build up before Gmail accepted it.
preventing impersonation is an important part on correctly attributing reputation to source domains.
Exactly. The people bashing SPF & DKIM don't seem to understand their intended purpose.
And it will mysteriously _stop_ being able to send mail to Google despite you doing everything right, because of whatever nonsense they use to determine reputation.
I am curious as to your experience with this.
Over the years, I have administered a few dozen small to medium domains (depending on the domain 10s to 10,000s emails per month) and the only thing that has ever affected delivery is the reputation of the sending IP address of the mail server (and ensuring DKIM/SPF alignment in more recent years).
I don't think this is correct? SPF and DKIM are about ensuring that the server actually is who it says it is, not about its reputation. In other words, when you receive an email that claims to be from Gmail, SPF and DKIM help you ensure that's where the letter actually came from, not from a server just pretending to be one of Gmail's servers.
SPF more like whether the email came from a server that's authorized to send emails on behalf of a particular domain.
> Unfortunately, that means that attempts by those larger providers to combat spam using such reputation will naturally hurt smaller providers
Tin-foil hat time, but I've always thought there was nothing unintentional or "unfortunate" (from Google's perspective) about this.
> That implies that we have some work to do on the problem of identity. As it is, there is not even a way for a known email sender to securely introduce an unknown email sender.
There is: gpg/pgp signature, but many people find it complicated, primarily because they are reluctant to read the documentation. And it’s popular to criticize it, especially here on HN, in favor of various half-baked alternatives.
I think everyone can agree that any technology that "isn't complicated if you read the documentation" is by definition complicated. I don't need to read the documentation for Gmail to use Gmail successfully.
Could I, as a trained programmer, use PGP and GPG? I'm sure I could if I spent some time reading about it. Could my 90 year old grandmother, who is otherwise quite comfortable with email and whatsapp? No, not to any meaningful extent.
There are times you need complexity enough to be worth training costs. There is one universal word "nanana", and maybe babies cry (it seems many babies have unique cries for different needs: I suspect that is training between babies and their parents - anyone done research on this?). All other language is because you spend years in training. If you can read this or write a response that implies training.
The important point from the above is it was worth the effort to learn. The only person I know who is a strong advocate of PGP was a missionary to Romania before the iron curtain fell - he had strong reason to hide what he was saying from government level actors and even today still is willing for extra effort to protect himself. For most of us though our threat profile isn't (or doesn't seem to be) that high and so learning how to use the tool isn't worth it.
I absolutely agree that PGP/GPG have important use-cases for which they are the state of the art and well worth learning. This doesn't mean that they are not complicated technologies though.
I highly disagree with this.
I just left a couple of comments regarding the use of "strtok". Its use is straightforward, just RTFM. Those were the golden days when people were less reluctant to read documentation. You could not even install Linux back then without an installation guide of some sort. You still need it for Gentoo, perhaps even Arch or Void. Are they wrong? No, just different target audience. If you do not want to become a "power user", that is fine.
My grandma can barely handle the TV controller. So what? I am really against dumbing things down, called "ease-of-access" or whatever they call it these days.
I agree on that, however, that GPG / PGP signatures should be more visible and whatnot, just add some visual feedback (verified? legit?, etc.), and some e-mail service providers actually do this.
> Are they wrong? No, just different target audience. If you do not want to become a "power user", that is fine.
Complicated doesn't mean bad. I'm not claiming that PGP or GPG are bad technologies because they are complicated to use.
> My grandma can barely handle the TV controller. So what? I am really against dumbing things down, called "ease-of-access" or whatever they call it these days.
The "so what" is simple: PGP is not the right anti-spam solution for your grandma, or mine, or any users like them. This is the context of this conversation: is PGP a good-enough answer for how to establish identity for email in the interest of anti-spam and anti-scam efforts? And the answer is a clear and resounding no, not for the vast majority of users of email.
This, again, doesn't mean that PGP/GPG are bad technologies - they are very good for certain use cases and certain users.
So what is a good-enough answer for my grandma? :P
There's no great answer, unfortunately. Gmail and other off-the-shelf email providers handle much of the spam and some of the scam prevention for you, but you still need to exercise caution on your own.
> many people find it complicated
That's what kills a lot of these "perfect" implementations.
HN members tend to be nerds, and we don't really have an issue with setting stuff up (many HN IDs, for instance, have Keybase auths).
Most non-HN types have no patience for that stuff. Security needs to be made accessible and easy-to-use, before the vast majority of folks will implement it. That's the single biggest conundrum, IMNSHO.
We really need a "Trust on First Use"(TOFU) system for messaging, that can be verifified, or pre-trusted offline, face to face. It'd be awfully nice for your bank to give you some thing that you can later verify that any communication from them (web site, online banking, text message, email, etc) are legit and verified.
Or if we can't trust users to handle TOFU, then some token/unique address/whatever that we can exchange face to face to enable trusted communication.
> We need to be able to treat anonymous email differently than email from people we actually know.
The simplest solution to that would be an "only show me emails from people in my address book" filter. That would mostly echo how we treat user trust on all other platforms. Genuinely surprised this doesn't exist in most email clients (or does it and I have just overlooked it so far?)
Of course that's only a partial solution and wouldn't work for accounts where you expect unsolicited mails from people you don't know. I'd see it more as a "low-hanging fruit" solution. You could also expand the heuristic, e.g. also consider previous conversations, mailing lists, etc.
(Interestingly, the "introduce a friend" functionality would come for free: You can already send contact details as a VCard in an attachment. When receiving such a mail, some email clients will show a button to quickly add the contact to the address book.)
> only a partial solution and wouldn't work for accounts where you expect unsolicited mails from people you don't know.
I actually think this would work fine. Imagine a quarantine inbox for new emailers that the user must scan and approve/block. This is exactly what hey has implemented.
> The root problem is that we don't actually need to keep track of email server reputation.
We actually do. Is the server allowing anyone to sign up so that it's sending 99% spam, or does it have a lot of anti-spam measures so sign-ups can't be automated and it blocks accounts as soon as it detects them sending spam?
> As it is, there is not even a way for a known email sender to securely introduce an unknown email sender. You know, the way that regular human people normally are able to transfer identities from one to the other.
That's exactly what PGP's web of trust model is for. Someone you know, and trust, can sign and send you a public key of someone that they trust.
This new key will be automatically trusted in your trust store because it's signed by someone you already trust, although in a lesser trust level to account for the degree of separation. If you later verify that key out of band you can upgrade it to a higher trust level.
SPF/DKIM, as well as TLS etc., is just stupid shit we do because we're too lazy and/or incompetent to make web of trust work for us. It's not a technology problem, it's a human problem.
> SPF/DKIM, as well as TLS etc., is just stupid shit we do because we're too lazy and/or incompetent to make web of trust work for us.
Having key signing parties for the entire world wide web does not seem scalable to me.
If you want to have complete trust in every key you hold then you need to validate it personally. This is exactly the same scaling factor as Signal or WhatsApp, for example.
Web of trust scales better than that, though. It gives you confidence in keys you haven't seen yet because they are signed by other keys that you do trust. The key signing parties strengthen the web of trust, making it more likely a potential correspondent will receive a key signed by someone they trust and therefore potentially not needing to verify it personally.
It all depends how much confidence you want to have for each key. At the end of the day there is no substitute for verifying each key personally if you want to be completely sure. PGP give you the option to hold keys with a lower level of confidence for e.g. less sensitive communications.
Well, yeah, we should use preexisting standards and OpenPGP would be perfectly fine here and is probably the best choice. That is a wheel we do not need to reinvent. But the actual system used to do the signatures and keep track of the reputation is the last thing we should be thinking about at this point. We should instead concentrate on how to create a system that the majority of people can use and understand. We should be concentrating on standardizing concepts...
Right, that is my point. I feel like there is a fundamental lack of understanding in the vast majority of the population about trust. We haven't helped by telling people "you can trust the little green padlock". Nobody asks "why should I trust it?". That is the problem. It really doesn't matter what technology we provide, so far none of it is really used by regular people to establish trust.
The other option, of course, is to design a trustless system, like BitCoin, but that has its own problems.
It's weird so see such a factually incorrect comment so high up on HN.
SPF/DKIM is literally how you establish sender identity instead of relying on the IP address of the email server so it is ironic to claim that they have anything to do with server reputation while lamenting a lack of sender reputation mechanisms.
Currently, SPF/DKIM are mostly used to prevent fraud, but they also provide the best tool we have to build sender based reputation systems.
> Unfortunately, that means that attempts by those larger providers to combat spam using such reputation will naturally hurt smaller providers.
Indeed. I am on a few mailing lists and many people on them host their own email. I use Gmail so that means visiting my spam inbox once a day to click "not spam" upward of a few dozen times. Day in and day out, same people end up in Googles spam folder. It's bullying and sabotage.
> No one says to themselves "Huh, this is from a Gmail address, it must be legit".
Indeed. I get a shit load of spam from google addresses so reputation is not there.
> We really want to keep track of sender reputation.
This is the hard part but I do think it's something to think about. I should be able to get an email from a known sender every time without $emailprovider making that decision for me. Some sort of attached signature or key that is proof of sender identity so I can route that right into my inbox.
The point of SPF/DKIM/DMARC is to bind emails to domains, so no more spoofing. It is naive to expect authentication alone can reduce spams.
To be fair, SPF saves mail.ru and outlook.com users from five, maybe six spam emails per month coming from my domain, based on DMARC reports. If those numbers scale to include every domain on the internet, that's a huge amount of spam being filtered out very easily and very early.
You'd think spammers would've learned to avoid SPF domains at the very least but they haven't, so despite SPF/DMARC/DKIM failing to get anyone out the spam folder, the technology is still catching spam bots.
While it may or may not reduce spam, it has definitely (based on my personal experience) reduced the amount of spoofed phishing emails and backscatter spam emails to nearly nothing.
In the early-to-mid '10s, before SPF/DKIM/DMARC became the law of the email land, one had to be much, much more careful with phishing emails, checking the wording, the logos, etc, because 9 out of 10 of them appeared to come from the actual domain the email purported to be from. In the past several years (I honestly don't know exactly when the change happened; I don't get a huge amount of phishing emails), it's shifted so that the first thing to check is the sender address. Usually that turns out to be some nonsense string @gmail.com or some long garbled domain.
All of these technologies are basically DOA because of how fickle they are and for lack of support across the board. Most policies are set to not to deny.
DMARC is nice though. It won't stop spam. It won't stop spoofing. But you will know that someone somewhere is spamming people using your domain name. How awesome. :)
I never found the DMARC reports actionable, so I quickly turned them off. What do you do with the information?
Of course, even with hard fail spf and dmarc, I still see some bounces from spam where some server accepted the mail to deliver it elsewhere and the next server denies it, so the first server sends me a bounce.
DMARC reports are for you to be sure that you configured SPF/DKIM correctly, not asking you to do something with the spoofing senders (which you can do absolutely nothing about).
Yeah, so have reports when you start, but once you get things set up correctly, turn off the reports. If you break things later, you should find out quickly when mail is refused, and you can turn on reports again, if you need to.
I observe the same thing. However, that does mean that SPF and DKIM are useless (although DMARC probably is).
It is correct that SPF/DKIM does not really avoid spam, because spammers are not stupid and can read these standards like anyone else. However, before SPF/DKIM, I remember that I got a ton of phishing mails with FROM containing "support@paypal.com" or similar. Then came Bayes spam filtering, and that would move legitimate mail from Paypal to spam, because obviously, the phishing mails are quite similar.
This problem has pretty much vanished, because Paypal clearly denotes which IP addresses are allowed to send mails from that domain via SPF and the client can verify the mail via DKIM. For instance, Spamassassin makes sure that mails with correct DKIM and from paypal.com get a massively reduced spam score so that your Bayes filter will not move it to spam. This is hardcoded for a lot of domains (see *welcomelist_dkim.cf).
Google are bad at SPF and DKIM.
—⁂—
1. I tried responding to a Chromium bug tracker message by email a couple of months ago, and it failed me:
> Unfortunately, your email to create/update an issue was not processed.
> Reason: SPF/DKIM check failed. Please ensure your domain supports SPF (https://support.google.com/a/answer/178723) and DKIM (https://support.google.com/a/answer/174124). If your domain does not support them, please use the Google Issue Tracker UI (https://issuetracker.google.com).
Trouble is, this is simply not true. My SPF and DKIM are fine. This makes me wonder whether the email ingestion system is simply broken for everyone.
—⁂—
2. I got involved in setting up a Google Workspace for someone a few months back, and the entire tool that their own documentation instructs you to use to check things, https://toolbox.googleapps.com/apps/checkmx/, has been laughably broken for years, sometimes not working at all, but mostly producing misleading nonsense results (e.g. claiming domains have no mail server set up when they do).
Then, to make it even more absurd, the feedback link they give you, https://toolbox.googleapps.com/apps/main/feedback?toolname=c..., iframes https://docs.google.com/a/google.com/forms/d/e/1FAIpQLSdnlp8..., but you haven’t been allowed to iframe such documents for I don’t know how long so it doesn’t load, and even if it did, it’s a private form that only Googlers, I suppose, can fill in. And there have been plenty of reports about all of this for years, and it’s still broken.
No they're not.
I run my own email server. Most spam crap cannot pass spf/dkim. Although this post has caused me to sit up and notice that the trendline is moving in the unfortunate direction, where I'd say 3 years ago the ones that pass were about 1/4, today it feels like 40-60% pass. The amount of mail I get that I expect, passes spf/dkim at around 90-95%
I suspect the delta between their any my results are the very restrictive sender rules I have prior to accept. In addition any_address@domain goes to my default mailbox, so I'm also probably selecting for laziness a bit more than most.
I also publish an email address without obfuscation on my site, which is getting very little spam, (near zero) which makes me wonder if most spam has given up on scraping the Internet for emails these days.
It’s far easier to buy the email addresses of known good people by buying dumps of websites that got breached.
Web scraping gets you a lot of fake emails, company sinkholes, and other low reward stuff. Paying $20 for 100k confirmed real emails with names? That’s gold.
Moved my mail over to Proton and they had a very nice process that made it easy to add the required DNS entries and verify that they were correct.
I was dreading this step as I hadn't done it before but turned out to be a breeze thanks to that.
I think the problem isn't necessarily that adding DNS entries is hard (especially compared to the rest of the process of hosting your own email), but that getting a clear overview of what email tools an organization uses is difficult.
You need IT to list all of the reporting tools, customer service to tell you about their support system, marketing to tell you about their mailing list tool of the week, the sales guys to warn you they're using this new AI email enhancer, and somehow get that shady email forwarding service the CEO uses to give up their mail server IP addresses. Then you need to figure out how to get coverage for all of those tools and keep on top of them whenever something changes.
A lot of companies promise to do great things for you if you just enter the email address you'd like to send email from, and a lot of people gloss over the important details because those sound hard and when they tested the tool on their personal email it worked fine so that's probably unnecessary anyway! Managing email for a corporate domain can be like herding cats.
I think pretty much all email providers (and other systems that want to send on your behalf) have this. More or less the same process where they tell you what to add and then a "check my stuff" button to verify. Which is great.
Sounds good. As I said it was my first time, and I'd just glossed over the specs and did not look forward to it (I usually don't enjoy sysadmin work). So, was just pleasantly surprised.
I moved to Fastmail, and they have a nice guide to set up what's needed on DNS:
https://www.fastmail.help/hc/en-us/articles/360060591153-Man...
Naively I thought that one value proposition of SPF, DKIM and DMARC is that reputation shifts from based on IP to be based on domain, once you set these up correctly. So as long as you can maintain a good reputation for your domain and have SPF, DKIM and DMARC correctly set up, then you can host your SMTP server at any IP and your emails will get delivered.
I wonder why it doesn't work this way.
IMHO, their main advantage is that third parties can’t send email which appears to originate from my domain.
I configure my domain to use SPF, so now spammers can’t sign it properly.
However, the fact that an email passes SPF verification only ensures that it was authorised by the domain owner. It doesn’t say anything about whether the domain owner is a spammer.
It does work like that except nobody actually knows Google or Microsoft's algorithms to allow or deny mail delivery. It's the whole SEO thing all over again.
It does work that way, but IP reputation is a thing as well so you need to keep that in mind. IPs need to be "seasoned" and "trusted" as well as domains.
This is how email-as-infra works, you're sending from a shared pool of their ips and they sign your emails with DKIM and you'll have SPF set up as well on your own.
Cause IP is a finite resource (even IPv6 where the granularity is more like /48) while domains are infinite.
I've been running my own spam filter for many years now based on this super-simple heuristic: My filter looks at my outgoing mail, and any mail received from an address I've sent mail to, or with a subject that has appeared in my outgoing mail (possibly with a "re:" prefix) is marked as non-spam. Everything else goes in spam, and any spam message from an address I've never received mail from before is marked as unread. I get hundreds of spams per day, but only about a dozen from new addresses. It takes me about ten seconds to scan them for non-spam cold calls, which are extremely rare. The other source of false positives is things like subscription confirmations, but because I know to expect those, they are always at the top of the spam folder.
I put this initial system in place expecting to have to augment it later with a more traditional content-based filter, but this simple heuristic works so well I've never felt the need to implement that additional step.
Someone posted on X advice that really helped me clean up my inbox
Add a filter looking for the word "Unsubscribe" and automatically put them in "Promotional" category or something similar. Also apply the filter to existing emails, and let it run for a minute.
Try it now! And comment if it reduced your inbox to like 2% of what it was :)
I've commented here before that it is obvious to me that gmail makes no effort to combat spam anymore given that unsubscribe links are legally required and generally present for spam in the US and are an obvious heuristic that aren't used. I would expect basically any trained filter to pick up on it, so my assumption is that they actually intentionally have rules to allow spam.
I get emails that literally say "This is an email advertisement". These are presumably being blasted out to tons of mailboxes. How does a model not notice this?
An advertisement is only spam if it's unsolicited. If you forget to uncheck the box "yes send me promotional offers and deals" when signing up it's not spam according to that definition.
If you're making people opt-out and setting them up to "forget" to do so, then you are spamming them, but even under that definition, I'd estimate that over 99% of what I'm calling spam still qualifies. A large amount of it is from businesses I've never interacted with, so obviously unsolicited.
Isnt it against GDPR and they could get hit with large fines in Europe for every recipient?
A mail service run by an advertising company fails to filter out advertising emails? I'm shocked. Shocked!
I tried that a long time ago and the problem with it was that it produced a lot of false positives for me because I subscribe to a lot of Google Groups.
Can you make a negative condition also, X but not Y?
Of course. But the problem is that the more complicated you make your filtering logic, the harder it becomes to maintain. I was constantly discovering new exceptions to my ever-more-complicated rules, which is why I eventually gave up on that whole approach.
I'm using something very similar, except incoming messages from never-seen-before senders are greylisted instead:
https://en.wikipedia.org/wiki/Greylisting_(email)
95% of spammers never retry.
The problem with greylisting is that it delays subscription confirmation emails when you sign up for a new service. I found that to be more trouble than it was worth. YMMV.
For a greylisting that sends 451 before DATA, that is indeed a known problem.
My server sends 451 after DATA, and keeps a copy of greylisted message, as marked-as-read entry in separate folder. Those are deleted after few hours, or moved out after a successful delivery retry.
That's a good idea. I was using an off-the-shelf greylister that didn't work that way, but I might implement that strategy now that I'm doing everything myself.
Providers should really stop using spam folder and refuse email at the session lvl, that alone would fix the false positive issue. I had a rant [1] about it a while ago.
My biggest problem with SPF, DKIM, DMARC is when you go to test this crap there's really only commercial apps. So people who are setting up things for a non-profit or a personal project are either forced to pay after doing 3 or 4 test emails or you wait like 24 hours or some crap.
And all that just for the privilege of being able to send email to some gmail accounts. Trying to get email to properly encrypt is pulling teeth and yet I still get hundreds of thousands of spam a month on my gmail account.
Any time I have to set up an email server on a new system I just kind of die a little.
I built a free DMARC/DKIM/SPF checker: https://dmarcchecker.app/. No usage limits, no ads—just a small footer link to one of my other projects. Made it for the exact reason you mentioned.
This is awesome. Thanks! Sensible, easy to use, easy to understand the results.
I've been trying the google one, for example, and it doesn't even work. "request timed out." Fishy, because yours works great.
much appreciated!
Yopmail does verify and show these results for free when receiving a mail, this not even being their core feature
That’s good to know. I’ve not seen it mentioned anywhere. The tools I’ve used in the past give me 3 or 4 test emails before they require me to sign up.
You can always first send it to yourself and read the Authentication-Results header. Though even Gmail displays SPF/DKIM/DMARC status when you view the raw source of an email.
Which works great unless you're trying to debug gmail refusing your emails because of some issue or another.
> Surely everyone (and by everyone I mean Google) is rejecting their mail? How do they not realize this?
Not sending email to google helps.
DKIM is not meant to block spam, it's meant to authenticate that the sender had access to the private key for the public key exposed on the domain that it was sent from, implying that the sender has sufficient permissions to send from the domain.
It should not be used to imply anything else, none of these have anything to do with spam, that's reputation (and yes, having DKIM set-up boosts your reputation but it is not sufficient) and should be "built" up by the domains sending the emails.
It is incredible how the vast majority of the problems with the internet have their root cause in the activities of marketers.
Spammers are worse at something no one tried to use AFAIK: require a fee to deliver email.
You may be surprised to learn that spammers, being criminals, have no issue with stealing money from others to spend on email delivery fees.
Edit: Proof-of-work "email postage" schemes are similarly doomed - The botnet that zero-day'd your mail servers does not care how much electricity they use.
Criminals yes, criminal spammers no. If they could make money with other crime there would be no point in wasting their time with sending spam.
The point to sending the spam is to enable further crime. Do you really think they don't stand to profit from what they promote? Spam is a business like any other, they aren't going to magically disappear just because their advertising costs suddenly become non-zero. Just like how they found ways of shifting the costs of hosting mail servers onto others, they will find ways of shifting the costs of any "email postage" scheme onto others.
This is very basic mathematics. If they have to do other crime to get their hands on the cash it would cost for them to spam, then why spam at all instead of only focusing on the other, more profitable crime?
Spam by definition is mass messaging. If the price per message is higher than the expected return per message, it becomes pointless.
Spam is advertising for their other more profitable criminal enterprises. It's a means to an end, not an end itself. If the goal were simply to send messages, there would be no content.
And the efficiency of spam is so low, that with a high enough cost per message, it would not be a net gain for the spammers. If you pay a million dollars to send a million messages, but only gain $100 000 from the subsequent scams, then it's a bad venture.
No, they will happily pay money to spam you.
IP reputation, proof-of-work, and various fee-for-receipt schemes are trying to solve spam in the same way: by charging low-volume users a trifle to send messages at a "normal" rate, which adds up to become more expensive when you start bulk mailing. The problem is that these schemes have to assume that there is a single market-clearing price[0] for sending messages that is both low enough to not inconvenience legitimate users and high enough to make bulk messaging uneconomic.
Such a concept goes against how communications networks actually function. The amount of communications resources the average person uses is so low that it's not worth billing for them. Sending any sort of data isn't free, but it's "cheap as free[1]". Any communications technology that bets against this will fail in the marketplace. People are not going to go back to, say, buying (virtual) stamps at 73 cents[2] a piece to send mail with. Hell, the $10/GB I pay with Google Fi is already enough to make me cringe every time I actively use my data plan.
On the other side, charging a fee for misbehavior legitimizes that misbehavior; if the fee is less than the value of the misbehavior then you are just imposing a cost of business. Spammers need to send lots of mail because the rate at which people fall for your scam is comically low. But when you do hook a sucker in, they yield a huge return.
So what we have here is that legitimate users would balk at per-message rates that wouldn't even be close to what would make spammers flinch. Which is, again, the same problem that SPF/DKIM/DMARC have. People whose job it is to send garbage e-mail for a living have a far higher tolerance for Internet bureaucracy bullshit[3] than people who use e-mail to get their real work done or to talk to friends and family.
[0] Getting your IP banned for spamming or having to burn energy brute-forcing a hash can be considered a price.
[1] Buy all our playsets and toys!
[2] Current price of USPS letter postage
[3] In the Graeberian sense
Do you know of any data on this? It seems like the kind of thing that could be studied and measured. I'm inclined to believe the opposite about the viability of e-stamps, but I will readily admit I have no data to back that opinion up.
Sorry, I didn't clarify something.
Require a fee to deliver email. If the email is deemed legitimate by the recipient then refund the fee.
> No, they will happily pay money to spam you
It's a question of price. They can't spam at scale at high enough price.
Things have gone in the direction of being favorable for spammers.
* WHOIS is effectively destroyed.
* Companies like Cloudflare actively protect known spammers & scammers, and have made abuse reporting time consuming and error prone.
* Consolidation of most email to several large players means filtering them causes problems.
* Large delivery companies such as Sendgrid, Salesforce and Google basically do nothing about reported abuse.
Yes, most spammers these days that set up their own domains have tools to make sure DKIM, SPF and DMARC are all good, but consider that we can't know anything about these spammers: TLS certificates no longer contain contact information, large providers don't provide useful WHOIS, don't forward abuse complaints, have no clue what an SOA record is, and so on.
The way things should work is that we get a spam, we see the network from which the spam came through WHOIS, we forward the spam to their abuse address or the address they list in WHOIS, and we're done.
The way things work is that they don't have working information in WHOIS, they ignore the abuse complaints, they act like they don't know what to do with it, or they reply with a form email saying to go and use a web page where it takes time and work to paste in each part of a spam.
I blame the large companies who do this. Make reporting abuse difficult and you'll get much less reported abuse.
I know very little about these protocols, except for having to deal with them a bit on those few sad occasions I need to get a server to send email. From those experiences I had a strong sense that Google pushes out all these complicated and difficult procedures on everyone just as a means of discouraging people from using email servers in the first place...."just use google, we control the whole thing anyway".
You may have success checking for common tracking and advertising elements in a mail. Good chance it's spam if there's not 100 trackers. Frustrating.
I used to run my own e-mail server for my personal address. In an attempt to reduce spam I configured Postfix to reject all inbound messages that weren't DKIM signed. The only time I ever had an issue was when somebody from the multinational publicly-traded company that I worked for tried to send a message to my personal inbox. They ran Exchange in the datacenter at the time (this would have been ~2017) and hadn't enabled DKIM signing. I had a friendly conversation with the sysadmin responsible for it and they had it enabled by the end of the week.
I suppose the moral of the story is that it's possible to do billions of dollars in business a year without having textbook-perfect mail infrastructure. Hell, I ran a mail server with bad MX records, a missing PTR record, and a mismatched HELO header and the world kept spinning (when I was a literal child with nobody to tell me better - I've since learned the error of my ways).
I have a much bigger issue with "legitimate" spam these days. Every service makes you give an email address, and they all force you to check a box allowing them to email you whatever they want. Then if they even have an "opt out" link, it takes you to a list of 500 different types of notifications and forces you to opt out of each one individually.
Usually I will just disable the iCloud hide-my-email I used for a site, but sometimes there are legitimate emails mixed in with the stream of crap. I opted out of marketing emails from my credit card company, and now they instead send me emails asking me to re-evaluate my email preferences...
It would be nice to see more done to fix this, but I guess it doesn't make anyone money. I guess I'll just have to use AI to filter signal from noise.
Related: there are known problems with DKIM, and there's a DKIM2 effort:
* https://datatracker.ietf.org/doc/draft-gondwana-dkim2-motiva...
* https://datatracker.ietf.org/wg/dkim/about/
* https://blog.redsift.com/email/dkim/first-look-at-dkim2-the-...
The recently-held IETF 122 had a session on it:
I'd also like to see an update to DMARC so you can require both SPF and DKIM in your policy, instead of just one out of the two.
Terrible idea, SPF is very hostile to (legitimate) forwarding. In general SPF should actually die.
If you have trusted forwarders, you just add them to the SPF policy (which can be recursive, though there is a pretty low limit on how many records can be looked up). I've not had an issue with this, personally. However, assuming DKIM can be tightened up as proposed above, I'm not sure SPF would be necessary anymore.
There are quite a few problems with that. Biggest issue is that it would require the domain owner's explicit cooperation with each forwarder. It would also allow more than just forwarding existing letters. Real-life shows that SPF really doesn't work with forwarders and it probably never will.
Of course they are. It's how they make their money. The big email providers generally don't make their money from selling email services, it's a thing they offer as an in to sell the services that do generate profits. On the other hand, successfully sending an email that bypasses both technical and human barriers is spammer's business.
Rights of passage in sw-dev and IT is to propose a solution or re-implement something like many before you. Mine on email SPAM: https://paulhammant.com/blog/did-you-send-this
This is missing the point.
To be clear, I'm not necessarily a fan of DMARC, particularly how it was introduced. But it is very obvious that spammers will eventually do everything to not be flagged as spammers.
What DMARC gives you is that it makes it less likely that your phishing mail will come from contact@yourbank.com. It will rather come from contact@y0urbank.com or some other domain.
How much of an improvement that is and how many people will notice is certainly debatable. But that's what DMARC can give you. Nothing more, nothing less.
Is there actually an "domain reputation as a service" provider, which controls a couple thousand gmail addresses, sends itself the emails and manually unmarks them as spam? Asking for a friend..........
I use NameCheap for domain registration, web hosting, and email.
Unless I pay extra for Premium DNS, my DKIM is set wrong because their Web Hosting DNS does not oet me set it correctly.
> Rejecting mail based solely on authentication failures of those deeply flawed authentication methods does more harm than good.
How is DKIM "deeply flawed"?
Just like spammers, scammers and other forms of predation will be better at crypto than everyone else, better at AI than everyone else, etc.
Mission accomplished.
Nobody wants to solve the problem of spam, this is just theater to put another tax on the smaller players in the Internet industry business.
Yeah, SPF, DKIM and DMARC are incomprehensible. The only people with the time, motivation and expertise to understand and apply them, are professional spammers. Most of the legitimate email I get, fail one or more of these tests.
If I recall, SPF limits the number of domains you can enumerate in your DNS records.
honestly my take away is the opposite
we by far don't enforce them strict enough, because if we where people would make sure to get it right
it's all a question about effort/turnout
if you make it so that most times you mails still somehow end up with the user even if you mess them up there is much less insensitive for companies to fix their mail or force their provider to fix it
so IMHO if all of SPF,DKIM and DMARC are not correct setup mails should just be directly discarded and not even delivered to spam
while being more flexible was reasonable when this tech was new, that was 10 years ago. If being flexible mean that you also will sometime deliver outright cyber attacks like spear phishing and similar to your user and everyone had 10 years to fix their systems then there is really no reason to still be flexible.
Also "scammers get it right so it's useless" is such a huge red hearing argument, yes they do get it right _for their domains_. It sill makes it harder (and if strictly enforced impossible, except if you give them permissions to do that*) for them to impersonate your domains.
And yes that doesn't fix scammers from using their own domains, but it also was never intended to do so. Doing so is a very different problem one which probably needs some form of reputation system which isn't something you can just solve technically as it touches on a lot of subtle social political issues. Also given that all of the huge mail vendors have insensitive to use their "intern proprietary obscure" reputation system I don't expect there to be a technical solution provided/adopted tbh.
(and yes SPF/DKIM/DMARC are all tech wise quite "meh", but we are kinda stuck with them, through that never was the issue IMHO, the issue is missing insensitive to bring the adoption up and missing insensitive for large mail providers to enforce it strictly)
EDIT: PS: In one point they are fully right so, that is, with how things are you can't give SPF/DKIM/DMARC a large weight for calculating reputation. Also they where always only meant to tell you if someone can't be trusted, but never if someone can be trusted.
SPF, DKIM, DMARC, IP reputation, whitelists, blacklists, graylists, spam filters, ...
I'm sure regular airline passengers trip the metal detectors more often than terrorists, doesn't mean we should get rid of the metal detectors.
A better analogy would be a passport. It doesn’t stop all terrorists from boarding a plane at lest it stops already know to authorities ones (unless they have a passport on someone else’s name which is not easy).
Or perhaps, why do you lock the door to your house? A few solid kicks will open most doors, the locks can be picked, someone can smash windows and enter, and many modern homes can be entered by ripping the wall open with a crowbar and axe.
It's to stop midrange threats.
Doors and locks are purely social construct. For majority of people it's much easier to justify stealing from a porch compared to breaking in.
No more, no less.
For spammers on other hand it's just a business, there will be no reprecussions like ever and we know quite a few big and legitemate companies who started their path with marketing spam sometimes using leaked email databases.
The way you're using "justify" here, makes it seem as if you think people feel it's morally legit to steal, if it's on a porch for... reasons?! From a moral perspective, theft is theft. There's no way someone can sanely claim they thought it was a free thing, because it wasn't locked away.
Doors and locks are there to make theft harder, more overt, loud, etc, and by no means validate when it's legit to be a vile thief.
Likewise, all spam is spam. The use of tools to make it more difficult for spammers to be spammers, is the same as having doors and locks. It makes it more difficult.
edit: What I said was, you clained they tried to justify it. So no worries, I was not implicating you.
I not trying to justify it, but if you actually look and check research about people who been caught stealing there is huge difference for them between stealing TV that dropped from a truck vs stealing from a porch vs stealing from inside the house even though it's the same TV.
Theft is theft, but for monkey brains there is huge difference between stealing someone wallet from a pocket vs picking dropped wallet and not returning it. So my point is that doors and locks work not because it's good technical measures, but due to how average Joe percieve social construct about them.
And for grey area activities online there is no such social construct because there is no percieved connection between bunch of email addresses and real people. Also in some countries it's totally legal to send you tons of physical mail spam.
I forgot the name of this fallacy, I read about it in Nassim Taleb’s Antifragile a long time ago, but basically being wrong at spam won’t cause a lot of damage while being wrong once about a terrorist may cause thousands of deaths