SPF/DKIM is really about mail server reputation. So it mostly benefits larger servers like the ones run by Google, Microsoft and Yahoo. Unfortunately, that means that attempts by those larger providers to combat spam using such reputation will naturally hurt smaller providers. So the actual effects of SPF/DKIM are on the whole negative.
The root problem is that we don't actually need to keep track of email server reputation. No one says to themselves "Huh, this is from a Gmail address, it must be legit". We really want to keep track of sender reputation. We need to be able to treat anonymous email differently than email from people we actually know. That implies that we have some work to do on the problem of identity. As it is, there is not even a way for a known email sender to securely introduce an unknown email sender. You know, the way that regular human people normally are able to transfer identities from one to the other.
>SPF/DKIM is really about mail server reputation. So it mostly benefits larger servers like the ones run by Google, Microsoft and Yahoo. Unfortunately, that means that attempts by those larger providers to combat span using such reputation will naturally hurt smaller providers. So the actual effects of SPF/DKIM are on the whole negative.
That paragraph is incorrect. SPF/DKIM is not about reputation. The main purpose is preventing domain impersonation from unauthorized senders. E.g. mail servers will reject fake emails from "[email protected]" because you don't control any email servers that's whitelisted in microsoft.com DNS TXT records.
E.g. I was able to register a brand new .com address and then successfully send to gmail and MS Outlook accounts within minutes because I had proper SPF/DKIM in the DNS records for that new domain. That new domain had zero reputation and yet Gmail accepted it because SPF/DKIM was configured correctly -- and -- the underlying ip address of the server it came from had a good reputation.
If SPF/DKIM was truly about "reputation", it would mean I'd have to wait days or months for reputation history to build up before Gmail accepted it.
preventing impersonation is an important part on correctly attributing reputation to source domains.
Exactly. The people bashing SPF & DKIM don't seem to understand their intended purpose.
And it will mysteriously _stop_ being able to send mail to Google despite you doing everything right, because of whatever nonsense they use to determine reputation.
I am curious as to your experience with this.
Over the years, I have administered a few dozen small to medium domains (depending on the domain 10s to 10,000s emails per month) and the only thing that has ever affected delivery is the reputation of the sending IP address of the mail server (and ensuring DKIM/SPF alignment in more recent years).
I don't think this is correct? SPF and DKIM are about ensuring that the server actually is who it says it is, not about its reputation. In other words, when you receive an email that claims to be from Gmail, SPF and DKIM help you ensure that's where the letter actually came from, not from a server just pretending to be one of Gmail's servers.
SPF more like whether the email came from a server that's authorized to send emails on behalf of a particular domain.
> Unfortunately, that means that attempts by those larger providers to combat spam using such reputation will naturally hurt smaller providers
Tin-foil hat time, but I've always thought there was nothing unintentional or "unfortunate" (from Google's perspective) about this.
While there's some truth to that, neither SPF nor DKIM nor DMARC have anything at all to do with reputation.
They serve to authenticate that the sender is somewhat connected to the person who controls that sending domain.
> That implies that we have some work to do on the problem of identity. As it is, there is not even a way for a known email sender to securely introduce an unknown email sender.
There is: gpg/pgp signature, but many people find it complicated, primarily because they are reluctant to read the documentation. And it’s popular to criticize it, especially here on HN, in favor of various half-baked alternatives.
I think everyone can agree that any technology that "isn't complicated if you read the documentation" is by definition complicated. I don't need to read the documentation for Gmail to use Gmail successfully.
Could I, as a trained programmer, use PGP and GPG? I'm sure I could if I spent some time reading about it. Could my 90 year old grandmother, who is otherwise quite comfortable with email and whatsapp? No, not to any meaningful extent.
There are times you need complexity enough to be worth training costs. There is one universal word "nanana", and maybe babies cry (it seems many babies have unique cries for different needs: I suspect that is training between babies and their parents - anyone done research on this?). All other language is because you spend years in training. If you can read this or write a response that implies training.
The important point from the above is it was worth the effort to learn. The only person I know who is a strong advocate of PGP was a missionary to Romania before the iron curtain fell - he had strong reason to hide what he was saying from government level actors and even today still is willing for extra effort to protect himself. For most of us though our threat profile isn't (or doesn't seem to be) that high and so learning how to use the tool isn't worth it.
I absolutely agree that PGP/GPG have important use-cases for which they are the state of the art and well worth learning. This doesn't mean that they are not complicated technologies though.
I highly disagree with this.
I just left a couple of comments regarding the use of "strtok". Its use is straightforward, just RTFM. Those were the golden days when people were less reluctant to read documentation. You could not even install Linux back then without an installation guide of some sort. You still need it for Gentoo, perhaps even Arch or Void. Are they wrong? No, just different target audience. If you do not want to become a "power user", that is fine.
My grandma can barely handle the TV controller. So what? I am really against dumbing things down, called "ease-of-access" or whatever they call it these days.
I agree on that, however, that GPG / PGP signatures should be more visible and whatnot, just add some visual feedback (verified? legit?, etc.), and some e-mail service providers actually do this.
> Are they wrong? No, just different target audience. If you do not want to become a "power user", that is fine.
Complicated doesn't mean bad. I'm not claiming that PGP or GPG are bad technologies because they are complicated to use.
> My grandma can barely handle the TV controller. So what? I am really against dumbing things down, called "ease-of-access" or whatever they call it these days.
The "so what" is simple: PGP is not the right anti-spam solution for your grandma, or mine, or any users like them. This is the context of this conversation: is PGP a good-enough answer for how to establish identity for email in the interest of anti-spam and anti-scam efforts? And the answer is a clear and resounding no, not for the vast majority of users of email.
This, again, doesn't mean that PGP/GPG are bad technologies - they are very good for certain use cases and certain users.
So what is a good-enough answer for my grandma? :P
There's no great answer, unfortunately. Gmail and other off-the-shelf email providers handle much of the spam and some of the scam prevention for you, but you still need to exercise caution on your own.
> many people find it complicated
That's what kills a lot of these "perfect" implementations.
HN members tend to be nerds, and we don't really have an issue with setting stuff up (many HN IDs, for instance, have Keybase auths).
Most non-HN types have no patience for that stuff. Security needs to be made accessible and easy-to-use, before the vast majority of folks will implement it. That's the single biggest conundrum, IMNSHO.
We really need a "Trust on First Use"(TOFU) system for messaging, that can be verifified, or pre-trusted offline, face to face. It'd be awfully nice for your bank to give you some thing that you can later verify that any communication from them (web site, online banking, text message, email, etc) are legit and verified.
Or if we can't trust users to handle TOFU, then some token/unique address/whatever that we can exchange face to face to enable trusted communication.
> We need to be able to treat anonymous email differently than email from people we actually know.
The simplest solution to that would be an "only show me emails from people in my address book" filter. That would mostly echo how we treat user trust on all other platforms. Genuinely surprised this doesn't exist in most email clients (or does it and I have just overlooked it so far?)
Of course that's only a partial solution and wouldn't work for accounts where you expect unsolicited mails from people you don't know. I'd see it more as a "low-hanging fruit" solution. You could also expand the heuristic, e.g. also consider previous conversations, mailing lists, etc.
(Interestingly, the "introduce a friend" functionality would come for free: You can already send contact details as a VCard in an attachment. When receiving such a mail, some email clients will show a button to quickly add the contact to the address book.)
> only a partial solution and wouldn't work for accounts where you expect unsolicited mails from people you don't know.
I actually think this would work fine. Imagine a quarantine inbox for new emailers that the user must scan and approve/block. This is exactly what hey has implemented.
> The root problem is that we don't actually need to keep track of email server reputation.
We actually do. Is the server allowing anyone to sign up so that it's sending 99% spam, or does it have a lot of anti-spam measures so sign-ups can't be automated and it blocks accounts as soon as it detects them sending spam?
> As it is, there is not even a way for a known email sender to securely introduce an unknown email sender. You know, the way that regular human people normally are able to transfer identities from one to the other.
That's exactly what PGP's web of trust model is for. Someone you know, and trust, can sign and send you a public key of someone that they trust.
This new key will be automatically trusted in your trust store because it's signed by someone you already trust, although in a lesser trust level to account for the degree of separation. If you later verify that key out of band you can upgrade it to a higher trust level.
SPF/DKIM, as well as TLS etc., is just stupid shit we do because we're too lazy and/or incompetent to make web of trust work for us. It's not a technology problem, it's a human problem.
> SPF/DKIM, as well as TLS etc., is just stupid shit we do because we're too lazy and/or incompetent to make web of trust work for us.
Having key signing parties for the entire world wide web does not seem scalable to me.
If you want to have complete trust in every key you hold then you need to validate it personally. This is exactly the same scaling factor as Signal or WhatsApp, for example.
Web of trust scales better than that, though. It gives you confidence in keys you haven't seen yet because they are signed by other keys that you do trust. The key signing parties strengthen the web of trust, making it more likely a potential correspondent will receive a key signed by someone they trust and therefore potentially not needing to verify it personally.
It all depends how much confidence you want to have for each key. At the end of the day there is no substitute for verifying each key personally if you want to be completely sure. PGP give you the option to hold keys with a lower level of confidence for e.g. less sensitive communications.
Well, yeah, we should use preexisting standards and OpenPGP would be perfectly fine here and is probably the best choice. That is a wheel we do not need to reinvent. But the actual system used to do the signatures and keep track of the reputation is the last thing we should be thinking about at this point. We should instead concentrate on how to create a system that the majority of people can use and understand. We should be concentrating on standardizing concepts...
Right, that is my point. I feel like there is a fundamental lack of understanding in the vast majority of the population about trust. We haven't helped by telling people "you can trust the little green padlock". Nobody asks "why should I trust it?". That is the problem. It really doesn't matter what technology we provide, so far none of it is really used by regular people to establish trust.
The other option, of course, is to design a trustless system, like BitCoin, but that has its own problems.
It's weird so see such a factually incorrect comment so high up on HN.
SPF/DKIM is literally how you establish sender identity instead of relying on the IP address of the email server so it is ironic to claim that they have anything to do with server reputation while lamenting a lack of sender reputation mechanisms.
Currently, SPF/DKIM are mostly used to prevent fraud, but they also provide the best tool we have to build sender based reputation systems.
> Unfortunately, that means that attempts by those larger providers to combat spam using such reputation will naturally hurt smaller providers.
Indeed. I am on a few mailing lists and many people on them host their own email. I use Gmail so that means visiting my spam inbox once a day to click "not spam" upward of a few dozen times. Day in and day out, same people end up in Googles spam folder. It's bullying and sabotage.
> No one says to themselves "Huh, this is from a Gmail address, it must be legit".
Indeed. I get a shit load of spam from google addresses so reputation is not there.
> We really want to keep track of sender reputation.
This is the hard part but I do think it's something to think about. I should be able to get an email from a known sender every time without $emailprovider making that decision for me. Some sort of attached signature or key that is proof of sender identity so I can route that right into my inbox.