I started writing a guide to IMAP back when I was working on an email client: https://github.com/rmccue/griffin/tree/master/docs/imap
Pulling large amounts of data for things like threading can be difficult on certain servers; my preferred approach ended up being to pull every ID and thread ID to maintain an in-memory tree. (This was, iirc, partially because Gmail’s implementation was slightly crippled with relation to threading.)
(I never finished the guide because I stopped on the project, alas - if IMAP were easier to work with, I might have finished it! And sadly, no JMAP support on Gmail, and the gateway was broken.)
> I started writing a guide to IMAP back when I was working on an email client
I would be very interested in hearing from developers about how they write email clients that need to work with all the servers out there, with the varying levels of IMAP4 (original, rev1, rev2, combinations of at least a dozen extensions) and various levels of buggy behaviour.
I'm assuming a client developer would implement various "profiles". With an "advanced" profile for servers that implement things like CONDSTORE/QRESYNC, and a "very basic" profile that only do the absolute minimum. And probably a profile or two in between. When you encounter unexpected behaviour (eg bad syntax in protocol), you would downgrade the profile for that server for a while (it may get fixed)? If it works like this, I'm curious to the profiles developer choose. If it's not like this, I wonder how client developers work around compatibility issues (can't just keep reconnecting and trying the same if it results in an error!).
As someone currently working on an IMAP client, my strategy has been to start by implementing the base IMAPrev2 specification and checking that it still works with a rev1 server. RFC 9051 has guidance on how to handle servers that use older IMAP protocol versions and is largely backwards compatible.
The goal is to have the client work without any extensions. E.g., if the server doesn’t have the UNSELECT capability it falls back to selecting a nonexistent mailbox (which has the same effect). Obviously there are some very useful extensions that the client needs to support, but those can be viewed as optimizations to reduce network traffic or add extra features if the server supports them.
> Gmail’s implementation was slightly crippled
Gmail is not "crippled". A tiny but vocal community of old nerds have a petrified mental model of email that they associate with unix IMAP software from the 1990's, but those concepts do not appear in the IMAP standards anywhere.
That is an immature view on how real products and real standards work. The standard document may say one thing, but what people do in the real world is the real standard.
For context: I spent 11 years at Intel managing pre-silicon and post-silicon processor validation. No processor that does only and exactly what the Programmers Reference Manual says, and takes the phrase "undefined behavior" seriously, will be successful. Google would do well to adjust their philosophy.
If history has taught us anything, it's that Google is happy to willfully ignore, rewrite, and use their market dominance to snuff out any existing standards if they see a way to seize control and make money off something.
Absolutely, but in the context of IMAP, it's already a mess of special cases. Gmail having a weirdness, an extension or two changes basically nothing. Not to mention the fact that what they offer over the web is more flexible than what IMAP usually allows, it simply does not and can not map directly to IMAP.
It's really dumb that one message has* to exist in only one location for example, labels are so much better.
It's been an odd running theme for me today that I've misinterpreted posts. Up until your final sentence, I thought that the thesis of your post was:
The standard document may say one thing, but what people do in the real world is the real standard. If your software has issues with the world's most popular IMAP server, you need to adjust your software to be compliant with the standard.
I'm personally more sympathetic to your actual conclusion, but it's odd how often a single argument can be used to support two conflicting beliefs.
Yeah, agreed! I titled my guide "Practical IMAP" for reason (and almost called it "IMAP As She Is Spoke"). The standards are useful to a point, but actually to jeffbee's point the internet has evolved a lot since then, and how to actually work with modern email is a pretty underdocumented - including by Google themselves on the Gmail-specific parts.
If an x86 implementation was imperfectly compatible with Intel CPUs, nobody would buy it. Gmail, on the other hand, is a massive market success. It is those who shout that IMAP must be exactly and only whatever mutt+uw was doing in 1997 who are on the wrong side of history.
It's a free email account, it is not at all clear that "weird IMAP" is core to that success?
Certainly I moved away to Fastmail, which has better IMAP support (but mostly because Google having full control of my email address was becoming too big of a risk and Google Apps is expensive for your own domain).
My memory is hazy on it, _but_ from memory, there were some issues around IDs and persistence which don't occur on other servers, as well as the auth being a bit funky. (Gmail also uses its own extension to the protocol for IDs instead of using eg OBJECTID.)
To be clear, I have no opinion on IMAP or what things _should_ do; I certainly was not using IMAP software in the 1990s :) However, trying to implement a client that works across providers does mean trying to operate per the standards, which was a struggle and why I started documenting it :)
> Gmail also uses its own extension to the protocol for IDs instead of using eg OBJECTID.
When was OBJECTID introduced and when did gmail roll their own implementation, do you happen to know?
Which concepts are you referring to?
RFC 9051 says that it was "developed for" RFC 822 and that one should now refer to RFC 5322 instead. And RFC 5322 does discuss threading, which is the what the comment you're replying to was talking about, so that concept does appear (by reference) in the IMAP standards.
Are there other parts of the community that implement IMAP using the same "mental model of email" that Gmail is using? Or are you saying that anyone using IMAP with anything but Gmail is an old nerd and a tiny minority?