What's in a name – mailbox names via IMAP

This is a technical blog post about standards, complexity and even a little bit about fashion and how it changes over time.

What you need to know:

• In any program other than FastMail’s own app (i.e. IMAP clients), Inbox may change case to INBOX
• There are no visible changes in the FastMail App or web interface
• For users of our old classic interface, Inbox will become INBOX (just a cosmetic change)

It’s always a hard choice to enforce change on people, but this upgrade is important to make things less complex and more robust for everybody in the future. For the first time, IMAP users will be able to create folder hierarchies that exactly match the FastMail web interface.

Our testing with popular clients and other major providers shows that these changes match the most common way of doing folder listings, which provides best compatibility and a good IMAP client experience for our users.

If you want to understand the gory detail of why we need to do this, read on.

Standards

The IMAP protocol is a whole lot of different specifications, but the key specification that defines the base protocol is RFC 3501, written by Mark Crispin and released in 2003. It’s a revision of the earlier RFC 2060 from 1996.

There were 114 changes/clarifications from RFC 2060 to RFC 3501, making it one of the most edited of all RFCs.

Like many standards created by committees, IMAP is a compromise that tries to be everything to everybody. Some people wanted to address email the way that they addressed newsgroups - alt.binaries.pictures, comp.lang.perl.misc, etc. Some people wanted to address it they way they addressed files on disk - ~/mail/archives/inbox-1995.

So IMAP supports arbitrary hierarchy separators. In practice only / and . appear to be common, though I’m sure there are still systems out there using \ which brings its own special hell in C-like programming languages which use \ as an escape character (as anybody who’s ever programmed for Windows will know).

Some operating systems use case significant file systems, others don’t. So on some systems, README.TXT and Readme.txt are the same file, but on others they aren’t. Of course IMAP chose to leave case sensitivity up to the implementation… sort of.

There’s one thing that IMAP didn’t leave up to the implementation. The special folder INBOX must always exist, and must not be case sensitive… sort of:

5.1. Mailbox Naming

The case-insensitive mailbox name INBOX is a special name reserved to
mean "the primary mailbox for this user on this server".  The
interpretation of all other names is implementation-dependent.

In particular, this specification takes no position on case
sensitivity in non-INBOX mailbox names.  Some server implementations
are fully case-sensitive; others preserve case of a newly-created
name but otherwise are case-insensitive; and yet others coerce names
to a particular case.

6.3.8. LIST Command

The special name INBOX is included in the output from LIST, if
INBOX is supported by this server for this user and if the
uppercase string "INBOX" matches the interpreted reference and
mailbox name arguments with wildcards as described above.  The
criteria for omitting INBOX is whether SELECT INBOX will return
failure; it is not relevant whether the user's real INBOX resides
on this or some other server.

So INBOX is case insignificant for SELECT, but it only has to show up in LIST output if the upper case version matches.

INBOX is even special when renaming a folder:

6.3.5. RENAME Command

... If the name has inferior hierarchical names, then the inferior
hierarchical names MUST also be renamed ...

Renaming INBOX is permitted, and has special behavior.  It moves
all messages in INBOX to a new mailbox with the given name,
leaving INBOX empty.  If the server implementation supports
inferior hierarchical names of INBOX, these are unaffected by a
rename of INBOX.

All other folders MUST rename their hierarchical inferior names (aka child mailboxes), but INBOX explicitly does the opposite.

Fashions on the field

When FastMail was started in 1999, RFC 3501 didn’t even exist yet, IMAP was still RFC 2060, and the netnews naming convention (dots as the separator) was all the rage. Cyrus IMAPd didn’t support the “altnamespace” yet , so folder names were all hierarchical inferiors of INBOX. This layout made sense at the time, allowing the user’s email to be kept neatly separate from the shared email “bulletin boards” of Carnegie Mellon University’s system.

We also didn’t have support for, or certainly didn’t choose to use, / as a hierarchy separator. This means that . hasn’t been available to use in usernames or folder names at FastMail - and we’ve used hacks like allowing _ in usernames and ‘^’ in folders to mean . which has caused many bugs over the years. Even when we switch to /, those issues won’t entirely go away, because subdomain addressing and web hosting still need a single name, and . is still the domain name system’s separator.

Anyway, the standard mail.messagingengine.com:993 IMAP server presents mailboxes as:

INBOX
INBOX.Archive
INBOX.Drafts
INBOX.Sent
INBOX.Trash

But it looks kind of ugly on systems where users only have their own email, and they want everything as top level folders:

INBOX
Archive
Drafts
Sent
Trash

Smarter clients recognise that INBOX is the top level namespace (there’s a facility in the protocol to discover namespaces), and transplant the subfolders to the top level. The FastMail web interface is one of these. But ALLCAPS INBOX is kind of ugly, so the FastMail interface also chose to translate INBOX to Inbox and back again. Then users also wanted to still be able to have subfolders of Inbox the way they can have subfolders of anything else:

Inbox
Inbox/ProjectX
Archive
Archive/ProjectX
Drafts
Sent
Sent/ProjectX
Trash

So we made the interface show INBOX.Inbox.ProjectX as a subfolder of Inbox, and everyone was happy. For a while…

Alt Namespace

I’ve written about altnamespace a bit already in the cross domain sharing post.

The alternative namespace presents the user’s top level folders alongside INBOX rather than as inferiors. It’s a configuration option to Cyrus IMAPd which was released in the early 2000s, after had already made our choice of configuration.

We already offer altnamespace on a separate port (992) and recommend port 992 for some clients because they are buggy with INBOX as the namespace prefix.

Until recently, Cyrus IMAPd’s handling of INBOX subfolders via Alt Namespace has been very problematic. You could wind up with duplicate INBOX folders and broken listings, which was a reason for not making it the default for new users.

I spent a lot of time workshopping our options, and in the end the RFC requirement that ALLCAPS INBOX had to match LIST and the fact that every other server always returns it as INBOX in that case convinced me that the folders really had to be subfolders of ALLCAPS INBOX, rather than using Inbox as the name. Also, any other spelling of INBOX needed to not be allowed at the top level, because of the case insensivity rule.

So I created another prefix, “Alt Folders” (actually it’s configurable), and names which won’t be valid at the top level get translated under there. You can read more about exactly how the name mappings work in my mailing list post written at the time.

This was the right choice, the compatible choice, everything good - except…

The choices of the past

Unfortunately, all the FastMail users’ subfolders of Inbox in the web interface are INBOX.Inbox.foo, not INBOX.INBOX.foo, because we chose that to be prettier back when we first implemented the option.

We could have built more layers of hacks on top of this, but the right answer is to rename those folders to be INBOX.INBOX.foo so they appear to the IMAP client as INBOX/foo rather than Alt Folders/Inbox/foo.

There is now a per-account hidden setting inside the FastMail client so that the web interface uses the INBOX.INBOX path for Inbox subfolders, and we are migrating users over and renaming their folders as we go. All users without subfolders and new accounts will be switched first, and then the users who require folder renames will be done.

The most complex cases are a handful of users who have folders both called INBOX.INBOX.foo and INBOX.Inbox.foo in their account. A naive rename will fail in those cases, and we may wind up contacting these users individually.