Help test proxy to improve IMAP performance

Update: Due to a packaging error, the 1.0 proxy wouldn’t work on most machines. The new version 1.1 should work on XP/Vista machines. You need to uninstall the version 1.0 proxy first before installing version 1.1. You will also need to install the MS VS 2008 redistributable. See below for details.

---

IMAP is a great protocol that allows you to access your email via a wide variety of email clients (Outlook Express, Thunderbird, Outlook, etc), while keeping everything on the server and having a consistent synchronised view of your email at home, work, or when accessing webmail from anywhere.

The IMAP protocol does have an annoyance though. When you switch folders in an email client, the email client has to re-fetch a list of “flags” from the server on all messages in the folder to check which messages are flagged, seen/unseen, etc. On small folders, it’s not a problem, but on large folders, that can be quite a lot of data to transfer (several megabytes) just to make sure the flags list is consistent. If you have large folders (eg folders with >10,000 messages), you might notice this delay as the email client “pauses” after you select a folder before doing anything else.

The long term solution to this is a feature called MODSEQ/CONDSTORE, but this requires changes to IMAP clients and servers, and while FastMail’s server does support it, it’s not likely many clients will be supporting it soon.

There is another solution that can help. The data fetched when retrieving the flags data is very repetitive, which makes it easy to compress. There’s an extension to the IMAP protocol to allow compression of data. Now again, this requires client and server changes. The good news is that FastMail’s server now supports this. The bad news is that there’s no IMAP clients out there yet that do.

However, there is a work around for this, a “proxy” that compresses/expands the data between your email client and the FastMail server. For this to be useful, the proxy has to run on your local machine, and your email software has to be changed to talk to the proxy, which will then compress/decompress everything and talk to the FastMail server.

We’ve now developed such a proxy, and you can try it out today. Currently this is only for Windows (XP/Vista) users. Here’s what to do:

1. You need the Visual Studio 2008 C++ redistributable (this is needed to make the OpenSSL encryption library work). Download it from http://www.microsoft.com/downloads/details.aspx?familyid=9B2DA534-3E03-4391-8A4D-074B9F2BC1BF (~1.7M) and follow the directions to install it

2. Download the FastProxy installed from here:  http://robm.fastmail.fm/downloads/setup_fastproxy_1_1.exe (~5.4M)

3. Run the setup_fastproxy_1_1.exe program, which will install FastProxy on your machine

4. Run FastProxy from the Start menu

5. You should see a new notification icon appear in the bottom right hand corner that looks look a small house. If you click on it, it will bring up an about window with some basic instructions. If you right-click on it, you’ll get an option to Quit if you want to stop it running. If you hover over it, it will display some statistics.

6. Now you need to change your email client to access FastMail via the proxy. Doing that will depend on your email client, but the main things you have to change are:

   • Change the IMAP server from “mail.messagingengine.com” to “localhost”
   • Change the IMAP port number from 143/142/993/992 to 8143/8142/8993/8992 respectively (eg add an “8” on the front)
   • Turn off any SSL encryption. Don’t worry, if you’re using port 8993/8992, the connection and any data sent to/from FastMail will still be encrypted by the proxy, it’s just that you can’t use encryption when your email client talks to the proxy

7. After you do that, save any changes. Your email client might ask you if you want to update your folder list. If possible, avoid doing that, because it might cause the client to throw away local cache information and it doesn’t need to do that.

8. You should now be able to access your email account as usual, but hopefully you’ll find accessing large folders or downloading a number of emails faster than it was before.

9. One more thing. When you shutdown/restart Windows, the proxy won’t start automatically. You can make it do that by putting a shortcut to FastProxy in your Startup folder. You can do that by doing the following:

   • Right-click on the Start menu and select “Explore all users”
   • Find the “fastproxy” folder in the tree on the left and click it
   • In the section on the right, right-click the “FastProxy - FastMail IMAP Compression Proxy” item and select “Copy”
   • Find the “Startup” folder in the tree on the left and click it
   • In the section on the right, right-click in a blank area and select “Paste”
   • That should create a link to the FastProxy program and make it start automatically at Windows startup time

If you hover over the notification icon, you should be able to get an idea of how much data the proxy is saving you. We’ve found that for everyday use (eg not bulk email downloading, but just reading new emails, switching between folders, etc), savings usually average around 80% (eg your IMAP sessions use 1/5th the bandwidth they previously did).

I’ll start a forum thread shortly where people can discuss the experiences with the proxy.