Fastmail Fights Off Ransom Cyberattack

Many email providers were hit with a Distributed Denial of Service attack last week. For Fastmail customers, no mail has been lost and, as always, your data remains safe.

---

What happened: in a nutshell

Over the last week, Fastmail and other email providers were subject to ongoing Distributed Denial of Service (DDoS) attacks from someone demanding payments. We have experienced attacks like this in the past (read about the last big attack in 2015) and have protection in place at multiple levels to weather these intrusions.

We believe that there is no specific reason that Fastmail was attacked. DDoS attackers have just chosen to attack a set of email providers at this current moment.

DDoS network “owners” have a resource (a large network of compromised computers) with a limited lifetime until they lose access, and they target companies to extort for bitcoin payments.

We never pay extortionists. Doing so encourages further ransom payments and future threats to us and to others.

What is a DDoS attack?

A denial of service attack is where an individual or organization tries to overwhelm a computer system by sending through so many requests that it can’t cope, and it crashes. Identifying a flood of messages from a single location and blocking that location is relatively simple to detect and protect against.

A distributed denial of service attack is where the individual or organization sends these requests from a lot of different locations making it much harder to isolate the bad requests from the good.

In simple terms: think of your Fastmail account like a shop down the road. When a DDoS attack happens, the attacker creates a big traffic jam by filling the entire road with cars. The shops are still there, and everything inside the shops is functioning normally, but the roads are all blocked and your car can’t get you to the store.

An attack normally consists of a number of different approaches:

• A volume-based attack is when attackers send a flood of traffic to overwhelm a website’s available bandwidth. If there’s more traffic coming than the link can handle, then it doesn’t matter how good the server is at handling the requests, because the network link is full. For instance, we saw traffic coming at us in excess of 270Gb/sec (our normal load is usually under 10Gb/sec);
• A Protocol attack, such as a SYN-flooding attack, is when attackers send just the initial piece of a connection (a network packet tagged as “SYN” or synchronize, asking the server to start a connection). Servers use resources to track connections, and a large number of SYN packets without continuing the rest of the connection can use up server resources while they wait for the rest of the connection, which never comes;
• An application attack makes repeated requests to resources that are expensive for the server to generate. Unlike the other two attacks, they require some knowledge of the specific site being targeted.

An attacker will often use a network of computers to generate the attack, such as a botnet. These are compromised computers around the world, which respond to remote commands to send requests.

How we manage an attack

Effectively managing a DDoS attack requires work at multiple levels.

• On the Fastmail service itself, we have defenses within our code to detect and prevent inappropriate requests from clogging up our server - such as caching, rate limiting, and local block lists;
• In our data center, they are able to route and defend against bad traffic by detecting behaviors, and rate-limiting specific regions that are producing excessive traffic flows before they reach our systems;
• At the network edges, we use a DDoS mitigation service that detects and scrubs botnet traffic before it enters our data center. The challenge for this service is distinguishing between valid and invalid traffic so that it can keep the traffic small enough to fit through our “street” (the link between our systems and the internet) without interrupting legitimate customers—so this is only switched on when the traffic quantity gets higher than we can handle internally.

Attackers modify their strategy during attacks, so it’s never just a matter of set-and-forget when dealing with an active attacker.

When an attack happens, we are in constant communication with our providers to coordinate our responses and adapt to the changing shape of the traffic being sent our way. Sometimes, this means that some of our customers can see significant slowdowns, while others may not even be aware that an attack is underway.

For instance, during volume-based attacks, we need to work with our providers to implement filters further out in the network so that they don’t overwhelm the capacity of the network links inside our data center. In terms of the car analogy above, we need the provider to make sure that cars that don’t intend to make purchases at our shop don’t enter the road in the first place! The filter doesn’t have to be perfect, but it does need to keep the traffic down to a manageable amount while producing as little impact as possible on legitimate customers and visitors.

What our customers see during an attack

Depending on the nature of the attack at any given moment, customers might:

• not be able to access us at all for a period of time, you’re stuck in the gridlock traffic jam, because there’s just too much traffic, or they live in the region where a large amount of botnet traffic is coming from;
• be able to access us, but slowly, you’re crawling along the interstate during peak hour, because there’s still a lot of traffic, but we’re processing requests still;
• be able to access us normally, the street isn’t too busy at the moment you showed up, and they don’t notice anything is wrong.

Why were we targeted?

We don’t believe we were specifically targeted, just that the attackers decided to target email providers. Others also saw attacks from the same person, as reported by The Record. We are all independent mail providers with a small enough network presence that a powerful botnet could overwhelm our service if directed at us.

• Runbox blogged about their experience.
• Posteo blogged about their response.
• Mailbox.org blogged about their experience.
• Tutanota tweeted about a DDoS attack in the past week as well.

None of us have paid the ransom, and we are working together and with our respective law enforcement to prevent this attack and anything like it in the future, to us or to anyone else.

Here’s the first ransom note to us, which we received to multiple of our contact email addresses on Friday:

From: Cursed Patriarch <cursed.patriarch@[...]>
Subject: DDoS Attack

Hi,

I will start 1-2 hours attack on your site. It will not be hard as I don't want to impact your business now. Just check your logs to see that I'm for real.

Pay me 0.06 BTC to [...] and I will never attack you again.

If you don't pay within until Monday, total shut down is coming, cheap protection will not help my fee will increase and if you refuse you will lose much more then that.

Pay 0.06 now to prevent suffering.

Best regards,
Cursed Patriarch

*P.S. This is disposable email. Do not reply.*

They contacted us from multiple email accounts, including a Fastmail trial account, which was used to contact both our support and some of their other victims. In all their interactions with our service, connections were made via Tor—a networking service used for anonymity, meaning that their actual location and identity are hidden from us.

Our next steps

Fighting off DDoS attacks can be like trying to fight spam. The moment one technique becomes effective at detecting and quarantining bad content, the attackers shift to a new approach.

Obviously, this also means that we do not want to detail the entire scope of our countermeasures, or the response times that each of them requires, as that information is useful to a motivated attacker.

During this attack, we developed several new tools to mitigate future similar kinds of behavior we saw. We are also continuing to discuss improvement strategies with our network providers and DDoS specialists.

Even once this current set of attacks finishes, new attackers can come at any time. Keep an eye on our Twitter account and on our status page to stay up to date with any service availability changes.

Lastly…

We know that Fastmail is a tool that people rely on to stay connected. Especially during an attack, but at all times, we work around the clock to keep you up and running. We’re sorry to those who were impacted by the work of this bad actor. The whole team at Fastmail appreciates the messages of support and solidarity we were sent during this time over Twitter and through support tickets as we worked hard to remain available. Thank you for your patience and understanding.