How many of us can count stopping an Exchange DDOS as a highlight of one's life?

What's so great exactly about stopping a DDOS? Imagine that you're a postman who delivers letters, and one day, you've received plenty of letters to be delivered. As a postman, it is your task to deliver these letters. And when the recipient received these letters, they barred you from delivering letters anymore because you were delivering advertisements, useless stuff to them. And you've received plenty of these letters so much so that

  1. you're paralyzed from the volume of these letters
  2. the recipients have barred you.

That was an analogy of what the Exchange DDOS had caused the customer.

One of the greatest and most exciting thing to ever happen to me was to be in the thick in the middle of a DDOS, and stopping it. It didn't start like that at all. That morning, I was going about my routine, when I received a call, stating that the customer's email was not being sent out, and this was impacting the whole company (which I later found out to number at least about 100+ people). I immediately went to the customer's data center.

On arriving, I was briefed by the tech guy there that emails weren't getting sent out.

I first ran a few simple tests:

  • Telnet to a SMTP server
  • Sending a simple email
  • Receiving the same email above

Next, when I looked at the Exchange queue, I was surprised to find hundreds of thousands (at least 400K to 500K) of emails. And it kept growing. The first thing I did was to investigate where the mails were coming from, and I took steps to block the originating source. When the emails stopped coming, then I started clearing the queue. Initially, I took the documented step of clearing the queue, but when it became clear that was too tedious, undocumented step of clearing the queue was taken.

Then, I tried to send a simple email using Outlook, however, it didn't come out. I puzzled that out for a while, before eventually thinking that the Exchange server was on a blacklist. And when I checked, my assumption was correct. The Exchange server was blacklisted. Since that was the case, after checking with the tech guy that they have another resource elsewhere to send emails, I rerouted the emails to be sent elsewhere.

After rerouting, I ran a few simple tests, confirmed that things were working as they should, and left.

That took me nearly the entire day to fix. Had the customer have ArcSight ESM installed, it could have been faster for me.

And that was the story of how I managed to stop an Exchange DDOS.