Requests to DOBT servers timing out
Incident Report for Department of Better Technology
Postmortem

The Domain Name Service, or DNS for short, is the way computers everywhere convert the “nice” domain names like dobt.co to less friendly IP addresses like 192.30.252.153.

On Monday afternoon at around 2pm EST, someone attacked DNSimple, the third-party provider we use for our DNS, and brought it to its knees through a massive distributed denial-of-service (DDOS) by flooding it with requests.

If you tried to access any of DOBT’s services the the few hours that followed, your computer would have been unable to look up our address and after a time, would have given up trying to contact our server.

In summary, a denial-of-service attack on our DNS provider made it impossible to find our servers on the internet.

What we did wrong

Basically, we had too many eggs in one basket. Although we rely on DNSimple because they have their own protections and redundancy in place, a massive DDOS like the one that occurred yesterday still targets a single, vulnerable point.

What we did right

We trust DNSimple to get most things right, and we feel they did. You can read about the attack here.

Before the attack subsided, we did manage to get access to our control panel and switched over to another third party DNS provider that was not under attack.

What we’ll do in the future

Although it’s actually built into the way DNS communicates, many DNS service providers do not not allow their clients to simply transfer their DNS lookup data to secondary DNS servers. We are currently working on a tool that will allow us to use more than one DNS provider and keep all our name service records up-to-date.

As always, we remain dedicated to building the best possible experience for our customers, and we know that reliability is a key component of what we do. We are deeply sorry for any inconvenience we may have caused, and we will work tirelessly to ensure that no similar incident affects us in the future.

Posted Dec 02, 2014 - 09:39 CST

Resolved
Due to a DDOS attack on our third-party DNS provider, DOBT's servers are not receiving requests and are therefore inaccessible.
Posted Dec 01, 2014 - 13:15 CST