Twitter VPS | Q&A: DreamHost CEO Simon Anderson Discusses Outages And Their Impact

DreamHost CEO Simon Anderson discussed the comapny's recent outages and their impact

(WEB HOST INDUSTRY REVIEW) - On January 20, DreamHost detected unauthorized activity on its database prompting it to reset all customer FTP passwords. About a week later, its VPS and dedicated hosting customers experienced outages after a Debian OS update went awry.

With more than one issue in the span of a week, DreamHost technical support was working overtime to hang on to unhappy customers. While many DreamHost customers vowed their loyalty to the web host on Twitter, others were quick to threaten moving their business elsewhere, as some customer sites clocked more than 24 hours of downtime.

DreamHost

In an email QA with the WHIR, Anderson talked about how DreamHost is working to maintain customer relationships, and its plans to avoid a similar incident in the future.

WHIR: First, can you provide a bit of background on what happened during the outages? I know there was one related to a security breach, and a separate incident related to a Debian OS autoupdate. I also noticed on Monday that DreamHost's own website was down. Were the incidents all related?

Simon Anderson: DreamHost hosts over 1.2 million domains, and we're a big proponent of free speech, so we sometimes naturally become a target for people wanting to shut down certain sites or other malicious activity. We detected a potential hack of one of our internal databases containing FTP and shell access passwords (but no personally identifying information) on the morning of January 20, and although there was no evidence that the hacker downloaded the database table, we took the precaution of resetting all customers' FTP and shell passwords to protect against any malicious activity on their sites. We've taken a whole series of steps to further secure our systems against hacks, including increasing separation and limiting access between internal systems, and increasing our password protections for customers.

The second incident on January 29 was some downtime for VPS and dedicated customers due to an update of our Debian operating system that failed, and in the process removed some essential files that had to be reinstalled by our systems administrators. We've changed our software update processes to ensure that this event will not reoccur.

WHIR: It seemed that customers had a lot of negative things to say about DreamHost on Twitter and in the comments section of the status page. How do you deal with this? How did the outages affect your relationships with customers?

SA: We take our customer's experience with DreamHost very seriously. DreamHost is well known for our super-engaged support. Our tech support team was working overtime during both incidents to communicate the issues with customers and ensure that customer services were fully functioning as soon as possible. We are big fans of social media for quickly communicating customer satisfaction, or not, because it enables us to swiftly act on issues. There were a range of both positive and negative comments on Twitter during the incidents; we were encouraged that the positive comments recognized our openness in communication and the preventative action we took to protect customers, even though it meant that we'd see broad public awareness of the issues. We believe we need to earn the trust of our customers every day, every year, so that's what we're focused on right now.

WHIR: How is DreamHost dealing with customers wanting to leave? Is there some kind of discount offered to affected customers?

SA: We've been working with customers individually to ensure that we're living up to the standards we've committed to in our uptime guarantee. We want to do right by our customers, and the best way to do that by acting quickly to address issues. We've already begun work on hardening our infrastructure and improving our systems. Ultimately we believe that customers will stick with us for the great value and flexibility that DreamHost offers.

WHIR: What are some of the key steps in dealing with an outage of this scale? DreamHost technical support manager Brian Hill briefly talked a bit about this process in an interview with the WHIR earlier this month, but can you give a more specific overview of the process during this outage?

SA: We have a detailed Emergency Action Plan for any outage that affects customers. Our EAP includes rapid communication between our CEO, VPs and managers 24/7 to understand the issue, make quick decisions, coordinate our internal and external response and communications, and above all get customers' websites, apps and blogs back online. The EAP was used in both the instances described above, and based on learning during these recent events, we have worked to improve and refine our procedures.

WHIR: Can you describe the next steps that DreamHost is taking to avoid this in the future?

SA: We've implemented new software and reconfigured our internal systems architecture to close off the path used for the intrusion. We've also implemented some additional security enhancements to harden the security of our systems overall. We're also collaborating with other hosting companies to ensure we share information on potential security threats for mutual knowledge and defense. We've also made changes to the way we update software to ensure that the Debian update issue, or others like it, will not reoccur. With regards to the network, we'd been testing some new hardware for DDOS mitigation and we'll be implementing that in the next month.

We've always been focused on constant improvement and innovation at DreamHost, which is why so many individuals, businesses and developers have selected us as their web host of choice. We're investing significant dollars in new infrastructure and software in 2012 " like the Ceph distributed storage system " that will enable us to provide the next generation of web services for our customers, with great features and at a great price.