Fisher Plaza Fire Felt from Seattle to East Coast: Lessons from a Data Disaster
In case you missed it over the holiday, an electrical fire and power outage at Fisher Plaza near Seattle Center late on Thursday night disrupted a number of websites and services, including those of local tech companies Redfin, Survey Analytics, and Big Fish Games, as well as Microsoft’s Bing Travel site (formerly Farecast), Verizon’s DSL service in the Seattle area, and local television and radio stations including KOMO.
There were no injuries, and most operations were back to normal by the weekend, though Bing Travel was down until late Saturday morning. The news was reported by local and national outlets, including TechFlash, The Seattle Times, The Seattle P-I, and CNET.
Internet companies were directly affected as far away as Marlborough, MA-based Authorize.net (now owned by CyberSource), a credit-card service for online merchants that uses a data center at Fisher Plaza. And there was a ripple effect from there. Annette Tonti, the CEO of Rhode Island-based MoFuse, a network of build-it-yourself mobile sites, says her company’s service was disrupted on Friday because it uses Authorize.net to process credit cards. “The issue for us was getting customers signed up,” Tonti says. “However, we were not affected too long and everything appeared to be working fine by later in the day.”
“Companies should have servers at various physical locations, spread far apart, to keep isolated incidents like a fire from taking down a service,” says David Berube, MoFuse’s founder and chief architect. “I’m sure Authorize.net does have a redundant system, and their quick response to get service back up shows to me that they do have some sort of redundancies in place.”
Closer to home, there has been quite a lot of discussion about what went wrong, and how companies can better prepare for such outages, which seem rather inevitable. The cause of this particular fire is still under investigation.
Praerit Garg, co-founder of Symform, a data storage startup in Seattle, agrees it’s important to keep crucial data backed up in different places. And providers of Internet cloud services should ensure that distributed storage happens automatically, he says. “Localized disasters will continue to happen and this one is yet another reminder that cloud computing services must be architected accordingly,” Garg says in an e-mail. “We believe it is critical to build attributes like redundancy and geo-distribution inherently into the cloud service in order to recover quickly.”
But Vivek Bhaskaran, co-founder of Seattle-based marketing startup Survey Analytics—whose servers went down until early Saturday morning as a result of the Fisher Plaza outage—points out that having redundant data centers can more than double your costs. “This has opened my eyes to the vulnerabilities that I have,” he says. In the short term, Bhaskaran says, he will move his company blog to a separate hosting service, and set up an automated outgoing phone message and an error page on the company site in case of emergency.
In the long run, Bhaskaran says his company needs to set up a redundant data center, despite the cost. “We [already] have full redundancy within the data center,” he writes. “So if any one of our servers dies (hard drive failure, etc.), other servers pick up the slack automatically. If one of our database servers crash, we have replicated servers that will come online automatically within seconds. However, if the entire data center goes offline, our current plan does not have a solution to move to another data center within minutes. We have full copies of the data stored offsite—but that is only the data.”
He continues, “What we need to get to, is to operate out of a different data center in case of a massive emergency like this. This undoubtedly will double our operating expenses, but given the business we are in, we simply need to do this. Over the next three months, we’ll be figuring out a solution so that we can sustain turning off power to our primary data center and things move to our backup data center.”