Image via Wikipedia

I was just reminded yesterday of how delicate technology is and especially that having to do with date centers and major websites. Yesterday I proceeded to my website’s Statcounter stat service and noticed unique new blog post on their company blog. The reason this post caught my attention is because it read “Outage – Explosion at The Planet“. Talk about an interesting title so I went ahead and checked it out. I learned that one of their various data center providers had an electrical issue which cause an explosion. This led to a fire in the electrical room which forced the fire department to cut power. Without time to spare, someone cut the power and let all of the servers shut off. This of course did not turn out well for a bunch of them, as they were not shut down properly although the data center really had no choice because they had to evacuate and allow the fire department to fight the fire.

The damage was pretty severe. Even though it sounded like it was pretty contained and did not damage any equipment, the fire reportedly damaged power lines inside the data center beyond repair. The Planet decided they would rerun new conduit and lines all the way inside from the curb. During the time they inspected the damage, they also worked hard to get their servers there at the data center back online. This was a difficult and I’m sure humbling task as they had thousands of servers at this data center. This was in an unstable environment with damaged wiring and faulty generators. In the end there were a bunch of servers that were damaged beyond a single reboot. Many of these had to have OS re-installs or had corrupted hardware such as hard drives and boards. There were many companies that lost thousands upon thousands of dollars through this crisis that spanned a few days.

It just makes me think about how dependent we are on not only reliable technology but also in redundancies of systems and their backups in case of catastrophic failures. The frightening thing was that this situation wasn’t even that catastrophic. Just think of what the loss would have been if even a hundred servers caught fire and were destroyed. The loss of a small percentage of servers would have amounted to loss of hundred of websites and a bunch of critical information which would have been a nightmare for an IT specialist to recover. Some of the planet’s customers are experiencing such a loss now as there are corrupted systems which have to be sent to data recovery specialists. Remember that in anything you do, create multiple backups and do it in a safe and secure environment that is in a different location from the computers or servers you are backing up. Even the most cautious technicians will run into these troubling realities.