Converged Data Center Infrastructure
Showing results for 
Search instead for 
Do you mean 

When IT nightmares become disasters

GuestBloggerCDI 2 weeks ago 2 weeks ago


Brian K.jpgBy Guest Blogger, Brian Knudtson, Senior Technical Marketing Manager

Like many other people in the tech field, I am a fan of HBO’s Silicon Valley. Also like many other people in the tech field, some of the episodes feel a little too real. The episode where Richard Hendricks has night sweats stressing over managing his startup really caught my attention.

Losing sleep due to worrying about business is something many IT people, including myself, can relate to. While something as drastic as an entire company being deleted with one line of bad code rarely happens (as that story was a hoax), every IT professional who has more than a year or two of experience probably has a IT nightmare story to tell.

My personal story came early in my career when I was still a web developer. I was working on a commerce website testing new code in our development environment and wanted to test the code on a clean database. Having done this dozens of times, I logged into the database and cleared all rows out of the main table. When I refreshed the webpage I was working on, it still showed the old data. Then it hit me. The last time I had logged into the database, it was to troubleshoot a production problem. Sure enough, I had wiped out the production database.

I then performed the walk of shame as I went to get the database restored from a backup, saying short little prayers that we had good backups of the database. Lucky for me, thenightmare.png database administrators had a rigorous backup policy in place and were able to restore the production database with only a few hours of lost data, less than an hour of downtime, and, thankfully, very little teasing.

Hoping I wasn’t alone in having such a horrible IT day, I asked a few customers and solution architects I’ve worked with in the past for other IT nightmare stories and how they recovered from them. And as they used to say on Dragnet, I've held back the names to protect the innocent.

One customer had a story that was very similar to mine, only with more impact. After accidentally overwriting a production database on their proprietary AIX server in the middle of the day, they found that restoring the database was not so easy. Since it was one of their core systems, employees had to revert to keeping paper records during the day, and the IT team spent the evening keying the data back into the database after it was finally restored and business had ended for the day. This resulted in a higher-than-normal error rate and a significant pizza bill.

While these nightmares will inevitably still happen, HPE SimpliVity solutions can simplify recovery. Whether restoring individual files, entire databases, full virtual machines, or even a complete data center from a disaster, the HPE SimpliVity technology has the built-in data protection that will help an IT professional sleep better at night.

Another customer shared a story where they accidentally deleted a critical virtual machine. This story had a much happier ending, though, because they had SimpliVity hyperconverged infrastructure deployed. Not having any local backups configured, the administrator feared that the entire company was in jeopardy – but he realized that the SimpliVity technology was taking remote backups of the virtual machine. Now his fear was slightly relieved as he sat down to wait for the remote restore to complete. Three minutes later, the restore process was complete and the VM was powering back on.

While HPE SimpliVity’s technology saved this story from escalating to a complete nightmare, disaster recovery is still a nightmare I have, even though I don't directly support a production data center. A disaster recovery nightmare like this is often a reason why customers eventually choose HPE SimpliVity technology to avoid running into this problem again. Before deploying SimpliVity hyperconverged infrastructure, one customer had a replication setup for its data center that never fully worked. When disaster hit, the disaster recovery process became extremely complex due to incomplete replication. This customer now has the SimpliVity technology for its core data center and disaster recovery solution to avoid a recurring IT nightmare.

Whether it’s a BitLocker attack, a remote data center with really old equipment, a server that doesn’t come back from a hard reboot, or a simple bottle of Tres Comas accidentally sitting on the delete key — only three of which actually happened in real life — every IT infrastructure has its risks. Making sure you have the proper data protection strategy in place, and an infrastructure that will make it easy to implement, is critical.

Learn more about HPE hyperconverged solutions and how they can help you avoid your IT nightmares.

Brian Knudtson is an IT industry expert and vExpert with 18 years of experience. He holds a BS in Computer Science from the University of Nebraska Omaha, and several certifications including VCAP-DCA, VCAP-DCD, VCP-DT, MCITP-Enterprise, and HP Master ASE. Brian has previously worked for SimpliVity as a Technical Marketing Manager. Before that, he was a Design Architect for OneNeck IT Solutions, a Systems Administrator for Blue Cross Blue Shield Nebraska, and a Senior Web Programmer at Inacom. He also wrote for TechTarget at and the SimpliVity Blog on topics including VMware/virtualization, VDI, disaster recovery/business continuity, data center infrastructure design, cloud computing, and hyperconverged infrastructure.

Follow HPE Composable Infrastructure

0 Kudos
About the Author


June 6 - 8, 2017
Las Vegas, Nevada
Discover 2017 Las Vegas
Join us for HPE Discover 2017 in Las Vegas. The event will be held at the Venetian | Palazzo from June 6-8, 2017.
Read more
Apr 18, 2017
Houston, TX
HPE Tech Days - 2017
Follow a group of tech bloggers for a new HPE Tech Day, a full day of sessions about how to create a hybrid IT, from hyperconverged to Composable Infr...
Read more
View all
//Add this to "OnDomLoad" event