Shifting to Software-Defined
Showing results for 
Search instead for 
Did you mean: 

When IT nightmares become disasters



Brian K.jpgBy Guest Blogger, Brian Knudtson, Senior Technical Marketing Manager

Like many other people in the tech field, I am a fan of HBO’s Silicon Valley. Also like many other people in the tech field, some of the episodes feel a little too real. The episode where Richard Hendricks has night sweats stressing over managing his startup really caught my attention.

Losing sleep due to worrying about business is something many IT people, including myself, can relate to. While something as drastic as an entire company being deleted with one line of bad code rarely happens (as that story was a hoax), every IT professional who has more than a year or two of experience probably has a IT nightmare story to tell.

My personal story came early in my career when I was still a web developer. I was working on a commerce website testing new code in our development environment and wanted to test the code on a clean database. Having done this dozens of times, I logged into the database and cleared all rows out of the main table. When I refreshed the webpage I was working on, it still showed the old data. Then it hit me. The last time I had logged into the database, it was to troubleshoot a production problem. Sure enough, I had wiped out the production database.

I then performed the walk of shame as I went to get the database restored from a backup, saying short little prayers that we had good backups of the database. Lucky for me, thenightmare.png database administrators had a rigorous backup policy in place and were able to restore the production database with only a few hours of lost data, less than an hour of downtime, and, thankfully, very little teasing.

Hoping I wasn’t alone in having such a horrible IT day, I asked a few customers and solution architects I’ve worked with in the past for other IT nightmare stories and how they recovered from them. And as they used to say on Dragnet, I've held back the names to protect the innocent.

One customer had a story that was very similar to mine, only with more impact. After accidentally overwriting a production database on their proprietary AIX server in the middle of the day, they found that restoring the database was not so easy. Since it was one of their core systems, employees had to revert to keeping paper records during the day, and the IT team spent the evening keying the data back into the database after it was finally restored and business had ended for the day. This resulted in a higher-than-normal error rate and a significant pizza bill.

While these nightmares will inevitably still happen, the HPE SimpliVity 380 powered by Intel® can simplify recovery. Whether restoring individual files, entire databases, full virtual machines, or even a complete data center from a disaster, the HPE SimpliVity 380 has the built-in data protection that will help an IT professional sleep better at night.

Another customer shared a story where they accidentally deleted a critical virtual machine. This story had a much happier ending, though, because they had SimpliVity hyperconverged infrastructure deployed. Not having any local backups configured, the administrator feared that the entire company was in jeopardy – but he realized that the SimpliVity technology was taking remote backups of the virtual machine. Now his fear was slightly relieved as he sat down to wait for the remote restore to complete. Three minutes later, the restore process was complete and the VM was powering back on.

While HPE SimpliVity’s technology saved this story from escalating to a complete nightmare, disaster recovery is still a nightmare I have, even though I don't directly support a production data center. A disaster recovery nightmare like this is often a reason why customers eventually choose HPE SimpliVity to avoid running into this problem again. Before deploying SimpliVity hyperconverged infrastructure, one customer had a replication setup for its data center that never fully worked. When disaster hit, the disaster recovery process became extremely complex due to incomplete replication. This customer now has the SimpliVity technology for its core data center and disaster recovery solution to avoid a recurring IT nightmare.

Whether it’s a BitLocker attack, a remote data center with really old equipment, a server that doesn’t come back from a hard reboot, or a simple bottle of Tres Comas accidentally sitting on the delete key — only three of which actually happened in real life — every IT infrastructure has its risks. Making sure you have the proper data protection strategy in place, and an infrastructure that will make it easy to implement, is critical.

Learn more about HPE hyperconverged solutions and how they can help you avoid IT nightmares.

Brian Knudtson is an IT industry expert and vExpert with 18 years of experience. He holds a BS in Computer Science from the University of Nebraska Omaha, and several certifications including VCAP-DCA, VCAP-DCD, VCP-DT, MCITP-Enterprise, and HP Master ASE. Brian has previously worked for SimpliVity as a Technical Marketing Manager. Before that, he was a Design Architect for OneNeck IT Solutions, a Systems Administrator for Blue Cross Blue Shield Nebraska, and a Senior Web Programmer at Inacom. He also wrote for TechTarget at and the SimpliVity Blog on topics including VMware/virtualization, VDI, disaster recovery/business continuity, data center infrastructure design, cloud computing, and hyperconverged infrastructure.

Follow HPE Composable Infrastructure

0 Kudos
About the Author


June 18 - 20
Las Vegas, NV
HPE Discover 2019 Las Vegas
Learn about all things Discover 2019 in  Las Vegas, Nevada, June 18-20, 2019
Read more
Read for dates
HPE at 2019 Technology Events
Learn about the technology events where Hewlett Packard Enterprise will have a presence in 2019.
Read more
View all