HPE Infosight Cross Stack Analytics saves a customer millions in unnecessary hardware purchases

[NancyBerliner]

Earth Machines has a large VMware VDI virtual desktop environment running on their 3PAR storage array.  Performance is usually good, so the end users are surprised when all of a sudden they experience high latency to their VDI applications.  They become more and more concerned as the situation shows no sign of improving over the next week and in fact, the impact to the business seems to grow every day.   Employees struggle to do their jobs as even simple day-to-day tasks have become difficult to do quickly.  Finally, the company’s billing and revenue recognition systems become impacted and even Earth Machine’s customers are reporting that they notice delays.

A hardware purchase is a last-ditch effort to fix the problem

The issue is escalated to the CTO who contacts the Director of IT to get working quickly on a resolution.  Virtualized environments, while extremely powerful and flexible, can also present challenges when it comes to diagnosing issues.  As a host and its storage are shared by many Virtual Machines, it can be difficult to differentiate between the different potential sources of a problem. Earth Machines is scrambling to find a resolution, when external consultants hear about the issue and immediately recommend moving some workloads to a new storage array as a way to solve the bottleneck.   Without a clear indication of exactly what is causing the poor performance, the IT Director acquiesces and contacts HPE to purchase more storage.    

The Storage Administrator hopes HPE InfoSight can help

In the meantime, the Storage Administrator gets wind of the issue in the weekly staff meeting.  Fortunately, Earth Machines has registered their storage arrays and vCenter instances with the no-cost HPE InfoSight portal.  InfoSight ingests millions of sensor data points and utilizes cloud-based machine learning to draw out predictions and recommendations for each customer’s specific systems, applications, and workloads.  InfoSight Cross Stack Analytics for HPE 3PAR and HPE Nimble arrays is unique in that its analytics look across the stack, enabling visibility into storage, network and host CPU and memory resources. 

The admin goes directly back to his desk to consult Cross-Stack Analytics in an effort to pinpoint the problem.  The I/O Contention Treemap is an intuitive view of the relative performance of individual volumes and Virtual Machines from the VMware point of view. It includes all the datastores seen by a given instance of vCenter.   The storage admin knows the Treemap will tell him at a glance if there is any VM contention. 

The admin knows the following about the Treemap:

• The size of the box correlates to the number of IOPS being generated by a VM or datastore
• The color indicates the latency that object is experiencing
• The darker the color, the higher the latency

InfoSight’s Cross Stack Analytics identifies a rogue VM

Sure enough, in just seconds the admin is able to look at the left hand side of the image and see that:

• Two Virtual Machines are consuming an outsize portion of the resources.  They are responsible for close to ten times the average workload of the other VMs. 
• The darker boxes on the other VMs indicate that they are being starved of resources. This is most likely responsible for the high latency the users are experiencing.  Hovering over each VM shows each one’s total I/O and average latency.

 Now that the admin has identified a possible source of the problem, he is able to drill down and look at the VMs from a variety of perspectives.  By clicking on the largest box, he is quickly able to see the resource consumption for that runaway VM. 

The top graph in this view is the Latency graph for the VM.  The latency is further broken down into its host and storage/network components.  He looks at the past 24 hours and sees that both the host CPU and the storage array are being stressed during the time in question.  Scanning down the page, the next two graphs show the activity – the IOPS and throughput.  He hovers his mouse over the most recent interval so that the graph will show the exact number of reads and writes for that time period.  “Something strange is going on here” he thinks to himself.  The activity on this VM is virtually 100 percent read.

He immediately enlists the help of the client team who dive into the issue.  The first thing they notice is that the runaway VMs are running a new test version of their anti-virus tool.  The security team consults with the anti-virus vendor who identifies a bug that causes their software to consume an inordinate amount of system resources.  This is starving the other VMs on those hosts, causing the performance bottlenecks.  The vendor supplies a patch for the anti-virus tool and Earth Machines is now rolling it out to all Windows hosts running the new anti-virus software which will prevent any future issues.

HPE InfoSight saved the day

The storage admin was very pleased that with the help of InfoSight Cross-Stack Analytics he was able to resolve this performance issue so quickly, pinpoint the root cause, and notify his Director of IT that no new storage purchase was necessary.  They are both aware that traditional vendors would have thrown more storage at the issue without discovering that a rogue VM was at the root of the problem.   

InfoSight saved Earth Machines millions of dollars in incremental storage investment as well as the time and effort to stand up the infrastructure for a new storage array.   The end result was fast resolution and fast return to normal operations, thanks to HPE InfoSight.