VSA Performance issue. Strange VMWARE CPU latency!

[GarethRUK]

Hi,

We have a set of Dell R720's with 20TB of mixed SSD and 10kSAS running HP VSA. We are experieing some performance issues but the cause is not clear. Only the VSA runs on these nodes!

The provided install instructions etc don't make any recommendations regarding VMWARE configuration. However, I'm suspecting the further configuration is required.

We have a VMWARE virtualised web servers which do a lot of VERY small IO. However, it does this constantly. We are seeing latency as high as 1 second within the guest and not far off that for the datastore within the VMWARE performance stats. All storage provided by the impacted VSA appears to have high latency.

However. things get very very strange form here on. If we use DD against the web server to artificially create load the latency drops as the IOPS rise.

As an example the latency can be as high as 300-400ms but when we load the storage it drops to 10-20ms.

After a lot of digging around I've disocvered that the two cluster VSA we are using seems to have something going on with CPU scheduling. Within the VSA VMWARE host I can see that CPU usage is round robined across all 12 cores on the machine. The VMWARE CPU latency metric is showing as 50% constantly and CPU wait looks high.

I don't observe the CPU latency issue on other VSA's pairs but they are not exhibiting similar latency issues and I'm pretty sure they are not loaded quite as high. For the impacted VSA the CMC is showing a load of around 5,000 IOPS.

I find the VMWARE CPU usage strange. Surely, you would want to ensure the VSA exclusively uses 2 vCPU's with affinity to two physical CPU's? Is there any futher instructions for VSA. What further diagnostics can I undertake?

Regards,

Gareth 

VSA servers  are as follows

2x Dell R720 12cores/64GB RAM 4x LB806M 744GB @ 21000 IOPS, 20x ST1200MM0007 1175GB @ 295 IOPS

[a_o]

Seems like the iSCSI  vmkernel/vSwitch  sharing issue discussed previously on this board.

Take a look at the posts towards the end of this thread.