topic Re: VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100 in StoreVirtual Storage

VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

Ben Drawbaugh — Tue, 21 Dec 2010 22:38:23 GMT

Just deployed a fresh new VSA 9.0 on my ESXi 4.1 (260247). Configured the IP and hostname and installed the latest patches. I created a new management group, new cluster and new volume. I configured the VMA NIC on a dedicated iSCSI virtual switch that only has the VSA and the VMKernel port that has been added to my ESXi software HBA -- this uplinks to my core switch.

Everything appears to be fine, but when I try to run a physical to virtual conversion with the destination volume on the VSA, a error is logged about 10 minutes into the file copy process. At the same time, the destination virtual machine locks up and the process fails.

The error is.

Message: The storage system 'ausvsa1.xxxxx.local', in cluster 'AUS1', latency = 65.013, which exceeds 60.000.

EID_LATENCY_STATUS_EXCESSIVE E00060100

This is followed by two more events:
The system CPU utilization (100%) status is 'Saturated'.
The storage system 'ausvsa1.xxxxx.local', in cluster 'AUS1', latency status is 'Normal'.

The ESXi server has plenty of CPU and memory headroom and the VSA is configured for 2000hz of CPU as required in the VSA Install and Configuration Guide.

The very same p2v job runs successfully to the local disc of the ESXi server. Anyone know what this error means?

Re: VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

Scott Smith_7 — Wed, 22 Dec 2010 23:37:50 GMT

Ben,

Is the uplink to your core switch a non-blocking port? I've seen oversubcription cause latency issues.

Regards,

Scott

Re: VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

Ben Drawbaugh — Thu, 23 Dec 2010 04:04:00 GMT

I thought of this and put the VSA and VM_kernel port on the same virtual switch so that ESX<>SAN traffic avoided my core switch all together. In fact the only reason why I have an uplink port is so I can replicate a volume to another VSA (and management). I can try removing the uplink port completely if you think that will make a difference. The switch is a Extreme Networks Black Diamond that I don't think very much of.

It is good to know that this is a network issue, I wasn't sure what latency it was talking about (network, disc, who knows).

Re: VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

Ben Drawbaugh — Sat, 01 Jan 2011 02:27:18 GMT

I gave up and decided to punt. I re-installed ESXi 4.1 and then redeployed the VSA and the issue didn't persist. So I guess it was a VMWare issue all along.

Re: VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

virtualmatrix — Mon, 03 Jan 2011 19:10:59 GMT

We had seen very similar symptoms (latency+CPU @ 100%) caused by vmkernel blocking on certain VMFS operations and causing the VSA vCPU to get stuck at 100% I/O-wait until the I/O timed out.

We converted all of our VSAs to use RDMs instead of VMFS-based VMDKs and haven't seen the problem since. :-)

Re: VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

Dan Nichols — Tue, 11 Jan 2011 17:46:38 GMT

I have just experienced the same issue with VSA version 9.

I installed a new ESXi server 4.1 build 320137 on a HP DL365 G5 and put a version 9.0.00.3561 VSA on there. I created a 1.95TB VMDK and attached it to SCSI 1:0 on the VSA.

I created a 500GB thin volume on the VSA and presented it to the ESXi host via an internal only vSwitch.

I created a VM on the new VSA volume and whilst Windows 2008 R2 was installing the CMC logged the latency > 60ms error a few times and eventually the VSA declared the volume inoperable and the VM hung completely.

I was watching the disk latency values and the VSA lun was reporting latencies of 45 seconds+ whilst the physical lun that the VMDK was sat on was reporting maximum latencies of 20ms (average <5ms).

I have now deployed a VSA running 8.5 and the issue has vanished.