Email Subscription Notifications Suspended Temporarily
We are in the process of making navigation in the Servers and Operating Systems forums simpler and more direct. While doing this, we have to temporarily suspend email notifications for subscriptions. If you are subscribed to one or more discussion boards or blogs in the community, please check them daily to see new content. Notifications will be turned back on in a few days. We apologize for any inconvenience this may cause. Thanks, Warren_Admin
StoreVirtual Storage
cancel
Showing results for 
Search instead for 
Did you mean: 

VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

Ben Drawbaugh
Occasional Advisor

VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

Just deployed a fresh new VSA 9.0 on my ESXi 4.1 (260247). Configured the IP and hostname and installed the latest patches. I created a new management group, new cluster and new volume. I configured the VMA NIC on a dedicated iSCSI virtual switch that only has the VSA and the VMKernel port that has been added to my ESXi software HBA -- this uplinks to my core switch.

Everything appears to be fine, but when I try to run a physical to virtual conversion with the destination volume on the VSA, a error is logged about 10 minutes into the file copy process. At the same time, the destination virtual machine locks up and the process fails.

The error is.

Message: The storage system 'ausvsa1.xxxxx.local', in cluster 'AUS1', latency = 65.013, which exceeds 60.000.

EID_LATENCY_STATUS_EXCESSIVE E00060100

This is followed by two more events:
The system CPU utilization (100%) status is 'Saturated'.
The storage system 'ausvsa1.xxxxx.local', in cluster 'AUS1', latency status is 'Normal'.

The ESXi server has plenty of CPU and memory headroom and the VSA is configured for 2000hz of CPU as required in the VSA Install and Configuration Guide.

The very same p2v job runs successfully to the local disc of the ESXi server. Anyone know what this error means?
5 REPLIES
Scott Smith_7
Occasional Contributor

Re: VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

Ben,

Is the uplink to your core switch a non-blocking port? I've seen oversubcription cause latency issues.

Regards,

Scott
Ben Drawbaugh
Occasional Advisor

Re: VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

I thought of this and put the VSA and VM_kernel port on the same virtual switch so that ESX<>SAN traffic avoided my core switch all together. In fact the only reason why I have an uplink port is so I can replicate a volume to another VSA (and management). I can try removing the uplink port completely if you think that will make a difference. The switch is a Extreme Networks Black Diamond that I don't think very much of.

It is good to know that this is a network issue, I wasn't sure what latency it was talking about (network, disc, who knows).
Ben Drawbaugh
Occasional Advisor

Re: VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

I gave up and decided to punt. I re-installed ESXi 4.1 and then redeployed the VSA and the issue didn't persist. So I guess it was a VMWare issue all along.

Re: VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

We had seen very similar symptoms (latency+CPU @ 100%) caused by vmkernel blocking on certain VMFS operations and causing the VSA vCPU to get stuck at 100% I/O-wait until the I/O timed out.

We converted all of our VSAs to use RDMs instead of VMFS-based VMDKs and haven't seen the problem since. :-)

Dan Nichols
Advisor

Re: VSA error EID_LATENCY_STATUS_EXCESSIVE E00060100

I have just experienced the same issue with VSA version 9.

I installed a new ESXi server 4.1 build 320137 on a HP DL365 G5 and put a version 9.0.00.3561 VSA on there. I created a 1.95TB VMDK and attached it to SCSI 1:0 on the VSA.

I created a 500GB thin volume on the VSA and presented it to the ESXi host via an internal only vSwitch.

I created a VM on the new VSA volume and whilst Windows 2008 R2 was installing the CMC logged the latency > 60ms error a few times and eventually the VSA declared the volume inoperable and the VM hung completely.

I was watching the disk latency values and the VSA lun was reporting latencies of 45 seconds+ whilst the physical lun that the VMDK was sat on was reporting maximum latencies of 20ms (average <5ms).

I have now deployed a VSA running 8.5 and the issue has vanished.