Very High Write Latencies in EVAperf? Indicative of What?

[Alzhy]

And slow to occasional hiccups on a somewhat huge vMware ESX (14 ESX 3.02 Proliant DL580s) + a sprinkling of HP-UX.

Latencies (Wtite) reaching/bursting over 1000ms even 5000+ ms.! And FP Queue Depths of more than 20 to over 100!

Could this be indicative of our VRAID layouts? All are vRAID5 (Mix of FATA and FC).


Any inputs will be appreicated.


EVA8000, XCS 6100.
2Gbit FrontEnds x 8.

[Tom O'Toole]

What do mirror port usages look like?

[Alzhy]

How do I Check man?

[Jonathan Harris_3]

Mirror port throughput can be measured through evaperf vdg

Generally, there should be a low correlation between write throughput and mirror port throughput. If it's high, that would indicate badly aligned or random VRAID5 writes.

Generally, this is more an issue for the older EVA GLs (3000s and 5000s) than the EVA XLs (4/6/8000). Because the EVA GLs operated active / passive, any I/O for a vdisk that arrives on the controller not controlling the destination vdisk has to pass across the mirror port. With active / active this is less of an issue, but it's still worth looking at.

Anyway, let's chuck a few other things in to the mix...

What are you using your FATA disks for? FATA is fine for sequential I/O in non-critical areas. Use it for random I/O and you'll find that stuff will grind to a halt. I'm also assuming that you're not mixing and matching in your DGs.

What's the general throughput on the FPs like during periods of poor performance?

Have you compared throughput on your switch ports to try and correlate periods of poor performance with high bandwidth utilisation from one or more servers? Do the hic-cups occur at a time when somebody's running an intensive batch job?

Are you running CA? If so, what sort of bandwidth have you got to the partner?

How is the EVA carved up? How many disk groups? How many disks per group? Which vdisks are in which group?

What applications are running (eg, databases, file and print, etc)? How are these distributed between disks and disk groups? How many virtual machines are running on the ESX?

The figures you're touting are really bad for an 8000. Unfortunately, diagnosing poor performance on an EVA can be incredibly difficult, not least because HP are convinced that you don't need to monitor performance in an EVA (it's all supposed to take care of itself). However, hopefully, we can start narrowing down the possibilities and point you at the source of your problem.

[Uwe Zessin]

> Because the EVA GLs operated active / passive, any I/O for a vdisk that arrives on the controller not controlling the destination vdisk has to pass across the mirror port.

That is not what I have been taught: a read or write I/O to a SCSI LUN on a non-owning controller is rejected.

> With active / active this is less of an issue, but it's still worth looking at.

It can be an issue for the GLs with A/A firmware, because they have only one mirror port. So read I/Os through non-optimized paths go over this port as well as all write I/O (either non-owned or cache mirror traffic).

[Alzhy]

We are still experiencing these unbelievable performance out of our EVA8000 and our vMware environments.

The Diskgroup where the vMware VRAI5 luns are carved from are about 96x300GB 10K rpm FC Disks.

The hiccups and vMware "pauses" (3-10 seconds) coincide with the EVA8000 stats showing these host port latencies which sometimes approach up to 13000 ms!

When we first moved the vMware environments to this EVA, there was some bit of a miscongiguration on the vMware side as well as onm the EVA host mode side. The vMware side was set to MRU indtead of FIXED (with distibution of the paths). On the EVA side, the host mode was "custom" (for A/P carried over from an EVA5K) instead of "vMware". Those misconfigurations have already been fixed BUT these hiccups/pauses on the vMware Guests (all Windows XP Professional) linger on with the EVa8K stats still continuing to show latencies whenever these hiccups/pauses occur.

EVA Controller Stats never exceed 25% for both data and cpu. The UNIX "CLients of these EVA" seem to be unaffected. The UNIX clients use the same diskgroup as the vMware as well as another FATA DG.

We're taken a few stats on the ESX hosts and indeed the vMware LUNS show latencies as well. Q Depth settings on the ESX host is set at 64. The typical vmware ESX LUN is sized 300GB each with 10-15 vMware Guests.

Any other ideas?


Currently I am testing one Virtual Machine to be hosted on a VRAID1 LUN ...

[Alzhy]

Update:

My performance is significvantly better on this vRAID1 LUN. There still appears to be hiccups and pauses and the FP latencies still show..

I also started gathering LUN Level stats via "evaperf vd -fvd". All the VRAID5 LUNS have consistent write latencies ranging from 1 to 23 seconds! Whilst our lone VRAID1 LUN also had latencies in seconds but to a lesser recurrence..

Do you think now the issue is possibly with the XCS 6100 firmware on this EVA 8000?

We are about to engage vMware and HP...

[Amar_Joshi]

I strongly believe that data layout on VDisk is not optimized or disks are heavily used for random IOs. High number in QueueDepth and high write latency is a great combination to conclude this. But, you may wanna see and match the write latency from the Host versus write latency on the EVA, if you see the same numbers it's EVA cache which is not performing because host should have low write latency due to its writing onto the cache.

Great idea would be to check and compare random-read, random-write, sequential-read & sequential-write IOPS from individual host. If your random numbers are high, you may want to redesign the VDisk layout, such as create a separate DiskGroup and then check the performance.

My 2 cents worth...

[Theta Wizard]

Did you ever resolve the 3-10 second pause issue using VMWare? Currently experiencing the same issue on our network.

[Jeff Lawrence]

FYI - i had this issue with vpshere and an EVA with read/write latency spikes and found that I had a slow GBIC in one of my EVA controller ports - changed that out to the appropriate speed GBIC and the spikes went away. I had a 2GB GBIC and replaced it with a 4GB GBIC