topic EVA4400 performance issues when restoring a backup in HPE EVA Storage

EVA4400 performance issues when restoring a backup

Xevious — Wed, 10 Mar 2010 12:05:04 GMT

Hi all,

we are running an ESX cluster on an EVA 4400. Virtual machine snapshots are backed up with VMware Data Recovery. The dedup store of the Data Recovery virtual appliance again is written to a tape library with a TSM client that is installed on the VDR. I wanted to test the backup by writing back the dedup store (about 300 GB) to an empty 500 GB FATA LUN. First everything runs fine, but after a minute or two, the performance of the whole SAN collapses. Even VMs on completely different datastores and a different disk group (FC) are affected so much that they don't respond any more. The performance on the Data Recovery VM itself collapses aswell (from ~200 CMDS/s to ~25 CMDS/s). It somehow looks like first the controller cache is filled and then the whole thing almost comes to a halt. I tested the restore process outside the business hours where the average workload shoulnd't be too high - same thing. Does anyone have an idea how to address this issues?

Re: EVA4400 performance issues when restoring a backup

Uwe Zessin — Wed, 10 Mar 2010 12:59:02 GMT

Replace the FATAl disk drives with serious "FC disks".

Re: EVA4400 performance issues when restoring a backup

Xevious — Wed, 10 Mar 2010 13:14:33 GMT

The FC disk group is affected by the performance collapse aswell (athough nothing is written to them). Any other suggestion maybe?

Re: EVA4400 performance issues when restoring a backup

Uwe Zessin — Wed, 10 Mar 2010 13:22:35 GMT

Yes, I understand that - it is caused by a firmware design problem.
Other suggestion would be to put the FATAl disk drives into a second EVA or use any other storage -- just don't mix FC and FATAl disks in one box if the latter group is hammered with I/Os.

Re: EVA4400 performance issues when restoring a backup

Víctor Cespón — Wed, 10 Mar 2010 13:29:31 GMT

Disable write cache on the FATA vdisks.

Set write cache = Writethrough.

And make sure the FATA disks have firmware HP06. They can became very slow with previous firmwares.

Re: EVA4400 performance issues when restoring a backup

Xevious — Wed, 10 Mar 2010 14:37:48 GMT

vcespon: Excellent. Disabling the cache did the trick.

@Uwe: Thanks for your advices aswell.

Re: EVA4400 performance issues when restoring a backup

Tom Lanyon — Mon, 14 Jun 2010 12:31:30 GMT

vcespon or Uwe - you obviously seem to be aware of a known problem in this regard, are you able to explain further what causes this issue and how disabling the write cache helps to fix the issue?

Is there simply too little write cache and it's getting exhausted? I can't understand how heavy writes to one disk group can be impacting another disk group so poorly.

Tom