EVA4400 performance issues when restoring a backup

[Xevious]

Hi all,

we are running an ESX cluster on an EVA 4400. Virtual machine snapshots are backed up with VMware Data Recovery. The dedup store of the Data Recovery virtual appliance again is written to a tape library with a TSM client that is installed on the VDR. I wanted to test the backup by writing back the dedup store (about 300 GB) to an empty 500 GB FATA LUN. First everything runs fine, but after a minute or two, the performance of the whole SAN collapses. Even VMs on completely different datastores and a different disk group (FC) are affected so much that they don't respond any more. The performance on the Data Recovery VM itself collapses aswell (from ~200 CMDS/s to ~25 CMDS/s). It somehow looks like first the controller cache is filled and then the whole thing almost comes to a halt. I tested the restore process outside the business hours where the average workload shoulnd't be too high - same thing. Does anyone have an idea how to address this issues?

[Uwe Zessin]

Replace the FATAl disk drives with serious "FC disks".

[Xevious]

The FC disk group is affected by the performance collapse aswell (athough nothing is written to them). Any other suggestion maybe?

[Uwe Zessin]

Yes, I understand that - it is caused by a firmware design problem.
Other suggestion would be to put the FATAl disk drives into a second EVA or use any other storage -- just don't mix FC and FATAl disks in one box if the latter group is hammered with I/Os.

[Víctor Cespón · Accepted solution]

Disable write cache on the FATA vdisks.

Set write cache = Writethrough.

And make sure the FATA disks have firmware HP06. They can became very slow with previous firmwares.

[Xevious]

vcespon: Excellent. Disabling the cache did the trick.

@Uwe: Thanks for your advices aswell.

[Tom Lanyon]

vcespon or Uwe - you obviously seem to be aware of a known problem in this regard, are you able to explain further what causes this issue and how disabling the write cache helps to fix the issue?

Is there simply too little write cache and it's getting exhausted? I can't understand how heavy writes to one disk group can be impacting another disk group so poorly.

Tom