EVA 6000 slow performance after upgrades

[Peter Turek]

I'm experiencing slow performance on large sequential reads and writes to the same SAN volume on a win2003 host. Immediately after the upgrade to XCS 6110 on this EVA 6000, this one host is showing a very significant drop in performance when doing a large (dozens of GB)copy of a single file in win2003 using a "cp" in cygwin (dont ask). It's literally gone from a max of 2.5hrs to over 12 hrs. The LUN is unfortunately on a fairly active FATA disk group with 36 drives in it.

XCS went to 6110, disk firmware is a mix of HP02 and HP03 on the affected FATA drive group. QLogic 32port switch firmware was also upgraded to 5.3.0d. Win2003 host runs HP FC2143 HBA with firmware 272A2 and driver 5-120A7 dated 2/17/05 MPIO v2.01.

I've tried a file copy test with a 2.2GB file from the server's C: drive (local pair of SCSI in RAID1) to the SAN drive (F:) and oddly it takes 30secs to copy from C to F but 4.5 minutes to copy from F to C.

I tried the same thing on another win2003 host using the same disk group with the same 2.2gb file it and takes about the same time.

For comparison another win2003 host with an MSA20 w/5 x SATA drives takes 15 seconds up and down from C: to array.

I completely formatted and restored all of the data on the SAN drive thinking it was fragmented, etc. But it didn't help and the evidence strongly suggests either a SAN problem or something went nuts on the host.

Tomorrow I'm attaching a local array with 18x146GB SAS 15k drives to test and see if it's really just the SAN.

Anyone ever have this happen to them?

TIA

Pete

[Uwe Zessin]

Sounds like an XCS firmware problem. On some arrays, the cache/readahead handling does not work after the upgrade. Log a call with HP - they are aware of this.

[Ivan Ferreira]

Have you tried changing the Controller preference for the LUN? What is your multipath software?

[Peter Turek]

Thanks for the advice gents. I've opened a call with HP and they had us run the usual drill of sending in all the logs etc. Hopefully they come back with something.

[Peter Mattei · Accepted solution]

There is a known bug with XCS6.110 that very rarely occurs. It does disable read caching for some LUNs.
It can be fixed with a controller resync.

However, it is always wise to open a call at HP and verify upfront if it is really this bug.

Cheers
Peter

[Uwe Zessin]

> It can be fixed with a controller resync.

Unfortunately, there are some rare cases where this does not help...

[Daniel Keisling_2]

I had the exact same thing happen to me on writing large amounts of data to FATA disks on my EVA8000. After many evaperf collections and talking to a higher level of HP support, it was determined that writing to the slower FATA disks basically filled up the entire write cache, affecting performance for other servers (regardless of disk group). The solution was to a) stop or throttle data being written to the FATA disks or b) change the write cache method (on the VDisk) to "Write-through" instead of "Write-back."

We chose option b, as well as trying to move heavy writes off of FATA (for example, DB backups are now done via snapshots). Going to write-through will drastically decrease your write speed (3x or more), so be careful.

Daniel

[Peter Turek]

Thanks for all your help, particularly Uwe and Peter M. We did hear back from HP and they confirmed it is a firmware issue with no fix at the moment. The workaround where we reboot & resync the controllers will likely be done tomorrow.

I also appreciate your comment Dan K re the FATA drives using up all of the read cache. I don't think that's our problem as yet but given what you said, I'd be willing to bet it's impacting our 15k drive LUN performance. Not good.

Great work and comments all! Thanks again!