Storage Boards Cleanup
To make it easier to find information about HPE Storage products and solutions, we are doing spring cleaning. This includes consolidation of some older boards, and a simpler structure that more accurately reflects how people use HPE Storage.
Disk Arrays
cancel
Showing results for 
Search instead for 
Did you mean: 

EVA 6000 slow performance after upgrades

SOLVED
Go to solution
Peter Turek
Frequent Advisor

EVA 6000 slow performance after upgrades

I'm experiencing slow performance on large sequential reads and writes to the same SAN volume on a win2003 host. Immediately after the upgrade to XCS 6110 on this EVA 6000, this one host is showing a very significant drop in performance when doing a large (dozens of GB)copy of a single file in win2003 using a "cp" in cygwin (dont ask). It's literally gone from a max of 2.5hrs to over 12 hrs. The LUN is unfortunately on a fairly active FATA disk group with 36 drives in it.

XCS went to 6110, disk firmware is a mix of HP02 and HP03 on the affected FATA drive group. QLogic 32port switch firmware was also upgraded to 5.3.0d. Win2003 host runs HP FC2143 HBA with firmware 272A2 and driver 5-120A7 dated 2/17/05 MPIO v2.01.

I've tried a file copy test with a 2.2GB file from the server's C: drive (local pair of SCSI in RAID1) to the SAN drive (F:) and oddly it takes 30secs to copy from C to F but 4.5 minutes to copy from F to C.

I tried the same thing on another win2003 host using the same disk group with the same 2.2gb file it and takes about the same time.

For comparison another win2003 host with an MSA20 w/5 x SATA drives takes 15 seconds up and down from C: to array.

I completely formatted and restored all of the data on the SAN drive thinking it was fragmented, etc. But it didn't help and the evidence strongly suggests either a SAN problem or something went nuts on the host.

Tomorrow I'm attaching a local array with 18x146GB SAS 15k drives to test and see if it's really just the SAN.

Anyone ever have this happen to them?

TIA

Pete
7 REPLIES
Uwe Zessin
Honored Contributor

Re: EVA 6000 slow performance after upgrades

Sounds like an XCS firmware problem. On some arrays, the cache/readahead handling does not work after the upgrade. Log a call with HP - they are aware of this.
.
Ivan Ferreira
Honored Contributor

Re: EVA 6000 slow performance after upgrades

Have you tried changing the Controller preference for the LUN? What is your multipath software?
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
Peter Turek
Frequent Advisor

Re: EVA 6000 slow performance after upgrades

Thanks for the advice gents. I've opened a call with HP and they had us run the usual drill of sending in all the logs etc. Hopefully they come back with something.
Peter Mattei
Honored Contributor
Solution

Re: EVA 6000 slow performance after upgrades

There is a known bug with XCS6.110 that very rarely occurs. It does disable read caching for some LUNs.
It can be fixed with a controller resync.

However, it is always wise to open a call at HP and verify upfront if it is really this bug.

Cheers
Peter
I love storage
Uwe Zessin
Honored Contributor

Re: EVA 6000 slow performance after upgrades

> It can be fixed with a controller resync.

Unfortunately, there are some rare cases where this does not help...
.
Daniel Keisling_2
Occasional Advisor

Re: EVA 6000 slow performance after upgrades


I had the exact same thing happen to me on writing large amounts of data to FATA disks on my EVA8000. After many evaperf collections and talking to a higher level of HP support, it was determined that writing to the slower FATA disks basically filled up the entire write cache, affecting performance for other servers (regardless of disk group). The solution was to a) stop or throttle data being written to the FATA disks or b) change the write cache method (on the VDisk) to "Write-through" instead of "Write-back."

We chose option b, as well as trying to move heavy writes off of FATA (for example, DB backups are now done via snapshots). Going to write-through will drastically decrease your write speed (3x or more), so be careful.

Daniel
Peter Turek
Frequent Advisor

Re: EVA 6000 slow performance after upgrades

Thanks for all your help, particularly Uwe and Peter M. We did hear back from HP and they confirmed it is a firmware issue with no fix at the moment. The workaround where we reboot & resync the controllers will likely be done tomorrow.

I also appreciate your comment Dan K re the FATA drives using up all of the read cache. I don't think that's our problem as yet but given what you said, I'd be willing to bet it's impacting our 15k drive LUN performance. Not good.

Great work and comments all! Thanks again!