P2000 slow since RAID reallocation

[fukawi2]

Hi all,

I have an iSCSI P2000 loaded with 12 x 300gb DP SAS drives.  This is connected to 2 DL385 servers with 64gb RAM running XenServer. The SAN network is multipathed via 2 gigabit switches (4 x copper links from the P2000 to each switch, then each XenServer has a copper link to each switch)

Recently I re-evaluated my install-time decision to create 2 vdisks: a RAID-5 and a RAID-10. I decided the RAID-5 was pointless and destroyed it, then used those disks to extend the RAID-10 into one big array, with 2 hot-spare disks.

Since that was done (just over a week ago) disk performance has been quite horrible; looking at `top` on both the XenServer hosts show ~15% I/O wait. hdparm inside a VM shoes around 1.12mbps reads from the SAN.

There are 2 other Win2008R2 machines connected to the SAN also. I have previous performance tests showing they too are experiencing a significant difference so I don't beleive it is a problem on the XenServer hosts.

8k random reads previously ~24.75MB/s, now 1.67MB/s

64k random reads previously ~170MB/s, now 3.61MB/s

The RAID extension has completed, everything is "OK" in the SMU. The XenServers are quite "empty" (7 VM's on one of them, 4 VM's on the other) but it is a production environment.

I'd love some input on what could be wrong and where to investigate further.

[AnkitM]

When it comes to Read Performance, IMHO RAID5 provides equivalent Read performance compared to RAID10 in both random and sequential reads. Its the Write Performance where RAID10 gains edge over RAID5 since R5 has High Write Penalty.

Anyhow, from your detailed Problem Description I understand you must have gone through P2000 G3 Best Practices Guide: http://h20195.www2.hp.com/v2/GetPDF.aspx/4AA3-2141ENW.pdf It would be very difficult to check why the Read Performance is so degraded on RAID Change.

I would also verify the multipathing and check for the dead paths, if its as it should be.

I would request you to open a Case with HP Technical Support for Performance Review with latest store.log and Performance Logs attached for recommendations.

[fukawi2]

Thanks for your reply AnkitM :)

All those performance figures were for the RAID-10; prior to and after the extension.

Yes, I have been through the Best Practices guide and followed the relevant sections as best we can.

I will look into creating a "store.log". Is there a specific team to deal with performance issues in the support dept?

[AnkitM]

Performance Cases are worked by L1 and L2 Technical Storage Engineer(s) as a collaboration. If required, they may ask for more details about the environment.

For Eg. Cabling Diagram, Switch logs (if its not DAS) etc.

[fukawi2 · Accepted solution]

I think I've found the problem; someone enabled flow control on both the switches. As soon as I disabled flow control, performance is back to ~35MB/s inside VM's, and I/O wait in top on the XenServer hosts is back down and load average is reducing already.

I will monitor for 24 hours and and confirm if this is indeed the issue.

[AnkitM]

Awesome! Thanks for sharing that with us! :)

[fukawi2]

After 24 hours, this definitely seems like the issue. I/O wait is back to 0.0%, Load Average is back to normal, and disk I/O is also back to acceptable levels. Thanks for your input :)

[AnkitM]

No Problem. Happy to help! :)

If your issue is answered, please remember to mark your question Answered when you get the correct answer or helpful answer and award kudos to the person providing the answer. This helps others searching for a similar issue.

Ankit