MSA Storage
1748166 Members
3945 Online
108758 Solutions
New Discussion юеВ

Re: P2000 slow since RAID reallocation

 
SOLVED
Go to solution
fukawi2
Occasional Advisor

P2000 slow since RAID reallocation

Hi all,

 

I have an iSCSI P2000 loaded with 12 x 300gb DP SAS drives.  This is connected to 2 DL385 servers with 64gb RAM running XenServer. The SAN network is multipathed via 2 gigabit switches (4 x copper links from the P2000 to each switch, then each XenServer has a copper link to each switch)

 

Recently I re-evaluated my install-time decision to create 2 vdisks: a RAID-5 and a RAID-10. I decided the RAID-5 was pointless and destroyed it, then used those disks to extend the RAID-10 into one big array, with 2 hot-spare disks.

 

Since that was done (just over a week ago) disk performance has been quite horrible; looking at `top` on both the XenServer hosts show ~15% I/O wait. hdparm inside a VM shoes around 1.12mbps reads from the SAN.

 

There are 2 other Win2008R2 machines connected to the SAN also. I have previous performance tests showing they too are experiencing a significant difference so I don't beleive it is a problem on the XenServer hosts.

8k random reads previously ~24.75MB/s, now 1.67MB/s

64k random reads previously ~170MB/s, now 3.61MB/s

 

The RAID extension has completed, everything is "OK" in the SMU. The XenServers are quite "empty" (7 VM's on one of them, 4 VM's on the other) but it is a production environment.

 

I'd love some input on what could be wrong and where to investigate further.

7 REPLIES 7
AnkitM
Trusted Contributor

Re: P2000 slow since RAID reallocation

When it comes to Read Performance, IMHO RAID5 provides equivalent Read performance compared to RAID10 in both random and sequential reads. Its the Write Performance where RAID10 gains edge over RAID5 since R5 has High Write Penalty.

 

Anyhow, from your detailed Problem Description I understand you must have gone through P2000 G3 Best Practices Guide: http://h20195.www2.hp.com/v2/GetPDF.aspx/4AA3-2141ENW.pdf It would be very difficult to check why the Read Performance is so degraded on RAID Change.

 

 

I would also verify the multipathing and check for the dead paths, if its as it should be.

 

I would request you to open a Case with HP Technical Support for Performance Review with latest store.log and Performance Logs attached for recommendations.

 

 

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
fukawi2
Occasional Advisor

Re: P2000 slow since RAID reallocation

Thanks for your reply AnkitM :)

 

All those performance figures were for the RAID-10; prior to and after the extension.

 

Yes, I have been through the Best Practices guide and followed the relevant sections as best we can.

 

I will look into creating a "store.log". Is there a specific team to deal with performance issues in the support dept?

AnkitM
Trusted Contributor

Re: P2000 slow since RAID reallocation

Performance Cases are worked by L1 and L2 Technical Storage Engineer(s) as a collaboration. If required, they may ask for more details about the environment.

 

For Eg. Cabling Diagram, Switch logs (if its not DAS) etc.

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
fukawi2
Occasional Advisor
Solution

Re: P2000 slow since RAID reallocation

I think I've found the problem; someone enabled flow control on both the switches. As soon as I disabled flow control, performance is back to ~35MB/s inside VM's, and I/O wait in top on the XenServer hosts is back down and load average is reducing already.

 

I will monitor for 24 hours and and confirm if this is indeed the issue.

AnkitM
Trusted Contributor

Re: P2000 slow since RAID reallocation

Awesome! Thanks for sharing that with us! :)
Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
fukawi2
Occasional Advisor

Re: P2000 slow since RAID reallocation

After 24 hours, this definitely seems like the issue. I/O wait is back to 0.0%, Load Average is back to normal, and disk I/O is also back to acceptable levels. Thanks for your input :)
AnkitM
Trusted Contributor

Re: P2000 slow since RAID reallocation

No Problem. Happy to help! :)

 

If your issue is answered, please remember to mark your question Answered when you get the correct answer or helpful answer and award kudos to the person providing the answer. This helps others searching for a similar issue.

 

Ankit

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.