C7000, BL460c G6, MDS600, P700m

[Jane W.]

Recently provided with a "used" C7000 freshly populated with 3 BL460c G6 blades and 7 old BL460c G1 blades. Two of the G6 blades have P700m controllers to interface with a new MDS600. Initially one G6 blade used 5 drives (RAID 5) in one drawer and the other G6 used 10 drives from that same drawer. All along during POST both servers have had a message about Port 51 and the Expander detecting an error. Last week I backed up the 3 file systems stored on the 5-disk RAID 5 --- rsync kept stopping saying there were errors but eventually I got it to finish. I redid the RAID 5 as 10 disk RAID 6 with 5 drives added in the 2nd drawer of the MDS600. This mimics a configuration being used elsewhere. I was able to restore the three file systems - again the rsync stopped a few times but eventually all 1.2 TBytes were restored. Howerever, two of the new disks in the MDS600 ended up marked as failed. I had one spare so pulled the two failed drives out and put the spare in for the first one that failed. Rebuild started and got somewhere past 8% but then stopped; the /var/log/messages on the blade had a message "Rebuild abort - write err, LDrv=0 Port=51 Box=1 Bay=2" and that is the drive bay where the new disk had been inserted. This is with RHEL 5.4 64-bit; P700m firmware is up-to-date as is the MDS600. When I have had problems with replacement drives marked as failed in a DL380 it has sometimes turned out to be necessary to replace the part that the drives plug into. I am not sure if something similar may be necessary with this Blade/MDS600 configuration - it is my first experience with blades. Suggestions on how to proceed with diagnosis would be appreciated.

[gregersenj]

Blades is not much different from the traditional servers.

You put the servers (Blades) in a box along with some of the infrastructure.

I will recommend you to spent a day or 2 on studing the MDS and the SAS switches.

If you have got experience on zoning SAN, it will be easy to understand.

Blade / Enclosure / Infra structure.
I will recommend that you get every thing upgraded to comply with FW releases set 2010.10 (FW DVD 9.20)
http://h18004.www1.hp.com/products/blades/components/c-class.html#tab3_content
Choose the compatbility tap, and scroll down.
Upgrade the SAS switces and the MDS incl disks to latest FW.
Check that the cabling is correct.

Once thats all ok.
FW
Cabling.
reseat the drives.
Check the drives, using insight diagnostics:
Boot on a smart start cd, choose maintenance tab, then open insight diagnostics.
If you still got suspect a bad drawer:
Move the drives to new posistions, even to the other drawer. Create a new zone and try again.

It could be a bad source disk, read errors on one of the source disks will caurse the rebuild to fail.

If you have the System Management Homepage installed, that will make life easier.
From the SMH you can check the statisics of all the drives.

It could also be the drawer, just like on a traditional server. But it is not easy to replace, I have done that. And you need to power down the entire MDS.

I will try to re-find the links for the documentation that i read.

BR
/jag

[gregersenj]

Some links to study.

SAS Switch documentation:
http://h20000.www2.hp.com/bizsupport/TechSupport/DocumentIndex.jsp?lang=en&cc=us&contentType=SupportManual&prodTypeId=3709945&prodSeriesId=3832737&docIndexId=64180

MDS600 documentation:
http://h20000.www2.hp.com/bizsupport/TechSupport/DocumentIndex.jsp?contentType=SupportManual〈=en&cc=us&docIndexId=64179&taskId=130&prodTypeId=12169&prodSeriesId=3936271

MDS FW 2.66 / release notes:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=dk&prodTypeId=12169&prodSeriesId=3936271&swItem=MTX-7b9bdbef960a47679911f6a39f&prodNameId=3936272&swEnvOID=4048&swLang=8&taskId=135&mode=4&idx=1

SAS switch FW 2.2.4.0 / release notes:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=dk&prodTypeId=3709945&prodSeriesId=3832737&swItem=MTX-11994046e4e14df8913101560d&prodNameId=3832738&swEnvOID=2065&swLang=8&taskId=135&mode=4&idx=2

BR
/jag

[Jane W.]

The firmware is up-to-date. I will be back in the office next week and will pursue some of your other suggestions. Thank you.

[Jane W.]

I had an unexpected absence last week but got back to this today. As of today - replacing the replacement drive that was marked as failed seems to be working much better; the rebuild has reached over 45% after a number of hours. I will know tomorrow if it finished. Who knows - maybe the rebuild of the replacement for the 2nd of the orignally-marked-as-failed disks will finish. I work in an environment where it is not so easy to bring in new firmware revisions but am working on the approval for the latest disk drive version as one of the replacement drives has HPD3 whereas all the others have HPD2.

[Jane W.]

It seems like the new disks and/or the combination of power cycles have resolved the issues.

[gregersenj]

Great.

Reseat / power cycle, some times do magic.

Br
/jag