HP MSA Crash When Adding New Disks

[Cossy]

Hi Gents,

I have a HP MSA 1000 disk array with 9 x 146GB SCSi drives configured as a single LUN using RAID 5. The drive in bay 10 is configured as a spare.

We recently decided to populate the remaining 4 spare bays(11-14) with a further 4 x 300GB disks in what was going to be another RAID 5 configuration to yield a further 900GB of disk space.

I assumed that this would be a zero risk activity as the unit should be hot swappable.

I added the disks one after another and the whole thing went to pot.

Drives 8, 9 & 10 went bad and the LUN that was supporting several production VMWare based servers stopped responding.

We rebooted the disk array and after entering the 'accept' cli command the LUN rebuit parity and eventually came up.

One of the two redundant array controllers went bad and is still down. Not sure what happened to it.

So, does anyone have any ideas as to what on earth happened?

Is it ok to 'hot' add disks to the spare bays in the array? Should this have worked?

Where do I go from here? The disk array is under maintenance so I can get the RAID conntroller replaced but how do I populate the remaining disks?

Should you always switch an MSA off when changing the configuration?

Appreciate any help/advice you can give.

Cossy

[Johan Guldmyr · Accepted solution]

Ouch!

What firmware are you running on it? Any later out there with fixes to something like this?
Are there 14 in this? So that exactly half (the right one) of the disks whent bad? Maybe there is an issue with the backplane..

A show tech_support from the cli (in the front) would be interesting.

[gregersenj]

There's allways a risk.

It should work perfectly well. And what went wrong can be a theory only.
Possibly, your existing hasn't been properly seatet.
A common mistake is, when you install the drives, is to seat it closing the leaver.
That won't do. Once the leaver has been closed, push hard on the drive (I usely put my thumb on the LED cover, and push hard).
If the drives havn't been thoroughly seatet, and if the backplane has been movng slightly inwards, when you install new disks, that might have caursed the disk to loose connection.
Or one of the new drives might made a little noise on the bus.
Just theory from my side, based on my experience.

Beware, theres never a zero risk, nor a 100% uptime.

BR
/jag

[Cossy]

Here's the firmware details. Please let me know if you think its out of date. It seems consistent with the age of the kit.

CLI-1> show version
MSA1000 Firmware Revision:
7.10b1200 (SGM0734AL0)

Build Time:
2008-06-04 08:37:52
MSA1000 Hardware Revision:
a [AutoRev: 0x030000]
Fibre Module AutoRev:
0x010000

Box 1a PROLIANT 4L7I DB Rev: 1.86 (SGM0734AL0)
Box 1b PROLIANT 4L7I DT Rev: 1.86 (SGM0734AL0


Cossy

[Cossy]

There are 14 bays. I have 10 populated and four spares.

I simply inserted four disks andthat was it.

Cossy

[Joshua Small_2]

Hi,

Do you have automatic disk expansion turned on?

[Cossy]

Not sure where 'Automatic Disk Expansion' is configured.

I would add though that we have two other HP MSA's both running at the same firmware and one of them is running with 300GB disks and the other is running with all 146GB disks.

The only difference between those arrays and the one that I had issues with is that I inserted 300GB disks in to an array that had 10 146GB disks.

Not sure if mixing the disks threw it in to a spin.

Thanks for your feedback.

Cossy

[Johan Guldmyr]

7.10 should be all right.
7.20 doesn't have many fixes in it.

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=12169&prodSeriesId=377751&swItem=MTX-a8be59eac6d84c1eb3cfcef835&prodNameId=305215&swEnvOID=1060&swLang=8&taskId=135&mode=4&idx=2

[Cossy]

This problem turned out to be related to a bad disk.

The supplier took the disk offsite, checked it and experienced similar issues there too.

They replaced the drive and it went in no problem.

Cossy