Replacing a failed disk HP P2000 SAN - Invalid Metadata

[tomstephens89]

On Friday I had an absolute disaster. I went to the datacenter in order to rack a new server and at the same time, took a new disk with me to replace a drive that had failed a couple of weeks back in a HP P2000 G3 10Gbit iSCSI array. The P2000 is loaded with 12 x 2TB 7.2k MDL SAS disks and configured as 2 RAID10 arrays, each with 4 drives + 2 hot spares. I had already removed the failed disk a week earlier, its place in the array had been taken automatically by one of the hot spares as expected.

So, the task in hand was a simple drive replacement to give me back a hot spare. Simple right? Slide in the disk, make sure its visible in the GUI and configure as a hot spare....

No...

I opened the brand new disk from its packaging and slid it into place, instantly all the drives in the array went orange. I checked the GUI and every single disk was showing 'Invalid metadata', a quick check of running services using the two arrays confirmed that everything had lost visibility of the LUN's. I rescanned the channels, rebooted the controllers all to no effect. Drives started disappearing from the GUI and my VD's (LUN's) were now missing from the GUI as well. I removed the new disk as well, still nothing.

In an act of desperation and confusion I pulled the power to the P2000 and let it fully reboot. It came back online and I could see my VD's again... However both arrays had lost all redundancy, like each half of the 2 RAID10's had lost their mirror disks.

All the other disks that were once part of the array were now showing as available. I was able to configure them as hot spares and the 2 RAID10's began reconstruction. It is now running again, all be it without my new disk since I am too scared to put it in again.

Does anyone have a clue about what happened here?

The only thing I can think of is that the new disk must have contained metadata of its own (maybe a refurbed disk) and confused the P2000. However it was a new sealed disk from our usual supplier. However even if this was the case, I wouldn't expect the array to do anything with that disk that effects the existing RAID configuration!

Help please!

[WarNox]

Not sure what happend in your case but I had all disks on an MSA1000 go orange when I assigned an available disk as Global Hot Spare. Although hosts did not lose access to the LUNs and everything went back to normal a few minutes later, long enough for panic to set in anyway.