Smart Array 6i controller - Drive failure

[JoshIT]

Hi, I have a 'replace SMART drive' alert on one of drives on my Proliant DL 380 Server. There are three drives in a RAID 5 setup (distributed data guarding). The OS is windows 2003 and the server looks fine, for now.

How should I go about replacing this drive? Should I buy a new drive (exactly same size as the current ones), plug it it and let it rebuild? Each drive is around 72 GB. Do I need to do any configurations through the array configuration utility? How much time does it take to rebuild?
And I assume I can replace the drive while the OS is running?

Sorry for the basic questions, but I am more of a network person rather than a systems person and this has been kind of forced on me.

Thanks very much for your suggestons!

[Víctor Cespón]

Hi SMART is a reporting mechaninsm that warns when a drive is not performing right. The disks on the DL380 (G3 I asssume since it has a 6i), are hot-plug but cannot be removed until they have failed.
You can wait till the disk fails completly (amber LED steady ON), and replace it while the OS is running. Or you can shutdown the server and replace it.
The rebuild process will be launched by the controller and will take a few hours, maybe one day if the server has heavy disk usage during that period.

[JoshIT]

Thanks very much for your reply! I have a few more questions.

I thought the whole purpose of having SMART was to be able to replace faulty drives before they completely fail. Why is it different for the DL 380? I mean, why do I have to wait for the drives to fail in order to replace it? Incidentally, I have a DL 380 G4.

Also, since it's RAID 5 with just 3 drives and if one fails, the RAID won't even work right? I mean, since RAID 5 needs at least 3 drives. In this case, I guess the only way to replace is the drives is by switching off the server?

Thanks, once again!

[gregersenj]

That's not true vcespon.

S.M.A.R.T. does monitor and test the drives continously. It monitor several thresholds. RPM, spinuptime, retries etc.
One or more of these thresholds has been exceeded.
The drive is in degraded state, and HP recommend you replace it.
There is many variations of degraded:
Threshold exceeded, predictive failure, imminent failure etc.
The drive may fail within a short time, or it may keep on running for years.

If the server is under warranty or service contract, HP will replace you drive.

If you have to pay yourself, you may consider to wait.

Replacement:
You must replace it with a drive of minimum the same size.
It must be the same carrier.

Since the drive is degraded, you must re-check that you are running a redundent RAID.
And the Dl380 is hot swap.
So You simply remove the bad disk, and insert the new.
Rebuild will start automaticly.

Rebuild will take from 25 min to a few hours, depending on the speed of the drives, RAID level, If you got write cache and how busy it is.

I allways - allways - Recommend to ensure, that you have got a good backup, and disaster recovery plan. Even with a Smart Array controller.

Also you could try updating fw on the drives and the array controller.

BR
/jag

[JoshIT]

Thank you for your reply.

So I guess I can just take out the problem/faulty hard drive while the OS (windows 2003 server) is running, put in the new drive, and just wait for the rebuilding process to complete.I hope I don't have to do anything from the array configuration utility. The server will be in use all this time, though it will be as minimal as possible. As for the replacement drive, I will get a matching size with a better rpm (only because the same rpm does not seem to be readily available). Hope these steps will set it right.

Thanks.

[gregersenj · Accepted solution]

Yes :)

If you have followed my recommended precaurtions!
Then you just yank the bad drive out, and put the new drive in.

The controller will start the rebuild automaticly.
If you look at the LED's.
Right now you shuold have a red Failure LED flashing
When you insert the new drive, it has to spin up, then the controller will start the rebuild, appr. 10 sec's after insertion.
The middle LED (on-line LED) will start flashing, wich mean rebuild in progress. When the On-line LED is solid on, rebuild has finished.

You can monitor the progress, using the ACU og the insight manager server home page, if the IM agents is installed and running.
The agents will also make an entry in the windows systems event log.
Actually you will get 4 or 5 events telling you that you have removed the drive, insretet a drive, that rebuild has begun and finished

[JoshIT]

Thank you, I followed your suggestions, pulled out the faulty drive while the OS is running and put the new one in.Everything is okay now, however I am noting a few things here which might help someone else doing the same thing.

After I put in the new drive, the status on the logical drive (via the HP software)said 'ready for rebuild' and it just sat there at this same status for the next 12 or so hours. All this while the 'sectors written' on this new drive was increasing steadily and nothing else. I had an old version of the array controller firmware, the one that came with the machine I guess!. I was looking at updating it, rebooting the machine etc when it suddenly started to rebuild the array. Once it did that it only took slightly more than an hour for it complete the rebuild. It could have been a coincidence, but it went from the 'ready to rebuild' to the 'rebuilding' state when I opened up the array configuration utility to have a look at the controller. Earlier I was just using the system management home page to check out the drive/controllers.

Thanks, folks and sure hope this helps someone.

[gregersenj]

You're welcome.

That's a good input from you.
It shouldn't go into waiting for rebuild. It should begin within a few seconds.
And I have never seen that problem.
If you ever have to replace a drive again, and if you meet the problem again, you can try opening the ACU, and look for a possibility, to force a rebuild.

I do recommend to upgrade the firmware, at your earliest convenience.

cheers
/jag