Two Drives with Predictive Failures on ML350 G4 with Array 641

[Nicholas Martinez]

I have two drives with predictive failures in an array of 4 total in Raid 5.

I have replaced one of the drives with a new one. The array begins to rebuild the logical drives but then fails saying it cannot rebuild due to errors detected on the 2nd failing drive.

I fear that if I replace the 2nd drive.. it cause total failure on the raid and array.

Can someone PLEASE help and shed some light on this.. I do not want to pull that 2nd drive out and lose the server. It acts as our domain controller AND our exchange server!

[Michael A. McKenney]

Upgrade the firmware on the drives, controllers and server from ISO DVD. Make sure you do the server first, then controller, then get the drive firmware. It could fix the problem.

[Nicholas Martinez]

sorry to be such a novice.. but i see many firmware options on the software page for the server.. could you point out which specifcially i need on this page? http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=3279707&prodTypeId=15351&prodSeriesId=397642&swLang=8&taskId=135&swEnvOID=1005#29214

[Michael A. McKenney]

» Go to top

Firmware - CD-ROM Description Current version Size (MB) Estimated download time Previous version
Firmware Maintenance CD 8.50
8 May 2009 410 56K: >8h
512K: 1h 8.40
10 Feb 2009

You burn it to an ISO. It will handle most of the server. You still need to do the drives.

Review the revision history. Don't jump more than 2 fimrware revisions at a time. It goes back 8-10 revisions.

Call HP about getting your firmware on the drives.

Then, upgrade all the drivers.

[Nicholas Martinez]

Ok thanks!.. Ive updated the array and bios firmware and then pulled the replacement drive and placed it back in.. and it started the rebuild process but soon quit thereafter referencing the errors on the other failing drive again.
I am attaching the ACU diagonostics log to see if anyone sees something that may help..

[Michael A. McKenney]

Did you find the firmware for all your drives and install them?

[Michael A. McKenney]

Did you do a full backup? You should do one daily.

[Nicholas Martinez]

I have updated all the firmware. Machine rebooted and immediately started rebuilding.
The rebuilding of logical drive 1 failed due to errors on the 2nd predictive failure drive.
Rebuilding of logical drive 2 kicked in.. but that started last night around 7pm and it is now 3pm the next day and it has only reached 28%.. its only working at 1 percent an hour.. Is this normal? Im getting errors every min in Event Viewer about logical drive 1 having fatal read/write errors. Should i reboot the machine again? I have heard this shouldnt take longer than an hour with this many drives and 72.4 GB spare.. but i also dont want to kill the array by interrupting whats happening.. PLEASE ADVISE

[Michael A. McKenney]

Depends on the settings you have for rebuilding and size of drives. I set my rebuilding at 80% rebuild / 20% available to users. I want the array rebuilt at the expense of using the array.

I would not stop it. Let it finish. When done check your rebuild settings and adjust them for next time.

[Nicholas Martinez]

I called HP and they told me to reboot the server.. which was a big mistake.. it started rebuilding from the beginning.. but now the rebuilding has failed and both my logical drives are back to Ready to Recover status.. and my server is painfully slow and I can barely log into it and email isnt working.. Any suggestions to at least get it working to where it was before i started this? This is very nerve racking

[Michael A. McKenney]

You could check the rebuild % in the ACU and set it higher. Rebooting will cause a restart. Two failing drives is not good. RAID 5 will collaspe. Did you backup like I suggested?

You should never have a domain controller and Exchange server together. Exchange should be a member server. You should have two DC servers. Do you have a spare box you can spin up and make a DC server. Even a workstation that can install as a server.

OS should be RAID 1 or 10 only. Never RAID 5.

[Nicholas Martinez]

Thanks for your assistance. I had to call in a consultant company. We had to basically reinstall Windows Server 2003 and Exchange 2003 on the machine from scratch and restore exchange database from a two week old backup because wouldnt you know it.. my external drive i keep backups on has completely failed too.
I have not made the machine a domain controller again and i am using my backup domain controller for now.

The consultants removed the last remaining failing drive from the array and never put the replacement back in. I have done this, this morning and the logical drives are currently rebuilding.

[Michael A. McKenney]

I would have replaced every drive. I would also look into a new DC server. I hope you did not reinstall the OS on RAID 5. It should be on RAID 1 or 10. If RAID 5, has issues, your OS could become corrupt. I don't even use RAID 5. I only use RAID 1 and 10. Less problems. My performance is very good on RAID 10. Exchange does alot of writes so RAID 10 is faster. No parity.