Server Hang during Hot Swap Rebuild

[jenny_17]

Dear all,

I'm having a DL360 G3 Compaq Server, installing Win2000 Standard Server.
It has two identical harddisk, configured at RAID 1 (mirror), with Smart Array Controller 5i.

When I Hot Swap one of the harddisk, the server hangs. However, when I normal shutdown the Server, plug in that harddisk, the booting message required me to enter F1 or F2 (to automatically recover the data / not recover the data). After I choose automatically recover the data, the plugged harddisk then start rebuilding, and rebuild success.

To sum up, my server seems lost the capability of Hot Swap Function. I install the server by running Compaq SmartStart 4.6

Can anyone give me any hints ? either any firmware problem/BIOS problem/my installation procedure problem?

Remarks: it seems not hardware problem, as I got another DL360 G3 server delievered and start installed, the Hot Swap Function does not work also. Thanks.

[Terri Harris]

1. What exactly are you doing when you are "hot swapping" one hard drive? What is the purpose?

2. Have you ever done what you are trying to do successfully on any other Proliant? Other than the second DL360?

[jenny_17]

Hi, Thanks First.

"hot swapping" harddisk is:
Case 1:
The Server is boot to Windows 2000 Server with 2 harddisk (RAID 1). While the machine is running (only running the OS, no additional application program is running), I unplug one harddisk. This similar the case that one harddisk suddendly got hardware failure, but then my machine can be supposed to run by the remaining healthy harddisk.

Case 2:
The Server is boot to Windows 2000 Server with 1 harddisk only. While the machine is running normally, I plugin another harddisk.
I expect it can rebuild automatically, with priority (low) already preset in the CP array utility.

For Case 1, sometimes it hangs. 2 out of 4 times hang
For Case 2, i try 4 times, but it hangs 4 times.

When the server hangs, the login console cannot receive user interaction. It does not receive mouse click event but the mouse cursor can move freely. And I cannot connect to the Server by Terminal Session.

All I can do is to cold boot the computer by pressing the on/off button.

or the harddisk can re-build itself successfully by normal shutdown the Server when running one harddisk, then plug in the harddisk, answer the "with automatical data recovery" options at the boot up message,

The "Hot Swap" action (which is Case 1 & Case 2) is fully functioned for other type of RAID 1 Server ProLiant DL 360. (I have 7 DL 360 server purchased 1 year before, and it stills running normally till now, but is installed with a lower version of SmartStart 5.x)

Thanks for your help.

[jenny_17]

Sorry, I am here to correct a wrong imformation provided in the first message. I'm use Compaq SmartStart 6.4 to install the problemtic server (DL360 G3) instead of 4.6.

Thanks.

[Leon Rosier]

Hi,

The question is: What you are doing is that a good test?
I don't think so. Under normal operation the raid controller decides to put a disk in failed state, the fault led will go on. At this point the disk is not an active disk anymore and can be hot-removed. What you are doing is removing an active disk out of a raid set the hardway.
I know it works on other controllers, but does that mean it should work on all controllers? I wouldn't burn myself on that..
So my opinion is you are testing an non-realistic situation.

Leon

[jenny_17]

Sorry, I think it's a realistic case. And it works for other DL360 Servers.

[Doug Wogan]

Hey Jenny.
It seems to be a problem with the DL360G3. I've had the same problem. If you power down the server, pull one of the drives and power back on that should mimik a hdd failure. However when you plug the hdd back in, the server will freeze. It does it with W2k and W2k3. It's not a problem with the 5i controller because I've done the same test with a DL360G2 w/ a 5i controller and it does not lock the server up. I have called Compaq and the re-assured me if a hdd does fail you will be able to remove the dead drive and insert a new one w/out rebooting. I find this hard to believe. Did you have a hdd failure or are you testing?
Doug.

[jenny_17]

Hi, Doug
Thanks first. But I cannot understand the following message:
"If you power down the server, pull one of the drives and power back on that should mimik a hdd failure. "
or to be more accurate, I do not understand the meaning of "mimik". Sorry.

I can confirm these kinds of servers (DL360, DL360GX) supports what compaq said:
"if a hdd does fail you will be
able to remove the dead drive and insert a new one w/out rebooting."
and that's called "Hot Swap" function and is very useful in a critical system that cannot suffer Server Failure at any time. And I have face the harddisk failure problem before, but for the other server model DL360. When the harddisk failed, our system still running. And I have time to find another good harddisk, "hot plug" into the Server and it starts to rebuild automatically without reboot. With rebuild priority set to "low", it won't affect our system very much.

To ensure the Server can support this feature, I must do a testing before the Server go for production. Right now, the DL360 G3 freeze problem is a testing. Both of the harddisk are good, and while the Windows is running, I pull one of the harrdisk, after a while, push it back, then the Server freeze. I must solve this before the Server go for production.
I have perform the above test for other DL360 Servers before they go for production and the test work well. Now, the DL360 can run healthy and really can fault tolrence one harddisk failure.

Can anyone can confirm it's a problem of DL360 G3 and any solution?

[Doug Wogan]

What I mean by mimiking a failure is if you power the server down, pull the hard drive out, and power it back up, the system should see that as a hard drive failure. The hard drive didn't actually fail but you would think the system would see it as a failure. When you put the hdd back in, hotswap, it should rebuild. However it freezes. I'm seeing the same problem you are. And according to HP support there is NO way you can test the hotswap feature until you have a drive failure. I don't believe it any more than you do.

Also you might want to not pull your drives out while they are running. You could cause drive damage and data coruption. The hotswap feature is engineered for adding drives not removing them.

Having said all that it's my opinion HP has an engineering problem with the DL360G3's. If a hdd does fail I'm not at all confident you will be able to hotswap a good drive w/out locking your system.

[jenny_17]

I see.
Actually, I've also try the above cases, using two seperate DL360 G3 Server to ensure there's not hardware failure.

Server A:
Pull one harddisk while the windows is running, then push it back without reboot. The Server freeze. However, the "pulled" harddisk is still healthy as when I plug it back after the Server is shutdown, when boot up, I choose the option "with data recovery automatically", then the harddisk can rebuild successfully and no special error event was found in event viewer.

Server B:
I shutdown the computer, pull one harddisk, start Windows. Then push the harddisk back, the Server freeze.

And I don't believe it is because i does not have a really failure hdd.

[Bill Wood_3 · Accepted solution]

This is a known issue, see the following:

http://wwss1pro.compaq.com/support/reference_library/viewdocument.asp?source=EX031212_CW01.xml&dt=3

In the mean time, do not hot-plug the drives while the server is running. Please power off the server if a drive needs to be replaced. This issue is currently be worked in engineering and it should be resolved soon.

[Doug Wogan]

Finally some proof!

[jenny_17]

Bill Wood, Doug Wogan,

Thanks a lot, Doug Wogan's experience & Bill Wood's link.
Our supporting team is passing the link to US HP today for official confirmation. HP then request us to give diagnosis report for investigation, and now is still waiting for the result. I will post the result here once I got any confirmation. Hope it can also help others.

[jenny_17]

Hi all, finally the fix. I've tried. It works !

http://h18023.www1.hp.com/support/files/server/us/download/20142.html