NetRaid 1M error in Raid 5

[Peter Schöttler]

Hi everybody,

our Netserver LP100r reports an error on it's Netraid 1M controller. This problem has occured for two weeks and immediately we shut down the server.

The beep code says that two or more drives of the raid 5 have failed.
When I start the NetRaid Express Tools and view the current configuration only one disc is shown as failed, the drive A00-00. The others are online.

Why this beep code?

I want to mention that when the error occured there was shown a red light on the panel of the netserver which indicates that the hard disc in the left tray has an error. Meanwhile there isn't shown any red light. Maybe this happened, because I silenced the alarm in the Express Tools, isn't it? And if it was the same beep code than now, I don't know.
Btw when the error happened the first time we were still able to boot Windows on the server.

A simple question is which drive is shown as faulty by the Express Tools?
The list of the Express Tools orders the three drives by SCSI-ID 0 to 2 and shows the drive names A00-02 down to A00-00 in opposite order. I guess that the second number is the tray slot in the netserver, starting with 00 as the left one, 01 in the middle and 02 the right one. Is this correct?

In the meantime we replaced the as faulty assumed disc, but the error code beeped again and also the new hard disc was shown as faulty by the red light on the panel. It recognized that the old hard discs have an HP firmware HP04 while the new hard disc's firmware is HPC1. Meanwhile I shipped this hard disc back to our supplier to receive a going one.

I'm confused. Either unlikely we received an faulty hard disc again or the hard discs' firmware is the reason or the netraid controller is faulty. Unfortunately I don't have any other drives to test the controller or the Raid.

In the meantime we aren't able to boot Windows from the two left hard discs on the server anymore, as I mentioned it before.

Can anybody give me hint what to do now?

(Yes, I have a backup - thank goodness!)

Greetings,
Peter

[Sean T. Craig · Accepted solution]

Hi Peter,

Phew, where to start. Okay, first of all, this "Beep Code" you're getting; I don't think it is a code at all but rather, the NetRAID Alarm indicating a failed hard drive. There is no special code to indicate that there are more than 1 failed disks, only a message at post that says "One Logical Drive Failed".

In this case, I believe it more likely that you only had 1 physical drive failed making your logical drive degraded. Based on what you describe, your NetRAID Express Tools show the drives like this:
ID0 - ONLIN - A00-02
ID1 - ONLIN - A00-01
ID2 - FAIL - A00-00
Is this accurate?

Here is how the naming convention works:
ID0 = Left
ID1 = Middle
ID2 = Right
A00 = Array 0
Axx-00 = first drive in logical array
Axx-01 = second drive in logical array
Axx-02 = third drive in logical array

The firmware doesn't really have any effect on whether or not it can be used in the array; you can have 1xHP04, 1xHPC6 & 1xD94N; it doesn't really matter. What does matter is that the capacity meets or exceeds that of the smallest drive currently configured in the array.

I should also mention that a failed status in the Express Tools doesn't necessarily mean a bad drive because the utility isn't looking at the drive for this status, it is looking at the memory (NVRAM) of the controller. When a drive fails, it changes the status on the contoller to failed and it will stay there until some form of user intervention occurs; it will never show that drive as being online again until you either replace the drive, reseat the drive or force the drive online. Also, if auto-rebuild is disabled on the controller, or if the server is powered off, reseating it will not start a rebuild, nor change the status.

Here's what you should do:
1. Go into Express Tools
2. Select "Objects"
3. Select "Physical Drive"
4. Press on the first drive and verify that capacity is shown (not 0MB)
5. Verify that there are no Media Errors or Other Errors
6. Repeat this step with the other 2 drives.
7. Confirm that you replaced the drive on the right-hand side
8. Press on the failed drive
9. Select "Rebuild" from the menu

If you replaced the drive on the left-hand side, then we have a problem. If the server was powered down when you replaced it, the controller will still think it is online despite the fact that there is nothing on it.

If there is no way of getting your old drive returned to you, try to fail ID0 and force ID2 online. This will not work if the operating system had been running with this drive in a failed state because the stripe set will no longer be consistent with the rest of the array. In this circumstance, you will need to make all drives online, re-initialize the array, and re-install the Operating System, followed by a restore from backup.

I wish you all the best of luck with this one,

Sean T. Craig Sr. C.E.T., M.C.P.

[Peter Schöttler]

Hi Sean,

the different beep codes are shown in the manual of the NetRaid. The actual beep code should result of the failure I mentioned above. But I think, too, that there is a failure of one hard disc only.

(A short comment: I haven't shiped my old drive to the supplier back. I returned the new drive I received from him, because I assumed it was faulty, too. That means in the moment I have each of the three original drives.)

Meanwhile I don't believe in the Express Tools which drive is indicated as ONLINE or FAIL. The configuration of disk and of NVRAM are contradictorily. The disk configuration only shows drive ID2:A00-00 is failed. NVRAM configuration shows each drive failed.
When I view the properties of the physical discs by pressing only the first drive ID0:A00-02 has 'Other Errors' = 5. There are no Media Errors on this drive and no errors at all on the remaining two drives. This indicates that indeed the first drive in the left tray is the faulty one, I think.

As I already mentioned the server can't boot Windows anymore. I think, this behaviour was produced by myself, because I swapped the drives in the different trays to see which light of the front panel indicates the faulty hard drive. I hope, I didn't destroy the Raid finally.

Greetings,
Peter

[Sean T. Craig]

Okay, I think I see what's going on now. Because you have a configuration mismatch, the server is now stopping at that point. If you choose to continue the boot process, the server will use the NVRAM configuration which shows more than one drive failed.

What you need to do is:
1. Go into Configure-->View/Add-->Disk Configuration.
2. Escape out and choose "Yes" when it asks you to save. This will synchronize the NVRAM and the Disks (i.e. no more mismatch)
3. This should allow you to boot the server to the operating system.

Now, the non-media errors on ID0:A00-02 will likely cause a rebuild to fail; you need to get those errors corrected. HP says the fix for this is to update the Firmware of both the Disks and the Controller. The latest firmware for the NetRAID-1M controller is H.02.02 and can be found here:
http://h20000.www2.hp.com/bizsupport/CoreRedirect.jsp?targetPage=ftp%3A%2F%2Fftp.hp.com%2Fpub%2Fsoftlib%2Fsoftware2%2FCOL4353%2Fns-14418-1%2Ffw_nr_1m2m_h0202_5.exe

To update the firmware on the hard drives, the easiest way is to create a bootable CD using the .ISO image found here:
http://h20000.www2.hp.com/bizsupport/CoreRedirect.jsp?targetPage=ftp%3A%2F%2Fftp.hp.com%2Fpub%2Fsoftlib%2Fsoftware3%2FCOL5687%2Fns-18695-1%2FCAT43F_UTL4Q13_RELEASED.ISO

If the rebuild still doesn't complete, connect your hard drives to your embedded SCSI Controller and use the config utility to run a "Media Verify" of your drives.

Best of luck. Let us know how it goes,

Sean T. Craig Sr. C.E.T., M.C.P.

[Peter Schöttler]

Hi Sean,

let's start from the beginning.
I followed your instruction in your last mail.
I saved the existing disk configuration as the NVRAM configuration.
Successfully I updated the NetRaid controller to it's latest firmware.
Successfully I updated the HDDs to their latest firmware.
Successfully I rebuild the third HDD 'ID2 A00-00' in the right tray which was marked as faulty.
Successfully the consistency of the Raid5 was proven.

Nevertheless the netserver doesn't boot anymore -> 'No operating system'

I want to remark two points.
Each drive is indicated as ONLINE by NetRaid Express Tool. But it still shows 'Other errors = 5' for the HDD 'ID0 A00-02' in the left tray. This HDD was indicated as faulty by a red LED at the Netserver's front panel at the beginning. Because I didn't start the Netraid Express Tool not until I swapped the HDDs among each other (for testing purpose), the Express Tool didn't show this HDD 'ID0 A00-02' as faulty. Instead of this, it showed the 'ID2 A00-00' as faulty. I fear, I produced this error, because I swapped the HDDs and so I arranged the Raid5 in the wrong order. Which results in a failure of 'ID2 A00-00' and the logical drive completely.

Bad chance to get the logical drive to boot, isn't it?

Peter

[Peter Schöttler]

I want to add that it's possible to restore the most important data by a backup. Up to now I worked so hard and long to solve this problem, so I want to succeed without reinstalling the server and restoring the backup.

[Sean T. Craig]

Hi Peter,

Okay, it looks like there is a possibility that everything is gone. The "No Operating System" message can be caused by one of two things: 1) there is no boot sector on the logical drive or 2) the system is trying to boot to a different device.

Try this:
1. Clear the CMOS and reload setup defaults
2. Try booting to a DOS boot diskette and run FDISK to see if there is a partition defined. It won't mount an NTFS File System but it should at least show a "Non-DOS" Partition.

If there is no partition defined, then the stripe set is likely corrupted beyond repair and you will need to re-initialize the array and restore from backup.

Good luck,

Sean T. Craig Sr. C.E.T., M.C.P.

[Peter Schöttler]

Hi Sean,

I think, the Raid is corrupted and no data is on it anymore.
As my last try I booted from the W2k cdrom and used the drivers of the Netraid yesterday evening. But W2k's installation routine can't find any data on it. So I guess that the swapping of the HDDs in the trays among each other lead to the fatal error and I produced it by myself.
Ok, the conclusion is that I never should play DJ with the HDDs in a Raid5 :-(

Lastly I hope the backup tape isn't corrupt and it can be restored successfully.
Please, wish me good luck ;-)

And I want to thank you for the professional help on my problem.

Best regards,
Peter Schoettler

[Sean T. Craig]

Hi Peter,

I do, indeed wish you the best of luck with this. I hope some of these steps will help you in the future with configuring and managing your arrays. If any of these responses were of value, please feel free to rate them by assigning points to them.

Bye for now,

Sean T. Craig Sr. C.E.T., M.C.P.