HP Proliant DL180 G6 - Smart Array P410, bay 1 error

[azkerm]

Hi All,

I've been trying to search over ther internet for a solution to fix this. I'm the new IT person for my organization and our previous IT have not kept any records on certain things. I do understand, that its a bad practice but I'm now making all these on a documentation for any future reference.

Having said that; recenty I came across an issue on our server. We're using an HP Proliant DL180 Gen6 server with ESXi 5.0 ... The issue is; that I'm unable to power up certain VM's as it gave me I/O error. Below seen was the error;

Spoiler

Reason: 0 (Input/output error).
Cannot open the disk '/vmfs/volumes/4e7a4edb-08851e40-0c1e-1cc1de700f23/VM-GATEWAY ( 192.168.0.1 )/VM-GATEWAY ( 192.168.1.2 )-000001.vmdk' or one of the snapshot disks it depends on.

Somehow, by the help of this topic I managed to collect few information and able to understand certain things. If anyone can help me to fix this issue without loosing DATA(s) & also guide me what to do further by answering below;

• Since the array is on a RAID6, I understand that we can replace whether the array will be rebuild. In such case, will I loose any DATA(s)

Thank you in advance.

[Torsten.]

The firmware is 2.74, but should be 6.40 (B)!

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=3883931&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D%257CswItem%253DMTX_208de28c836945daaf355281a1%257CswEnvOID%253D4103%257CitemLocale%253D%257CswLang%253D%257Cmode%253D%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vig...

see also
http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?spf_p.tpst=kbDocDisplay&spf_p.prp_kbDocDisplay=wsrp-navigationalState%3DdocId%253Demr_na-c01857445-8%257CdocLocale%253D%257CcalledBy%253D&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken


your data is likely already corrupted (no longer readable), so the advisory says "restore backup".

[scharchouf]

Error Message: 1778-Drive Array Resuming Automatic Data Recovery Process

Audible Beeps: None

Possible Cause: A controller reset or power cycle occurred while Automatic Data Recovery was in progress.

Action: No action is required.

Error Message: 1716-Slot z Drive Array Controller - Unrecoverable Media Errors Detected on Drives

...Errors will be fixed automatically when these sectors are overwritten. Backup and restore recommended.

Audible Beeps:None

Possible Cause: A media error was detected on a drive and could not be corrected because of degraded fault tolerance or a media error at the same location on another drive in the same array. An unrecoverable read error will be returned to the operating system when this block address is read.

Action: Backup and restore the data on the drive. Sequential write operations to the affected blocks should resolve the media errors.

You need to update your server ;)

Controller Card Firmware is out of Date  v 2.74

[azkerm]

Hi,

Thank you for all the replies. Will updating the server firmware for smart array fix the problem??? or do I have to do anything else??

And any guide to achieve this will be helpful as I'm a newbie user to HP servers. :)

[Torsten.]

Replace the bad disk and restore the backup.

[azkerm]

I've ordered an HP disk which I will get it sooner this week but my ESXi controller reporst continuous I/O error due to the degraded disk. What I don't understand is that; RAID controller reports that I'm missing disk in bay 1 but the rest are okay...

Out 12 bays...

• 1st bay is busted (says missing)
• 4th bay says ok on RAID array (but the UID incators aren't seen),, why?
• 11th bay green UID indicator flashes but not like the other bays.. it flashes in a different manner..

Please have a look into below video which you'll understand.

Refer below for a sample image;

[Torsten.]

>> ESXi controller reporst continuous I/O error due to the degraded disk


I guess the issue is inconsistent data on the remaining disks, see the information in the upper posts.

[waaronb]

Yeah, the amber LED on that first drive means it's failed. No lights on the 4th drive is weird... if it weren't assigned to an array or if it was a spare, that would make sense (or simply broken and the system didn't know it was there). But since it shows up in the drive list as being assigned to the ADG array, that's odd. Could just be a broken LED or the light tube isn't aligned, so you can't see the light. With the system powered off, you could remove that drive and take a look to see if anything is strange. Don't do it with the power on since you're in recovery mode!

The flashing of the other drives is just normal activity... the different flashing of one of them could just be based on how the ADG is recovering things at any given time.

Replacing the failed drive should get the array rebuilding, although the other message about the drive having an error during a previous rebuild is concerning... hopefully you have a backup of the data in case something got corrupted.

The 1716 error means that it encountered a problem on one of the drives during the recovery mode you're in now.

And you should update to the latest P410 firmware. 2.74 is ancient, as in 2010. 6.40 is the latest. I don't know for sure that anything specific in the newer firmware would make a difference, but it can't hurt. When you upgrade to that latest firmware, on the next boot it may tell you that one or more drives needs a firmware update. I have a pretty good feeling that if the P410 firmware is that old, the drives need updates too.

If that error is related to one of the drives having older firmware, the tech advisory says that it might not actually be a problem with the drive, which would be good news for you.

See: http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?locale=en_US&objectID=c03014078

It says it only happens in zero memory mode (the controller doesn't have any cache installed). It looks like yours has 512MB so it might not be the case for you, but it's worth updating the drive firmware.

Best case, you replace the failed drive, get all the firmware updates in place, and the RAID 6 rebuilds itself and that 1716 error message goes away on it's own.

[Torsten.]

Yes, there are several advisories around all suggesting to upgrade the controller and disk drive firmware first.

Get the latest SPP, create a bootable USB stick using the included tool and boot from to update.

http://www.hp.com/go/spp

Does the error appear again after the update?

see

http://h20565.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?spf_p.tpst=kbDocDisplay&spf_p.prp_kbDocDisplay=wsrp-navigationalState%3DdocId%253Demr_na-c03014078-3%257CdocLocale%253D%257CcalledBy%253D&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

[azkerm]

Hello Again! 

I've downloaded the offline ACU to further check the array and found out that there were two disk showing error notifications. Out of 12 slots... 11th one showed that its rebuilding & once 100% completed, it said interim recovery failed. while 10th slot said that its predicting a failure soon. 

What my question is that I've got spare drive which I removed from the server before, and since its a SAS drive, is there a way where I can format and re-use the drive to the array can I just hot-swap the drive while data being in there...???

And what I can do about this interim recovery error?? because the slot shows continuous amber/blue light once rebuild is 100% with an error. when booting it asks to select "F1" & "F2". I didn't select any as it selected the defualt and now it shows the green UID...

If necessary I can attach the ADU report here..