Error message on front panel of autoraid 12H "SYSTEM WARNING`.DATA UNAVAILABLE"

[Ahmed ABDOU_1]

I have an autoraid 12H which experienced a failed disk drive(location: A1).
The autoraid automatically rebuild the data from the failed disc on the hot spare disc
and all was ok. When I replaced the deffective drive, the rebuild went successfully but
we had a message on the front panel of the autoraid 12H "SYSTEM WARNING?.DATA UNAVAILABLE".

System rebooted the system --> same problem.
Autoraid rebooted -----------> same problem.
Replaced the disc in A1 -----> same problem.
Cleared NVram of autoraid ---> same problem.

Decided to upgrade the firmware of the autoraid to the latest revision but we are not able
to do it since we have this error message showing on the front panel of the disc array.

Can somebody shed some light on this strange problem? Thanks in advance and cheers.

[Insu Kim]

I'm not sure but there might be another potential problems.
Use logprint command to get the logs in detail.
Take a close look at that sparing was good, rebuilding has completed successfully and other things.

# logprint -a

Best regards,

[Mark van Silfhout]

Hi,

Data Unavailable Troubleshooting Techniques:
The following is a procedure to follow to overwrite the bad block(s) of data to bring the customer?s array back to a READY state. This procedure can eliminate much of the time required for a restore of an entire LUN or LUNs by enabling you to find the file(s) affected and overwrite those locations with ?good? files.
The only way to get the array back to a READY state is to OVERWRITE the bad data with good data. The WARNING message will go away immediately when the bad block(s) are overwritten.
..... However I would recommend to let an HP hardware engineer carry out these procedures.
Please keep in mind that upgrading the firmware also implicates a new ARM utilities patch.

Regards,

Mark

[Vladimir Lemos]

What is the procedure to write over bad blocks ? Has anyone fixed the data unavailability problem this way ? If not, was the problem solved at all ?

[Charbel Bou-eid]

Dear All,

Did anybody has the procedure to overwrite the defective sector without having to delete/re-create the LUN and rstore data.
I have already replaced the HDD having one Bad Sector (found with arraylog -d) but the rebuild is not yet over for me to see if the warning will stay or not.

Also, I had upgraded the firmware to HP62 in order to detect the affected LUN (as per the information found on the HP download page) but the logprint did not change (I can't see any "Additional Code Sense").

Does anybody think that this warning may affect the I/O performance of the AutoRaid on which an Oracle DB is installed.

Please appreciate anybody's help as the situation is becoming critical at the customer site who is "blaming" the poor performance of Oracle on the AutoRaid.

Thanks,
Charbel

[Vladimir Lemos]

You may wanna try the following document Ids:
lpg67023
lpg75005
lpg67074

man page for the following commands

arraydsp
logprint
lvextend

Review related messages.
This is something that caused big headache!
All that I've saved from this nightmare are the above referenced documents.

I don't recall exactly what I've done, but something along the lines of:

Backup data
create partition using lvextend command
move data into good partition
delete bad partition
recreate partition
merge partitions
didn't work
removed a large .pst (outlool mail) file someone had in disk array -> problem solved!
rebuild disk array

Good luck!

[Gary Badger]

we experienced this problem and with hp support solved it by using the unix dd command on each lun until we found the one where records in did not match records out. Having found the lun on which this occured, we wrote a huge (too big) tar file to each directory. This writes data to all unused areas and removed the error.

[Charbel Bou-eid]

Thanks Vladimir for this info.

Hi Gary,
This is what I had started to do 3 days ago, but unfortuantely I can only run 1 dd session a day as this is a production server, and unfortunately I didn't yet detected the affected LUN.
When you say "write a tar file", do you mean that you have a big tar file on tape for example and you untar it on each of the filesystems mounted under this LUN?
Also please I want to know if this warning has affected the performance of your AutoRaid (slow IO, timeout, etc...) then or you continued to work normally on the Autoraid?

Many thanks to all in advance.
Charbel

[Gary Badger]

Charbul
The large tar file was created to write into each directory on the LUN. It was not unzipped, it was designed to just take up space. Also, the error did not seem to affect system response however, our server is not heavily loaded.

[Tonya Underwood]

I do have the procedure to overwrite the data unavailable if still needed... Will write it up now and post shortly.

[Tonya Underwood]

*** "Data Unavailable" means that both a block of data and the redundancy for that block of the data are not readable.

I assume that what probably happened is that one other drive had a hot plug event at the same time that A1 FAILED. The good news is that you normally only have one bad block after this sort of failure. I wish I could see the logprint. If you care to post it please do. I realize that this procedure is being posted after you have probably solved your issue. But, hopefully making it available to ITRC searchers will help another in the future.

I have outlined the steps below... I have used it successfully many times. MOST times the bad block is in free space... sometimes not...

So.... here is the procedure...

It is attached... it appears to be too lengthy for simple posting...

*********************************************

[Charbel Bou-eid]

Hi Tonya,

Many thanks to your reply.
Actually it isn't late as I still did not resolve the issue because the customer is using his AutoRaid 24/24 and he is not experiencing any problem but it is good at the end of the day to "remove" this annoying message.

I went thru your file and find it very interesting, but I have to tell you that the logprint is giving me an error message "logs contains bad content" or something like that, and although I have HP62 I still cannot see the affected LUN with the Sense Code (as described in one of the hp.com web pages)!!! I can't find the reason behind it.
Also, I have run "arraylog -d Ax/Bx" and haven't find any bad sector in the "growing defect list" of all 12 HDDs.

Anyway, the customer has decided to reformat the AutoRaid, not because of this message but because he wants to recreate his Oracle database in a more professional way and import the data files. But if I can discover the affected LUN with HP62, I can then dd'it and replace the affected sector/file.

Apprecaite if you can tell me exactly how I can use HP62 to dicover the affected LUN.

Thanks again,
Charbel

[Tonya Underwood]

If your logprint shows some "Bad Content" message, I would say that the firmware may not tell you the affected LUN. It would normally be in your logprint output. I'd be curious to see this "Bad Content" entry if you get some time to post that. Maybe run:
logprint > /tmp/logprint.out
and post? Also, if this AutoRAID is shared, the servers may also share logging (through a multi-initiator lock). This would mean you would need to run logprint from both servers to get complete logs.

As far as finding that LUN, I would say just dd each LUN until you find that bad block... If the customer is OK with reformatting the Array I would say go for it. It is certainly easier than finding and overwriting the bad block. However, I would keep this procedure for a later time "just in case". You may need it again!

Thanks,
Tonya Underwood