topic Re: VA7400 - Data Unavailable in Disk Enclosures

VA7400 - Data Unavailable

Brian Duthie_1 — Fri, 21 Feb 2003 09:24:29 GMT

Gooday - I can not backup the LUN's in my VA7400.
I get the following information from the VA.
Any ideas what I can do to force recover the ARRAY?

# armrecover -s VA01

Array State:
Warning

THE FOLLOWING WARNING(S) EXIST:
- Data Unavailable
- Redundancy Loss
- Rebuild Failed
No recovery needed on array VA01.
# armdsp VA01

WARNINGS

WARNING: The controller cannot retrieve some portion of host data.

WARNING: A rebuild operation failed.

WARNING: Some data in the device lacks redundancy, and is exposed to becoming unavailable if further drive removals or failures occur.

Re: VA7400 - Data Unavailable

Ricardo Rocha — Fri, 21 Feb 2003 10:36:57 GMT

Hi

From the file you attached I can see the symptoms but not the cause. The firmware should be changed to HP18 but only after the problem is solved. There are several other logs possible to get from the VA so you should call HP to do this and correct the faillure(s).

Can you see the luns from your system?

Good luck,

Ricardo

Re: VA7400 - Data Unavailable

Ricardo Rocha — Fri, 21 Feb 2003 10:52:14 GMT

I forgot to mention. You don't need to run the command armrecover. I think this is only used to get the map data from the disks. In your case the map is ok, that's why it's not needed.

Bye,

Ricardo

Re: VA7400 - Data Unavailable

Brian Duthie_1 — Fri, 21 Feb 2003 12:41:07 GMT

Hi Robert---- Thank you for this reply. This occured after a poer outage and the UPS's went flat - so unscheduled shutdown.

Thank you for the tip on the firmware - BUT the local HP agent wishes to upgrade anyway ? You say only AFTER applying some sort of fix ?

Re: VA7400 - Data Unavailable

Brian Duthie_1 — Fri, 21 Feb 2003 12:46:49 GMT

OOPS -Sorry that should have been Ricardo, not Robert......

And to answer the other question - YES I can see the LUN's from the System but I do have access problems to the data.

Re: VA7400 - Data Unavailable

Ricardo Rocha — Fri, 21 Feb 2003 12:48:06 GMT

I'm not Robert :)

Well, the VA won't let the upgrade to be performed while it is in a warning state. So it's not possible. The errors must be cleared before. From the armdsp -a it's not possible to identify the problem. There are other logs to get like armlog, but these ones must be peformed by HP, because only them can analyse it. From the log you sent all hardware seems ok, so it can be a logical problem (several problems are corrected with HP18).

Good luck,

Ricardo

Re: VA7400 - Data Unavailable

Eugeny Brychkov — Fri, 21 Feb 2003 13:04:38 GMT

Hello Brian,
note on armdsp -a: why private loop? If connected to switch should be in fabric (not related to your problem). Please zip 'logprn -t All -v -a ' and attach to your next reply. Make sure it's not bigger than 256KB (otherwise will not attach correctly).
Eugeny

Re: VA7400 - Data Unavailable

Brian Duthie_1 — Fri, 21 Feb 2003 14:39:41 GMT

Thnak you for the heads up on the interface.....

The logprn output is attached however I get the error below...

Device Unique Name = 50060B00000926E3
Device Name = HP StorageWorks Virtual Array 7400
Date = November 23, 2002
Time = 10:12:57 AM GMT
Controller Time Stamp = 2224139980276
Event Number = 117353
Sequence Number = 154
Event Type = Controller Event
Event Code = 571
Severity = 4
Event Count = 1
FRU Location = M
FRU Data = enclosureId/slot/component/subcomponent : 0x00/0xff/0xff/0xff
Reporting Controller = M/C1
Event Source Info = serialNum/moduleId/processId : 00PR00000921/0x36/0xffffffff
Event Description = NEW_DEVICE_FOUND_EH A new device was discovered by CFMhot.
Event Unique Info = Node WWN: 20000004cf13aec9
Port WWN: 21000004cf13aec9
HA: 1e
Slot: 004a
EXECUTION ERROR

logprn Usage:
logprn -a [-s ] [-e ]
[-t ] [-sv ] [-d ] [-b | -v | -xml] [-?]

Re: VA7400 - Data Unavailable

Vincent Fleming — Fri, 21 Feb 2003 16:02:01 GMT

What's with your date? It's not November 23, 2002!

Anyway, your last error message (in your post) indicates that the disk in slot 15 of your 2nd DS2400 (the 45th drive in the array) has reappeared. This would indicate a problem with that drive, if it's dropping off the loop and coming back in.

Actually, from the log, lots of them are doing this. If you haven't yet called and arranged for a service call, do so now.

Good luck,

Vince

Re: VA7400 - Data Unavailable

Ricardo Rocha — Fri, 21 Feb 2003 16:10:56 GMT

I think Brian was not able to get all the logs. You should try the same comand again but using the -s option, to get the logs ,for instance, from last week until now. What we see on the logs you already sent is the boot process of the VA, so the disk messages are normal. The las log I saw indicated the VA was fine. Please run the logprn command again with latest logs.

Bye,

Ricardo

Re: VA7400 - Data Unavailable

Eugeny Brychkov — Fri, 21 Feb 2003 16:26:30 GMT

Execution error may be caused by corrup host logs. Then attach 'armlog -e ' output zipped
Eugeny

Re: VA7400 - Data Unavailable

Brian Duthie_1 — Fri, 21 Feb 2003 17:21:50 GMT

I am still battling with the logprn command to get an output but here is the armlog output . Thank you

Re: VA7400 - Data Unavailable

Brian Duthie_1 — Fri, 21 Feb 2003 17:54:56 GMT

here is an output from "logprn -a VA01" All other options give the above error ...

Thank you for your trouble.

Re: VA7400 - Data Unavailable

Eugeny Brychkov — Fri, 21 Feb 2003 19:04:44 GMT

Brian,
VA disk array has a certain procedure how should it be turned off and on.
Off: turn off main (controller) enclosure first. Allow it to shutdown correctly writing all needed info (maps) to the disks. Then turn off all disk enclosures. Then operate main (cabinet, UPS) breaker.
On: turn on all disk enclosures, wait for some time to allow disks to start and initialize (~1 minute) and then turn on main (controller) enclosure.
Do you comply with these rules? Or do you do in another way?
I see many JBOD signal lost in the log, this may mean that you turn off JBOD before enclosure. In addition, I see power supply failure events, they may mean that you operate main breaker before turning off whole VA.
Powering disk enclosure off first may interrupt both backend loops, and if you'll power off main enclosure after it then waiting I/Os may be lost, and array recovery routine may try to recover data, but as soon as there will be more that one disk lost in each RG array will flag 'data unavailable'.
Eugeny

Re: VA7400 - Data Unavailable

Brian Duthie_1 — Sat, 22 Feb 2003 08:44:14 GMT

Good day Eugeny - That you for the above Detailed information. WHEN we can we stick to this procedure......

The problem that I live with is the following. the power is very unstable so we have UPS's, battery time 15 min. and a Generator. This normally keeps the data center up. How ever the generator does not allways kickin in time and the UPS die. We have 3 UPSs and the 2 PDUs in the rack go to separate UPS's some times 1 UPS dies first - this is the loss of power you see.

My problem is when the power goes off at night and the UPS's die and then the power is restored - The VA comes on line INCORRECTLY - that is - the controller comes on line at the same time as the JBOD's and not as per your e-mail allowing the 6 JBODS to stablise first. We are addressing this problem with the utmost Urgency.

Re: VA7400 - Data Unavailable

Brian Duthie_1 — Sat, 22 Feb 2003 08:55:26 GMT

To reply to Vincent - I logged this with HP Wednesday 19th. They changed one controller Thursday Evening - did not resolve problem. They now wish to upgrade Firmware - but reading above we will NOT be able to ? This is why I turned to the Forum to see if I could get any other Assistance.

Re: VA7400 - Data Unavailable

Eugeny Brychkov — Sat, 22 Feb 2003 22:31:51 GMT

As a workaround try to reconfigure aray in secure or even single controller mode (armmgr -J). I believe in this mode controllers' write cache will be disabled and all I/Os will bypass it and treated as forced unit access (disk access).
Please update us on your problem resolution progress
Eugeny

Re: VA7400 - Data Unavailable

Claudiu Schmidt — Mon, 24 Feb 2003 06:26:09 GMT

Hi Brian,

i'm glad there is somebody outsite haveing same problems as we did. We have(had) a VA7400 with the same simptoms as you described. We tried everything, exchanged both controlers, did partial resets, on the VA, read all of the luns, to let the VA know if some portion from the VA has any data loss, allocated all of the space of the VA in luns, and did a dd on this ones, to see if the data unavailable is not in the free space, but nothing helped. At last, we backuped all data from the VA, and formated the VA. Afterwards, everything was ok :)
One week after the formating, and firmware upgrade of controlers to HP18, and disks, i inserted a bad disk into one enclosure, and Va went back into the same status data unavailable, redundancy loss. And thats why we "HAD" a VA7400. It was exchanged with a VA7410 last week. HP was not able to tell me where this error comes from, even after sending MB's of logsfiles.
Have fun!

Re: VA7400 - Data Unavailable

Brian Duthie_1 — Mon, 24 Feb 2003 11:43:10 GMT

The Service Centre tried the update to no avail. Will try the armmgr -J command.

Re: VA7400 - Data Unavailable

Brian Duthie_1 — Mon, 24 Feb 2003 17:09:19 GMT

I issued the armmgr -J commands successfully. But the VA reamins in same state!

I issued the armrbld -r command and I get the same result. see attached armlog output.