VA7400 - Data Unavailable

[Brian Duthie_1]

Gooday - I can not backup the LUN's in my VA7400.
I get the following information from the VA.
Any ideas what I can do to force recover the ARRAY?

# armrecover -s VA01

Array State:
Warning

THE FOLLOWING WARNING(S) EXIST:
- Data Unavailable
- Redundancy Loss
- Rebuild Failed
No recovery needed on array VA01.
# armdsp VA01

WARNINGS

WARNING: The controller cannot retrieve some portion of host data.

WARNING: A rebuild operation failed.

WARNING: Some data in the device lacks redundancy, and is exposed to becoming unavailable if further drive removals or failures occur.

[Ricardo Rocha]

Hi

From the file you attached I can see the symptoms but not the cause. The firmware should be changed to HP18 but only after the problem is solved. There are several other logs possible to get from the VA so you should call HP to do this and correct the faillure(s).

Can you see the luns from your system?

Good luck,

Ricardo

[Ricardo Rocha]

I forgot to mention. You don't need to run the command armrecover. I think this is only used to get the map data from the disks. In your case the map is ok, that's why it's not needed.

Bye,

Ricardo

[Brian Duthie_1]

Hi Robert---- Thank you for this reply. This occured after a poer outage and the UPS's went flat - so unscheduled shutdown.

Thank you for the tip on the firmware - BUT the local HP agent wishes to upgrade anyway ? You say only AFTER applying some sort of fix ?

[Brian Duthie_1]

OOPS -Sorry that should have been Ricardo, not Robert......

And to answer the other question - YES I can see the LUN's from the System but I do have access problems to the data.

[Ricardo Rocha]

I'm not Robert :)

Well, the VA won't let the upgrade to be performed while it is in a warning state. So it's not possible. The errors must be cleared before. From the armdsp -a it's not possible to identify the problem. There are other logs to get like armlog, but these ones must be peformed by HP, because only them can analyse it. From the log you sent all hardware seems ok, so it can be a logical problem (several problems are corrected with HP18).

Good luck,

Ricardo

[Eugeny Brychkov]

Hello Brian,
note on armdsp -a: why private loop? If connected to switch should be in fabric (not related to your problem). Please zip 'logprn -t All -v -a ' and attach to your next reply. Make sure it's not bigger than 256KB (otherwise will not attach correctly).
Eugeny

[Brian Duthie_1]

Thnak you for the heads up on the interface.....

The logprn output is attached however I get the error below...

Device Unique Name = 50060B00000926E3
Device Name = HP StorageWorks Virtual Array 7400
Date = November 23, 2002
Time = 10:12:57 AM GMT
Controller Time Stamp = 2224139980276
Event Number = 117353
Sequence Number = 154
Event Type = Controller Event
Event Code = 571
Severity = 4
Event Count = 1
FRU Location = M
FRU Data = enclosureId/slot/component/subcomponent : 0x00/0xff/0xff/0xff
Reporting Controller = M/C1
Event Source Info = serialNum/moduleId/processId : 00PR00000921/0x36/0xffffffff
Event Description = NEW_DEVICE_FOUND_EH A new device was discovered by CFMhot.
Event Unique Info = Node WWN: 20000004cf13aec9
Port WWN: 21000004cf13aec9
HA: 1e
Slot: 004a
EXECUTION ERROR

logprn Usage:
logprn -a [-s ] [-e ]
[-t ] [-sv ] [-d ] [-b | -v | -xml] [-?]

[Vincent Fleming]

What's with your date? It's not November 23, 2002!

Anyway, your last error message (in your post) indicates that the disk in slot 15 of your 2nd DS2400 (the 45th drive in the array) has reappeared. This would indicate a problem with that drive, if it's dropping off the loop and coming back in.

Actually, from the log, lots of them are doing this. If you haven't yet called and arranged for a service call, do so now.

Good luck,

Vince

[Ricardo Rocha]

I think Brian was not able to get all the logs. You should try the same comand again but using the -s option, to get the logs ,for instance, from last week until now. What we see on the logs you already sent is the boot process of the VA, so the disk messages are normal. The las log I saw indicated the VA was fine. Please run the logprn command again with latest logs.

Bye,

Ricardo

[Eugeny Brychkov]

Execution error may be caused by corrup host logs. Then attach 'armlog -e ' output zipped
Eugeny

[Brian Duthie_1]

I am still battling with the logprn command to get an output but here is the armlog output . Thank you

[Brian Duthie_1]

here is an output from "logprn -a VA01" All other options give the above error ...

Thank you for your trouble.

[Eugeny Brychkov]

Brian,
VA disk array has a certain procedure how should it be turned off and on.
Off: turn off main (controller) enclosure first. Allow it to shutdown correctly writing all needed info (maps) to the disks. Then turn off all disk enclosures. Then operate main (cabinet, UPS) breaker.
On: turn on all disk enclosures, wait for some time to allow disks to start and initialize (~1 minute) and then turn on main (controller) enclosure.
Do you comply with these rules? Or do you do in another way?
I see many JBOD signal lost in the log, this may mean that you turn off JBOD before enclosure. In addition, I see power supply failure events, they may mean that you operate main breaker before turning off whole VA.
Powering disk enclosure off first may interrupt both backend loops, and if you'll power off main enclosure after it then waiting I/Os may be lost, and array recovery routine may try to recover data, but as soon as there will be more that one disk lost in each RG array will flag 'data unavailable'.
Eugeny

[Brian Duthie_1]

Good day Eugeny - That you for the above Detailed information. WHEN we can we stick to this procedure......

The problem that I live with is the following. the power is very unstable so we have UPS's, battery time 15 min. and a Generator. This normally keeps the data center up. How ever the generator does not allways kickin in time and the UPS die. We have 3 UPSs and the 2 PDUs in the rack go to separate UPS's some times 1 UPS dies first - this is the loss of power you see.

My problem is when the power goes off at night and the UPS's die and then the power is restored - The VA comes on line INCORRECTLY - that is - the controller comes on line at the same time as the JBOD's and not as per your e-mail allowing the 6 JBODS to stablise first. We are addressing this problem with the utmost Urgency.

[Brian Duthie_1]

To reply to Vincent - I logged this with HP Wednesday 19th. They changed one controller Thursday Evening - did not resolve problem. They now wish to upgrade Firmware - but reading above we will NOT be able to ? This is why I turned to the Forum to see if I could get any other Assistance.

[Eugeny Brychkov]

As a workaround try to reconfigure aray in secure or even single controller mode (armmgr -J). I believe in this mode controllers' write cache will be disabled and all I/Os will bypass it and treated as forced unit access (disk access).
Please update us on your problem resolution progress
Eugeny

[Claudiu Schmidt]

Hi Brian,

i'm glad there is somebody outsite haveing same problems as we did. We have(had) a VA7400 with the same simptoms as you described. We tried everything, exchanged both controlers, did partial resets, on the VA, read all of the luns, to let the VA know if some portion from the VA has any data loss, allocated all of the space of the VA in luns, and did a dd on this ones, to see if the data unavailable is not in the free space, but nothing helped. At last, we backuped all data from the VA, and formated the VA. Afterwards, everything was ok :)
One week after the formating, and firmware upgrade of controlers to HP18, and disks, i inserted a bad disk into one enclosure, and Va went back into the same status data unavailable, redundancy loss. And thats why we "HAD" a VA7400. It was exchanged with a VA7410 last week. HP was not able to tell me where this error comes from, even after sending MB's of logsfiles.
Have fun!

[Brian Duthie_1]

The Service Centre tried the update to no avail. Will try the armmgr -J command.

[Brian Duthie_1]

I issued the armmgr -J commands successfully. But the VA reamins in same state!

I issued the armrbld -r command and I get the same result. see attached armlog output.

[Brian Duthie_1]

Array in same state had a partitial rebuild the failed.

See latest logs...

[Eugeny Brychkov]

Brian,
armmgr -J command will not repair VA. If you'll set VA up to restricted or secure mode, I believe write cache will be disabled. After you'll repair VA (reformat it) this setting may help (or may not, I do not know).
I think there's something wrong with VA. It should be invulnerable to power failures. You need HP to troubleshoot and maybe replace some parts :o(
Please keep us posted
Eugeny

[Eugeny Brychkov]

To be correct on the 'invulnerable' word... I meant VA should come back ok after power failure without these data unavailable problems your VA experiencing
Eugeny

[Vincent Fleming]

That last log you posted is very interesting. Have you noticed that all the errors are being reported by MC/1 (controller-1)???

There's no mention of MC/2 in the file.

Also, it's pretty obvious that MC/1 is having trouble accessing the back-end FC loop(s).

Try this...

Remove controller MC/1, and run off MC/2 only.

If your errors go away - it proves that you need a new MC/1 controller.

Either way, you should log a hardware problem with HP service, and have a serice tech come out and look at it.

Good luck,

Vince

[Brian Duthie_1]

Good Day Vincent, Thank you for checking the logs again.

But as you can see above HP changed a controller. It was the one in MC/1 .... Did not help.

We also still have the call open with HP and they are as baffeled as I am at the moment.

Thank you. Brian