EVA4400 - System Inoperative due to Metadate Meltdown

[Cresswell Williams]

Hi Experts

We have this error on our EVA4400 when we connect via Console and reboot it:
FCS: FCSRSU - QXCR1000938832 Repro(?) unpack of char in the display
ef70 12-JAN-2010 11:48:38 FCS: FCSRSU - QXCR1000938832 Repro(?) unpack of char in the display
ef70 12-JAN-2010 11:48:38 SCSCSM - SYSTEM INOPERATIVE due to METADATA MELTDOWN meltdown
ef70 12-JAN-2010 11:48:38 SCSCSA - This controller found 36. total disks
ef70 12-JAN-2010 11:48:38 SCSCSA - Both controllers found 36. common disks
ef70 12-JAN-2010 11:48:38 SYSINOP failure code: 0002x
ef70 12-JAN-2010 11:48:38
ef70 12-JAN-2010 11:48:38 SCSCSM - Cell Realization failure <---------
ef70 12-JAN-2010 11:48:38 SCSCSA - S_node_wwn[2] set to 50014380x 025d3030x
ef70 12-JAN-2010 11:48:38 SCSCSA - S_port_wwn[2] set to 5001

Is ther anyway we can rebuild or try to rebuild this EVA without any data loss. i.e is there a command we can run to try and recover the metadata?

Thank you

[Johan Guldmyr]

Hi, have you had a call open with HP about this? Seems like the kind of thing you would want level 2 to have a look at..

[Cresswell Williams]

Yes I do, but i would just like to know your comments OR if somebody has gone though this without any data loss.

[Víctor Cespón]

It has found 36 disks. Is that correct or there should be more disks detected?

Did you remove several disks at once?

Did several disks go into failed state at the same time?

What firmware does the EVA have?

[Cresswell Williams]

36 disks are correct.
Firmware is XCS v09522000
No disks were removed at once from the EVA. All 36 Disks are showing green lights, without any activity.

Thank you

[Johan Guldmyr]

Oki =)

Did you do anything while this happened?
Any error leds on the controller/disk shelves or disks?

[Cresswell Williams]

Okay, forgot to mention the all disks are on HP06 firmware.

[Cresswell Williams]

Further more this is only picked up in Controller 2 and not on Controller 1.
scshb_util_poll_drives
ef70 12-JAN-2010 11:49:42 SDC->SCMI: Cache condition is CACHE_COND_MIRROR_OFF (2).
ef70 12-JAN-2010 11:49:42 SDC->SCMI: The condition of the embedded network switch's temperature sensor is TSENSE_COND_BAD (1).
ef70 12-JAN-2010 11:49:42 SDC->SCMI: 'This' controller's condition is NSC_COND_BAD (2),
ef70 12-JAN-2010 11:49:42 SDC->SCMI: because (SDC_MON_is_nsc_faulted() == TRUE).
ef70 12-JAN-2010 11:49:42 SDC->SCMI: The condition of the embedded network switch's temperature sensor is TSENSE_COND_BAD (1).
ef70 12-JAN-2010 11:49:42 SDC: Completed parsing Enclosure's PCA EEPROM.
ef70 12-JAN-2010 11:49:42 SDC: hardware revision: "005 "
ef70 12-JAN-2010 11:49:42 SDC: vendor id: "HP "
ef70 12-JAN-2010 11:49:42 SDC: product id: "HSV300 "
ef70 12-JAN-2010 11:49:42 SDC: world wide name: "500508B4000BCC0C"
ef70 12-JAN-2010 11:49:42 SDC: assembly serial number: "P6314E29SWW037 "
ef70 12-JAN-2010 11:49:42 SDC: assembly part number: "AG637-60501 "
ef70 12-JAN-2010 11:49:42 SDC: salable serial number: "SGA90100W1 "
ef70 12-JAN-2010 11:49:42 SDC: salable product number: "AG637A "
ef70 12-JAN-2010 11:49:42 SDC: spare part number: "461491-001 "
ef70 12-JAN-2010 11:49:48 OFFLOAD: DLQ EMPTY.
ef70 12-JAN-2010 11:49:48 OFFLOAD: No waiting processes; requesting control.
ef70 12-JAN-2010 11:49:48 OFFLOAD: Waiting in line...
ef70 12-JAN-2010 11:49:48 OFFLOAD: Control obtained.
ef70 12-JAN-2010 11:49:48 OFFLOAD: DLQ EMPTY.
ef70 12-JAN-2010 11:49:48 OFFLOAD: No waiting processes; relinquishing control.
ef70 12-JAN-2010 11:49:48 SDC->SCMI: Cache condition is CACHE_COND_MIRROR_OFF (2).
ef70 12-JAN-2010 11:49:48 SDC->SCMI: The condition of the embedded network switch's temperature sensor is TSENSE_COND_BAD (1).
ef70 12-JAN-2010 11:49:48 SDC->SCMI: 'This' controller's condition is NSC_COND_BAD (2),
ef70 12-JAN-2010 11:49:48 SDC->SCMI: because (SDC_MON_is_nsc_faulted() == TRUE).
ef70 12-JAN-2010 11:49:48 SDC->SCMI: Enclosure's condition is ENCL_COND_DEGRADED (2),
ef70 12-JAN-2010 11:49:48 SDC->SCMI: because (sub_is_degraded == TRUE).
ef70 12-JAN-2010 11:49:48 SDC: Exclusive access to the Offload PIC lasted 0% of the allotted time.
ef70 12-JAN-2010 11:49:52 SDC->SCMI: Cache condition is CACHE_COND_MIRROR_OFF (2).
ef70 12-JAN-2010 11:49:52 SDC->SCMI: The condition of the embedded network switch's temperature




So is Metadata resides on the disks, and one controller is faulty, the other controller should be able to continue working?

Thank you

[Víctor Cespón]

This is a case for HP L2 and L3. Don't touch anything and get them involved.

[Mark...]

Hi,
Hopefully HP will be able to resolve your issues soon.
Please post the resolution, when know, so that others may learn.
Mark...