EVA 4000 random Drive/Mirror Port Errors

[J.Kool_1]

Hi,

On one of my EVA4000 boxes i suddenly recieve random event 060b4709's scattered over various enclosures disks. The warning shows up on a daily bases. Anyone have any idea what they mean exactly.

--- The actual event:

everity: Warning -- not failed but attention recommended or required. A non-data exchange sent to a physical disk drive has timed out.

Error reading extended event description from c:\hsvmafiles\napp\eventout.txt

EIP09 - Fibre Channel Services Physical Disk Drive/Mirror Port Error
An error was encountered while attempting to access a physical disk drive or the mirror port.

Event Log Packet Event Specific Information

flags (Flags)
Flag field bits: 01000011

Time has been set on this HSV200 controller: TRUE
Time has been synchronized with all HSV200 controllers in the Storage System: TRUE
Event sequence number reset occurred: FALSE
Event reported out of sequence due to Final Event Block reconciliation or lost host event: FALSE
Event requeued following restart or resynchronization: FALSE
Event reported using LAB code: FALSE
Event reported by primary HSV200 controller (Note: Not valid until Storage System primary HSV200 controller is elected): TRUE
Single power supply HSV200 controller: FALSE

revision - Structure revision number: 2 [0x02]
count - Event specific information size in bytes: 172 [0xac]
sequence_number - Sequence number assigned to the event: 37566 [0x92be]
sw_version - HSV200 controller software version number string: 5020
baselevel_id - HSV200 controller baselevel build string: CR0920
ctrlr_model_id - HSV200 controller model string: HSV200
reporting_ctrlr - Storage System Management Interface Handle of HSV200 controller that reported the event: Noid: 0 IdType: 7 IdLength: 8 UUID: 5005-08b4-0010-4d03-0000-0000-0000-0000
report_time - Time event was reported: Date: 30-Dec-2008 Time: 03:52:14:850
report_location - Location of event report call: 1177492 [0x11f794]

header (Header Information)

** NOTE: The following is part of a UNION with more than one interpretation.
You will need to determine which union value is relevant.

** START UNION: ( u - Event Code Union ) **

ec (Event Code)
eiptype - Event Information Packet Type Code: 9 [0x09]
cac - Corrective Action Code: 71 [0x47]
Corrective Action: Dropped frames are potential indications of an impending Fibre Channel port or physical disk drive failure when they occur excessively. If frame drop becomes excessive, the indicated Fibre Channel port or the indicated physical disk drive will be placed in the inoperative state. This is a preemptive action warning, no immediate action is necessary.
evnum - Event Number: 11 [0x0b]
scid - HSV200 Controller Software Component Identification: 6 [0x06]

** UNION OR: ( u - Event Code Union ) **
value - Event Code Value: 101402377 [0x060b4709]
** END UNION: ( u - Event Code Union ) **
revision - Packet revision number: 4 [0x04]
type - Packet type: 9 [0x09]
count - Number of bytes in packet: 108 [0x6c]

device - UUID of physical disk drive associated with the event: 2000-0000-8776-a962-0000-0000-0000-0000
cerp_id - HSV200 controller enclosure rear panel Fibre Channel port attached to the physical disk drive or mirror port: DP-1A
exch_type - Frame exchange type: 2 [0x02]
port - HSV200 controller internal Fibre Channel port number attached to the physical disk drive or mirror port: 0 [0x00]
dencl_num - Enclosure where the physical disk drive is located: 3 [0x03]
al_pa - AL_PA of the physical disk drive or mirror port: 89 [0x59]
rack_num - Rack where physical disk drive is located: 0 [0x00]
bay - Enclosure bay where the physical disk drive is located: 9 [0x09]
fed_class - Fibre Channel Exchange Descriptor class: 33554476 [0x0200002c]

[IBaltay]

Hi,

THE MIRROR PORT
if it is random daily on the various enclosures/disks, then it could be the problem of the mirror port itself.
The failing element could be either the controller´s
a) chip
b) mirror port SFP
c) cable (copper)
EVA4000 has 4 mirror ports
There are the followin LED indications of the mirror port health:
GREEN - normal operation
AMBER - no signal detected or the controller port failure
OFF - no GBIC/SFP detected

NOTE
The firmware cannot determine the root cause of the failure. It can determine only that it was unable to properly communicate with the other mirror port. The problem may be in the transmission or the receiving function of either controller. The controller firmware retries the failed mirror port once per second. The port LED of a failed mirror port will return to green very quickly after the action that resolved the failure.

THE PHYSICAL DISK
if it is always the same enclosure/bay/disk, then it could be the problem of the physical disk

The actual event comes from the
1. DISK BACK END PORT DP-1A
cerp_id - HSV200 controller enclosure rear panel Fibre Channel port attached to the physical disk drive or mirror port: DP-1A
exch_type - Frame exchange type: 2 [0x02]
port - HSV200 controller internal Fibre Channel port number attached to the physical disk drive or mirror port: 0 [0x00]

2. ENCLOSURE 3
dencl_num - Enclosure where the physical disk drive is located: 3 [0x03]

3. AL_PA 89
al_pa - AL_PA of the physical disk drive or mirror port: 89 [0x59]

4. RACK NUMBER 0
rack_num - Rack where physical disk drive is located: 0 [0x00]

5. DISK BAY 9
bay - Enclosure bay where the physical disk drive is located: 9 [0x09]
fed_class - Fibre Channel Exchange Descriptor class: 33554476 [0x0200002c]

[Schwarzie]

Hi,

We have been experiencing the same entries in our EVA event logs for some time. After having HP Support investigate the issue it was determined that it is a known bug with specific types of fibre drive firmware which will be installed in your EVA. HP confirmed to us however that these errors are benign and do not pose a risk to our data. We had more of a problem trying to get that officially confirmed in writing however ;-)

For your own peace of mind, I would recommend you collect all the EVA controller event logs and open a new case with HP for them to investigate. Until they get back to you with more information I wouldn't worry too much about these entries.

[Cvetan Grigorov]

Hi, we have EVA6000 and have the same problem. But the problem is isolated only in encloser 3.
This problem started since we upgraded our EVA, and filled all the bays with disk drives. The failed disks are random but only within the bay3.
I also see the events EIP17, EIP05 and EIP13.

I think that we must replace the encloser. I don't think that there is other solution.
I dont think that this is a firmware bug. It happens with old and the new disk drives.

[Víctor Cespón]

One of the new disks may be causing problems on the loop and interrupting the coomunication to the other disks.

You need to contact HP and get someone with experience to analyze the logs, to see which disks show "missing data" or "dropped frames" and if they are all on the same loop.