System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Event Monitor complains about EMC CX600 paths

 
SOLVED
Go to solution
Tom Brown
Advisor

Event Monitor complains about EMC CX600 paths

HP engineers have installed fibre channel HBA's in all of our L-class HP-UX servers, and they are using an EMC SAN based on CX600 equipment.

Every day at 3:00 p.m., my server gets four SERIOUS event notifications. Each notification
appears to reference one of four possible paths to the SAN resources, all of which appear to be functioning perfectly.

Since I can administer space on the SAN via LVM commands without error, and since I can use filesystems mounted on logical volumes managed by LVM without any data errors whatsoever, I have a feeling that this is an issue with the Event Monitor or some other component of HP-UX, rather than an indication of a hardware problem of any kind.

Does anyone have an idea as to what might be going on here?

A recent notification:
>------------ Event Monitoring Service Event Notification ------------<

Notification Time: Tue Jan 20 15:00:24 2004

maximus sent Event Monitor notification information:

/storage/events/disks/default/0_2_0_0.101.6.239.0.0.0
is >= 3.
Its current value is SERIOUS(4).



Event data from monitor:

Event Time..........: Tue Jan 20 15:00:24 2004
Severity............: SERIOUS
Monitor.............: disk_em
Event #.............: 100472
System..............: maximus

Summary:
Disk at hardware path 0/2/0/0.101.6.239.0.0.0 : Device connectivity or hardware failure


Description of Error:

The device is not in a state where it can process requests. Manual
intervention is required.

Probable Cause / Recommended Action:

The device may have been powered off and may be being powered on.
Alternatively, one or both of the terminators on the SCSI bus may be
missing. Install the terminators in their proper locations at the ends of
the SCSI bus.

Alternatively, the SCSI cable may have become detached from the device.
Re-attach the cable.

Alternatively, the SCSI cable may have failed. Replace it.

Alternatively, the device may be in a state where it could not process
this, or any, request. Cycle power to the device.

Alternatively, there could be more than one device having the same address
on the SCSI bus. Make all the addresses on the SCSI bus unique.

Alternatively, the total length of all cable segments on the SCSI bus
exceeds 25 meters. Replace one or more cable segments until the total
length is less than this value.

Alternatively, if all of the above fail to correct the problem, the device
has experienced a hardware failure. Contact your HP support representative
to have the device checked.

Alternatively, if messages corresponding to this condition appear in the
log for more than one device on the SCSI bus, the device adapter may be in
a state from which it cannot extract itself. Perform a system shutdown,
cycle power to the computer and wait for it to reboot.

If, after reboot, messages corresponding to this condition continue to
appear in the log for this SCSI bus, contact your HP support
representative to have the adapter checked.

Additional Event Data:
System IP Address...: 10.50.16.237
Event Id............: 0x400d88d800000000
Monitor Version.....: B.01.00
Event Class.........: I/O
Client Configuration File...........:
/var/stm/config/tools/monitor/default_disk_em.clcfg
Client Configuration File Version...: A.01.00
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):
None
Additional System Data:
System Model Number.............: 9000/800
OS Version......................: B.11.00
STM Version.....................: A.30.00
EMS Version.....................: A.03.20
Latest information on this event:
http://docs.hp.com/hpux/content/hardware/ems/scsi.htm#100472

v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S -v-v-v-v-v-v-v-v-v-v-v-v

Component Data:
Physical Device Path...: 0/2/0/0.101.6.239.0.0.0
Device Class...........: Disk
Inquiry Vendor ID......: DGC
Inquiry Product ID.....: CX600WDR5
Firmware Version.......: HP03
Serial Number..........: APM00035102924

Product/Device Identification Information:

Logger ID.........: disc30; sdisk
Product Identifier: Disk
Product Qualifier.: DGC CX600WDR5
SCSI Target ID....: 0x00
SCSI LUN..........: 0x00

SCSI Command Data Block:

Command Data Block Contents:
0x0000: 00 00 00 00 00 00

Command Data Block Fields (6-byte fmt):
Command Operation Code...(0x00)..: TEST UNIT READY
Logical Unit Number..............: 0

Hardware Status: (not present in log record).

SCSI Sense Data:

Undecoded Sense Data:
0x0000: 70 00 02 00 00 00 00 0A 00 00 00 00 04 03 00 00
0x0010: 00 00

SCSI Sense Data Fields:
Error Code : 0x70
Segment Number : 0x00
Bit Fields:
Filemark : 0
End-of-Medium : 0
Incorrect Length Indicator : 0
Sense Key : 0x02
Information Field Valid : FALSE
Information Field : 0x00000000
Additional Sense Length : 10
Command Specific : 0x00000000
Additional Sense Code : 0x04
Additional Sense Qualifier : 0x03
Field Replaceable Unit : 0x00
Sense Key Specific Data Valid : FALSE
Sense Key Specific Data : 0x00 0x00 0x00

Sense Key 0x02, NOT READY, indicates that the logical unit addressed
cannot be accessed. Operator intervention may be required to correct
this condition.

The combination of Additional Sense Code and Sense Qualifier (0x0403)
indicates: Logical unit not ready, manual intervention required.


>---------- End Event Monitoring Service Event Notification ----------<
It is better to be rich and healthy than poor and sick.
7 REPLIES 7
Highlighted
Julian Perez_1
Valued Contributor

Re: Event Monitor complains about EMC CX600 paths

Hello,

EMC disks are not supported by EMS diagnostics, and for this reason these are not able to manage some events generated by this array.

In order to avoid these messages, solution is using file

/var/stm/data/tools/monitor/disabled_instances

adding EMC instances. Instructions are in same file.

Best regards,

Julian

Keep the faith
Donny Jekels
Respected Contributor

Re: Event Monitor complains about EMC CX600 paths

you need to configure EMS with the correct data file for the EMC disks. do not disable it.

find the EMS configuration file for other SCSI disks and then configure EMS to use it.
"Vision, is the art of seeing the invisible"
Tom Brown
Advisor

Re: Event Monitor complains about EMC CX600 paths

Thank you both for your *fast* responses. I'm looking at Julian's advice this morning.

Donny, if EMS does not support EMC SANs at all, where on earth would one find the "correct" data file? To me, "no support" would mean, "Why bother?"
It is better to be rich and healthy than poor and sick.
Andrew Merritt_2
Honored Contributor
Solution

Re: Event Monitor complains about EMC CX600 paths

Hi,
The problem you have is due to the fact that the EMC disks identify themselves (for historical reasons) as having a firmware level of HP03, which the older versions of the OnlineDiags took to mean that they were HP supported drives. You have a fairly old version of the OnlineDiags (A.30.00 is HWE0203, March 2002. See http://www.docs.hp.com/hpux/onlinedocs/diag/stm/stm_upd.htm#table ).

There is a fix for this problem in the HWE0309 (September 2003, A.42.00) release of the OnlineDiags. You should install this and the latest patch, PHSS_29563, and that should stop the events for the EMC drives.

Unfortunately, the release notes don't mention the fact that this problem is fixed, but it is covered by CR JAGae75278.

Andrew
Andrew Merritt_2
Honored Contributor

Re: Event Monitor complains about EMC CX600 paths

There is also a work-around until you can update the OnlineDiags:
To stop the events being generated, add the paths to the
disabled_instances file.

Run /etc/opt/resmon/lbin/moncheck to determine the names of instances
you would like to disable. (You can direct the output to a file).

With editor of your choice, Add/delete/modify instances
in /var/stm/data/tools/monitor/disabled_instances.
You will need to add a line for each disk or LUN.

Example:
Add the following to the disabled_instances file

/storage/events/disks/default/1_0_8_0_0.1.22*
/storage/events/disks/default/0_0_8_0_0.1.16*

Note a wildcard was used here * this means all devices
starting with 1_0_8_0_0.1.22 and 0_0_8_0_0.1.16 will
be disabled.

Run monconfig: Select the "E)nable Monitoring" command

Wait for monitoring to be re-enabled

Do a C)heck monitors
The resource class that you disabled should show up in the list with no
description of what to do with events. (ie. there are no monitoring
requests)

This work-around has the effect of blocking all EMS events for the listed device paths. Since these disks are not supported by the EMS Hardware Monitors, you are not losing any functionality by doing this.

Andrew
Tom Brown
Advisor

Re: Event Monitor complains about EMC CX600 paths

I would like to thank all of your for your quick, informative answers! This will help us solve our problems with several large HP-UX systems going on this SAN.
It is better to be rich and healthy than poor and sick.
Steven E. Protter
Exalted Contributor

Re: Event Monitor complains about EMC CX600 paths

I have the same problem.

If you don't have a disk configured on LUN0, your system detects the disk array itself and thinks its a disk.

Since its not, EMS complains about errors.

If there is a LUN configured on LUN0 of each fiber connection this message will stop happening.

It can safely be ignored or supressed as noted above.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com