EMS (bogus) error - Clariion EMC array

[Serge Poitras]

Hello HP community,

We seem to have a somewhat mysterious matter on our hands...a few months ago, we started to get logged events in syslog.log that ressembled this:

Aug 28 01:33:17 aph092 EMS [4168]: ------ EMS Event Notification ------ Value: "SERIOUS (4)" for Resource: "/storage/events/disks/default/0_6_2_0.32.16.0.0.0.6" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 273154138 -r /storage/events/disks/default/0_6_2_0.32.16.0.0.0.6 -n 273159025 -a

So we ran the command as stated, the output was:

root @ aph092:HP-UX/11.11[/var/adm/syslog]
# /opt/resmon/bin/resdata -R 273154138 -r /storage/events/disks/default/0_6_2_0.32.16.0.0.0.6 -n 273159025 -a

CURRENT MONITOR DATA:

Event Time..........: Fri Aug 28 01:33:17 2009
Severity............: SERIOUS
Monitor.............: disk_em
Event #.............: 100776
System..............: aph092.wheaton.com

Summary:
Disk at hardware path 0/6/2/0.32.16.0.0.0.6 : Software configuration error


Description of Error:

The device was unsuccessful in processing the current request message
because the message contained an invalid field in the command data block.
The request was not processed.

Probable Cause / Recommended Action:

The error most likely indicates that the device is not fully supported by
the current driver. This may or may not cause a problem in the operation
of the device.

Additional Event Data:
System IP Address...: 10.11.242.92
Event Id............: 0x4a976c1d00000000
Monitor Version.....: B.01.01
Event Class.........: I/O
Client Configuration File...........:
/var/stm/config/tools/monitor/default_disk_em.clcfg
Client Configuration File Version...: A.01.00
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):
None
Additional System Data:
System Model Number.............: 9000/800/A500-7X
OS Version......................: B.11.11
STM Version.....................: A.52.00
EMS Version.....................: A.04.20
Latest information on this event:
http://docs.hp.com/hpux/content/hardware/ems/scsi.htm#100776

v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v



Component Data:
Physical Device Path...: 0/6/2/0.32.16.0.0.0.6
Device Class...........: Disk
Inquiry Vendor ID......: DGC
Inquiry Product ID.....: CX3-80WDR5
Firmware Version.......: HP03
Serial Number..........: tCsFMIpSLJBRwRBB

Product/Device Identification Information:

Logger ID.........: disc30; sdisk
Product Identifier: Disk
Product Qualifier.: DGC CX3-80WDR5
SCSI Target ID....: 0x00
SCSI LUN..........: 0x06

SCSI Command Data Block:

Command Data Block Contents:
0x0000: 1A 00 1C 00 FF 00

Command Data Block Fields (6-byte fmt):
Command Operation Code...(0x1A)..: MODE SENSE
Logical Unit Number..............: 0
DBD Bit..........................: 0
Page Control Bits................: 0
Page Code Bits...................: 28
Allocation Length................: 255 (0xFF)

Hardware Status: (not present in log record).

SCSI Sense Data:

Undecoded Sense Data:
0x0000: 70 00 05 00 00 00 00 0A 00 00 00 00 24 00 00 00
0x0010: 00 00

SCSI Sense Data Fields:
Error Code : 0x70
Segment Number : 0x00
Bit Fields:
Filemark : 0
End-of-Medium : 0
Incorrect Length Indicator : 0
Sense Key : 0x05
Information Field Valid : FALSE
Information Field : 0x00000000
Additional Sense Length : 10
Command Specific : 0x00000000
Additional Sense Code : 0x24
Additional Sense Qualifier : 0x00
Field Replaceable Unit : 0x00
Sense Key Specific Data Valid : FALSE
Sense Key Specific Data : 0x00 0x00 0x00

Sense Key 0x05, ILLEGAL REQUEST, indicates that there was an illegal
parameter in the command data block or in the additional parameters
supplied as data for some commands (FORMAT UNIT, SEARCH DATA, etc.).
This sense key may also indicate that an invalid IDENTIFY message was
received.

The combination of Additional Sense Code and Sense Qualifier (0x2400)
indicates: Invalid field in cdb.

Background info:

OS: HP-UX 11.11
H/W: rp2470 (a.k.a A500)
patch list: see attachment
Support Tools Manager revision: B.11.11.17.02


Clariion array with multiple drives already in operation (approx. 20 or so), but only one displays this behavior. Firmware levels for all the array drives is the same across the board:

# /usr/local/emc/inq -no_dots
Inquiry utility, Version V7.3-425 (Rev 0.0) (SIL Version V5.2.0.0 (Edit Level 425)
Copyright (C) by EMC Corporation, all rights reserved.
For help type inq -h.



------------------------------------------------------------------------
DEVICE :VEND :PROD :REV :SER NUM :CAP(kb)
------------------------------------------------------------------------
/dev/rdsk/c0t2d0 :HP :DVD-ROM 305 : :---- : -----
/dev/rdsk/c1t15d0 :HP 36.4G:ST336753LC :HPC3 :3HX0T1VE :35566480
/dev/rdsk/c3t15d0 :HP 36.4G:ST336753LC :HPC3 :3HX0NMKE :35566480
/dev/rdsk/c32t0d0 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :127926272
/dev/rdsk/c32t0d1 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :127926272
/dev/rdsk/c32t0d2 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :14680064
/dev/rdsk/c32t0d3 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :114294784
/dev/rdsk/c32t0d4 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :127926272
/dev/rdsk/c32t0d5 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :10485760
/dev/rdsk/c32t0d6 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :31457280
/dev/rdsk/c32t0d7 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :5242880
/dev/rdsk/c32t1d0 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :52428800
/dev/rdsk/c32t1d1 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :10485760
/dev/rdsk/c33t0d0 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :127926272
/dev/rdsk/c33t0d1 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :127926272
/dev/rdsk/c33t0d2 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :14680064
/dev/rdsk/c33t0d3 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :114294784
/dev/rdsk/c33t0d4 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :127926272
/dev/rdsk/c33t0d5 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :10485760
/dev/rdsk/c33t0d6 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :31457280
/dev/rdsk/c33t0d7 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :5242880
/dev/rdsk/c33t1d0 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :52428800
/dev/rdsk/c33t1d1 :DGC :CX3-80WDR5 :HP03 :Ch2 CONT :10485760

Ioscan identifies this device file:
disk 267 0/6/2/0.32.16.0.0.0.6 sdisk CLAIMED DEVICE DGC CX3-80WDR5
/dev/dsk/c32t0d6 /dev/rdsk/c32t0d6

So in essence, we are out of options as to determine why this one drive generates these errors while no other in that array (RAID5) displays the same symptoms ?

Any leads will be appreciated.

Best regards to all,
Serge

[Fabio Ettore]

Hi,

issue

# cstm

cstm>map

Do you have Unknown under column Product and on lines of EMC disks?

Best regards,
Fabio

[Joshua Scott]

I've seen this often with Clariion arrays attached to HP-UX.

See this thread:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1060991

[Serge Poitras]

Fabio, here is the output of the command you suggested:
(no unknown devices listed)

cstm>map
aph092

Dev Last Last Op
Num Path Product Active Tool Status
=== ==================== ========================= =========== =============
1 system system () Information Successful
2 0 Bus Adapter (582)
3 0/0 PCI Bus Adapter (782)
4 0/0/0/0 Core PCI 100BT Interface
5 0/0/1/0 PCI SCSI Interface (10000
6 0/0/1/0.2.0 Optical Storage Device (H
7 0/0/1/1 PCI SCSI Interface (10000
8 0/0/1/1.15.0 SCSI Disk (HP36.4GST33675
9 0/0/2/0 PCI SCSI Interface (10000
10 0/0/2/1 PCI SCSI Interface (10000
11 0/0/2/1.15.0 SCSI Disk (HP36.4GST33675
12 0/0/4/1 RS-232 Interface (103c104
13 0/2 PCI Bus Adapter (782)
14 0/2/0/0 PCI Bus Adapter (8086b154
15 0/2/0/0/4/0 PCI 4 Port 100BT LAN (101
16 0/2/0/0/5/0 PCI 4 Port 100BT LAN (101
17 0/2/0/0/6/0 PCI 4 Port 100BT LAN (101
18 0/2/0/0/7/0 PCI 4 Port 100BT LAN (101
19 0/4 PCI Bus Adapter (782)
20 0/4/0/0 PCI Gigabit Ethernet Link
21 0/6 PCI Bus Adapter (782)
22 0/6/0/0 Fibre Channel Interface (
23 0/6/0/0.31 Fibre Channel Driver (Mas
24 0/6/0/0.31.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
25 0/6/0/0.31.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
26 0/6/0/0.31.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
27 0/6/0/0.31.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
28 0/6/0/0.31.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
29 0/6/0/0.31.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
30 0/6/0/0.31.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
31 0/6/0/0.31.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
32 0/6/0/0.31.16.0.0.1. SCSI Disk (DGCCX3-80WDR5)
33 0/6/0/0.31.16.0.0.1. SCSI Disk (DGCCX3-80WDR5)
34 0/6/0/0.31.16.255.0. SCSI Device (DGCCX3-80)
35 0/6/2/0 Fibre Channel Interface (
36 0/6/2/0.32 Fibre Channel Driver (Mas
37 0/6/2/0.32.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
38 0/6/2/0.32.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
39 0/6/2/0.32.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
40 0/6/2/0.32.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
41 0/6/2/0.32.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
42 0/6/2/0.32.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
43 0/6/2/0.32.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
44 0/6/2/0.32.16.0.0.0. SCSI Disk (DGCCX3-80WDR5)
45 0/6/2/0.32.16.0.0.1. SCSI Disk (DGCCX3-80WDR5)
46 0/6/2/0.32.16.0.0.1. SCSI Disk (DGCCX3-80WDR5)
47 0/6/2/0.32.16.255.0. SCSI Device (DGCCX3-80)
48 8 MEMORY (9b)
49 160 CPU (5e3) Information Successful
50 162 CPU (5e3)

[Serge Poitras]

Joshua,

Read that link, and it directed me in the right direction...since the SAN boys here are at a lost as to why this monitored event keeps cropping up, we decided to take out of OS monitoring, since they do their own monitoring on the SAN anyways.

Points awarded to all,
Thanks very much to all.

[Serge Poitras]

As per my last post, we took out the monitoring key for that specific device, and will let the SAN boys keep an eye on it from their side.

Points have been awarded to all who participated.

Thanks again,
Cheers, Serge