Syslog full of EMS Event Notification

[Vladimir Berezhnoy]

After we had a EMC SAN reconfiguration recently the syslog is being flooded with messages from EMS (this is just a single one of them)

Jun 17 04:13:43 dbserver EMS [3829]: ------ EMS Event Notification ------ Value: "SERIOUS (4)" for Resource: "/storage/events/disks/default/1_
0_8_0_0.1.4.0.0.3.2" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 250938841 -r /storage/events/disks/default/1_0_8_0_0.1.4.0.0.3.2 -n 250938067 -a

I've run all commands I'm aware of to check the status of this disk and see no issues with it. But messages keep appearing every day

# ioscan -fnC disk -H 1/0/8/0/0.1.4.0.0.3.2
Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
disk 346 1/0/8/0/0.1.4.0.0.3.2 sdisk CLAIMED DEVICE DGC CX3-40cWDR10
/dev/dsk/c33t3d2 /dev/rdsk/c33t3d2


# diskinfo /dev/rdsk/c33t3d2
SCSI describe of /dev/rdsk/c33t3d2:
vendor: DGC
product id: CX3-40cWDR10
type: direct access
size: 69206016 Kbytes
bytes per sector: 512


# powermt display dev=all
33 1/0/8/0/0.1.4.0.0.3.2 c33t3d2 SP A5 active alive 0 0

Please advise on a ways I can perform further checking of our disk drives. Or if the disks are really okey how can I disable these messages.

[sujit kumar singh]

hi


are you getting this only for One lUN of the EMC array or also on other LUNS of the same array assigned to the disk through the same fiber card?

Please post

#ioscan -fnCdisk
#ioscan -fnCfc
note the devices /dev/td0 and /dev/td1

#fcmsutil /dev/td0
#fcmsutil /dev/td1
#swlist

regards
sujit

[Michal Kapalka (mikap)]

hi,

at first :

/opt/resmon/bin/resdata -R 250938841 -r /storage/events/disks/default/1_0_8_0_0.1.4.0.0.3.2 -n 250938067 -a

execute this command, if the disk is available,

it means that maybe some HBA card was disconected from SAN, posible cable or HBA failure, or SAN admin plays with the zonning.

powermt check ==> for dead devices.

mikap

[Aneesh Mohan]

Hi ,

Please post the output of

#lssf /dev/dsk/*

Aneesh

[sujit kumar singh]

Hi Mikap!

this is already attached in the question and complaining for driver unsupported.

I assume this needs tobe checked more from the EMC storage config side.
regards
sujit

[Vladimir Berezhnoy]

Thanks everybody for the feedback. First I should note that we are not having any trouble with this server after the maintenance, all lv's seem to be in their places. The only issue is a bunch of messages from EMS everyday.

2sujit
I'm attaching ioscan -fnCdisk

Looks like we get the notifications for every disk installed in the system
# cat syslog.log|grep "Jun 18"|grep EMS|wc -l
192


#ioscan -fnCfc
Class I H/W Path Driver S/W State H/W Type Description
===================================================================
fc 0 1/0/8/0/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
/dev/td0
fc 1 1/0/10/0/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
/dev/td1



# fcmsutil /dev/td0

Vendor ID is = 0x00103c
Device ID is = 0x001029
XL2 Chip Revision No is = 2.3
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x00128c
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x010a00
N_Port Node World Wide Name = 0x50060b0000238615
N_Port Port World Wide Name = 0x50060b0000238614
Driver state = ONLINE
Hardware Path is = 1/0/8/0/0
Number of Assisted IOs = 33015873
Number of Active Login Sessions = 4
Dino Present on Card = NO
Maximum Frame Size = 2048
Driver Version = @(#) libtd.a HP Fibre Channel Tachyon TL/TS/XL2 Driver B.11.11.12 PATCH_11.11 (PHSS_31326) /ux/kern/kisu/TL/src/common/wsio/td_glue.c: Sep 5 2005, 10:14:40

# fcmsutil /dev/td1

Vendor ID is = 0x00103c
Device ID is = 0x001029
XL2 Chip Revision No is = 2.3
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x00128c
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x010a00
N_Port Node World Wide Name = 0x50060b000023860f
N_Port Port World Wide Name = 0x50060b000023860e
Driver state = ONLINE
Hardware Path is = 1/0/10/0/0
Number of Assisted IOs = 32176859
Number of Active Login Sessions = 4
Dino Present on Card = NO
Maximum Frame Size = 2048
Driver Version = @(#) libtd.a HP Fibre Channel Tachyon TL/TS/XL2 Driver B.11.11.12 PATCH_11.11 (PHSS_31326) /ux/kern/kisu/TL/src/common/wsio/td_glue.c: Sep 5 2005, 10:14:40

[Torsten.]

Your diags version is March 2003!!!


First of all, update the diags.

[Vladimir Berezhnoy]

I'm attaching lssf /dev/dsk/*

#swlist

B2491BA B.11.11 MirrorDisk/UX
B3701AA_TRY C.03.70.00 Trial HP GlancePlus/UX Pak for s800 11i
B3913DB C.03.65 HP aC++ Compiler (S800)
B3929CA B.11.11.03.03 HP OnLineJFS
B3935DA A.11.14 MC / Service Guard
B5139DA B.01.09 Enterprise Cluster Master Toolkit
B5140BA A.11.11.02 MC/ServiceGuard NFS Toolkit
B5725AA B.4.2.110 HP-UX Installation Utilities (Ignite-UX)
B5736DA A.03.20.01 HA Monitors
B6826AA B.11.11.01.06 Partition Manager - HP-UX
B6960MA A.05.10 HP OpenView Storage Data Protector
B7609BA A.03.20.01 Event Monitoring Service
B8324BA B.01.04 HP Cluster Object Manager
B8339BA A.02.05.01 HP-UX ServiceControl Manager
B8724AA A.01.08 CIFS/9000 Client
B8725AA A.01.08 CIFS/9000 Server
B8843CA A.02.00 HP-UX Workload Manager
B9073AA B.05.01 HP-UX iCOD-purchase (Instant Capacity on Demand - purchase)
B9788AA 1.3.1.02.01 Java 2 SDK 1.3 for HP-UX (700/800), PA1.1 + PA2.0 Add On
BUNDLE B.2008.10.03 Patch Bundle
BUNDLE11i B.11.11.0102.2 Required Patch Bundle for HP-UX 11i, February 2001
Base-VXVM B.03.50.5 Base VERITAS Volume Manager Bundle 3.5 for HP-UX
CDE-English B.11.11 English CDE Environment
DP_PATCH_BUNDLE_I B.11.11 DP Patch Bundle I
FDDI-00 B.11.11.02 PCI FDDI;Supptd HW=A3739A/A3739B;SW=J3626AA
FEATURE11-11 B.11.11.0209.5 Feature Enablement Patches for HP-UX 11i, Sept 2002
FibrChanl-00 B.11.11.09 PCI/HSC FibreChannel;Supptd HW=A6684A,A6685A,A5158A,A6795A
GOLDAPPS11i B.11.11.0712.475 Applications Patches for HP-UX 11i v1, December 2007
GOLDBASE11i B.11.11.0712.475 Base Patches for HP-UX 11i v1, December 2007
GigEther-00 B.11.11.14 PCI/HSC GigEther;Supptd HW=A4926A/A4929A/A4924A/A4925A;SW=J1642AA
GigEther-01 B.11.11.07 PCI GigEther;Supptd HW=A6794A/A6825A/A6847A
HPUX11i-OE B.11.11.0303 HP-UX 11i Operating Environment Component
HPUXBase64 B.11.11 HP-UX 64-bit Base OS
HPUXBaseAux B.11.11.0303 HP-UX Base OS Auxiliary
HWEnable11i B.11.11.0612.458 Hardware Enablement Patches for HP-UX 11i v1, December 2006
IEther-00 B.11.11.03 PCI Ethernet;Supptd HW=A6974A
Ignite-UX-11-11 B.4.2.110 HP-UX Installation Utilities for Installing 11.11 Systems
J4189-11001C E.10.34 Hewlett-Packard JetDirect Printer Installer for Unix
KRMonitor B.11.11.04 EMS Kernel Resource Monitor
OnlineDiag B.11.11.10.11 HPUX 11.11 Support Tools Bundle, Mar 2003
RAID-00 B.11.11.01 PCI RAID; Supptd HW=A5856A
ShadowPassword B.11.11.02 HP-UX 11.11 Shadow Password Bundle
T1335AC A.02.02.00 HP-UX Virtual Partitions
T1456AA 1.4.2.08.02 Java2 1.4 SDK for HP-UX
T1471AA A.05.10.006 HP-UX Secure Shell
perl B.5.6.1.C Perl Programming Language
#
# Product(s) not contained in a Bundle:
#

Auxiliary-Opt B.11.11.06 Auxiliary Optimizer for HP Languages.
DDE B.11.11.06 Distributed Debugging Environment DDE 4.26
EMCpower HP.5.1.0_b160 PowerPath
HP_LTT46 4.6.0.0 Library & Tape Tools - HP-UX
NAVIAGENT 6.26.7.0.81 Navisphere Disk Array Management Tool (AGENT)
NAVICLI 6.26.7.0.81 Navisphere Disk Array Management Tool (CLI)
PHSS_26558 1.0 linker startup code / SLLIC ELF support
PHSS_29143 1.0 OV DP5.10 patch - DOC packet
SYMCLI V6.5.1.9 EMC Data Storage Systems Private Limited
SYMCLI V6.5.1.9 EMC Data Storage Systems Private Limited
VNC 4.1.1
WLM-Toolkits A.01.03 HP-UX Workload Manager Toolkits
admsnap V2.26.0.0.4 admsnap
expat 2.0.1 expat
fontconfig 2.6.0 fontconfig
freetype 2.3.7 freetype
gettext 0.17 gettext
ghostscript 8.62.0 ghostscript
gzip 1.3.12 gzip
jpeg 6b jpeg
libiconv 1.12 libiconv
libpcap 1.0.0 libpcap
libpng 1.2.29 libpng
nmap 4.76 nmap
openssl 0.9.8j openssl
pcre 7.8 pcre
popt 1.7 popt
rsync 2.6.9 rsync
sudo 1.6.9p11 sudo
zip 2.3 zip
zlib 1.2.3 zlib

[Torsten.]

OnlineDiag B.11.11.10.11 HPUX 11.11 Support Tools Bundle, Mar 2003

New version:

http://h20293.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=B6191AAE

[Aneesh Mohan]

Hi Vladimir,

As Torsten suggested you may need to upgrade STM version.

Also ,I belive you have alot of unnecessary old device files in your system,you may need to house keep the same.

#lssf /dev/dsk/* |grep "???" |awk 'NF {print $NF}' ---> to listing old files

Regards,
Aneesh

[sujit kumar singh · Accepted solution]

hi,

you should upgarde the STM as suggested.
Your FC driver patch seems to be updated.
but the HW enablement bundle is Old.

in the syslog are you getting the errors for this particular disk only?

/dev/rdsk/c33t3d2
/dev/rdsk/c32t3d2
/dev/rdsk/c24t3d2
/dev/rdsk/c26t3d2
these seem to be the same disk , but /dev/rdsk/c33t3d2 -- has the HW path of 1/0/8/0/0.1.4.0.0.3.2 -- that is == /dev/td0.


are you getting that message for other LUNS that is other disks and also other paths like c24 , c26 and c32 also or only for c33.


regards
sujit

[Vladimir Berezhnoy]

Thanks for reply to all. The software upgrade is not an easy option to perform as we'll need a lot of approvals for this, and as far as all of this was working fine before upgrade and now there are no notable issues I will leave the sw upgrade as a last option.

2Aneesh
how am I supposed to perform a safe cleanup of old device nodes for nonexistent disks in /dev/? I mean thise which arwe with ???

2sujit
we get 192 of such messages every day, I believe for all of disks present in the system.
I'm attaching the part of syslog.log for today


This is a part from 'powermt display'. We have 48 of such LUNs. And 48x4 gives 192 messages in syslog a day.

CLARiiON ID=APM00081100775 [CLUDB01]
Logical device ID=600601603CE11A0046A31659002BDE11 [LUN 36]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP A, current=SP A Array failover mode: 1
==============================================================================
---------------- Host --------------- - Stor - -- I/O Path - -- Stats ---
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
24 1/0/10/0/0.1.0.0.0.3.2 c24t3d2 SP A4 active alive 0 0
26 1/0/10/0/0.1.4.0.0.3.2 c26t3d2 SP B5 active alive 0 0
32 1/0/8/0/0.1.0.0.0.3.2 c32t3d2 SP B4 active alive 0 0
33 1/0/8/0/0.1.4.0.0.3.2 c33t3d2 SP A5 active alive 0 0

[Torsten.]

Keep in mind almost every patch say about fixed problems "... it can happen ..." but not "it will happen ...".

So if you are unlucky, you get the problems fixed by a certain patch. With some luck, you don't.

BTW, some of your software components (e.g. the diags) are unsupported versions anyway.

However, IMHO the multipathing software is more suspect.

[Aneesh Mohan]

>>2Aneesh
how am I supposed to perform a safe cleanup of old device nodes for nonexistent disks in /dev/? I mean thise which arwe with ???

1)
rm $(lssf /dev/dsk/* |grep "???" |awk 'NF {print $NF}')
2)
rm $(lssf /dev/rdsk/* |grep "???" |awk 'NF {print $NF}')

Regards,
Aneesh

[Vladimir Berezhnoy]

Thanks Torsten for pointing me to EMC. This is what i found in their kb:

ID: emc1244
Usage: 149
Date Created: 11/09/2000
Last Modified: 09/16/2008
STATUS: Approved
Audience: Customer

Knowledgebase Solution



Question: Why do I see EMS error messages for hardware paths associated with the CLARiiON array in the HP-UX syslog file?
Environment: OS: HP-UX
Environment: Product: CLARiiON FC-Series
Problem: The HP-UX system log, /var/adm/syslog/syslog.log, shows Event Monitor Service (EMS) error messages.
Problem: Diagnostic System Messages in the syslog file during boot.
Problem: Error msg like: EMS Event Notification ------ Value: "SERIOUS (4)" for Resource: "/storage/events/disks/default/8_0_1_0.98.27.19.0.0.4" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 121176082 -r /storage/events/disks/default/8_0_1_0.98.27.19.0.0.4 -n 121176200 -a
Fix: EMS does not have a valid monitoring process for the more recent EMC (CLARiiON) arrays and instead tries to use generic disk monitoring routines to interpret what it detects. As a result, erroneous error messages are generated when:

(1) During boot in the syslog. EMS messages during the boot process occur when HP-UX verifies that both paths to a volume group are valid (if pvlinks is used). The process of sending I/O down 1 path, then down the other path, will cause EMS to think that an error has occurred to trigger the change in the path being used to access the volume group.

(2) A vgchange command or a vgextend command is issued. These commands cause HP-UX to send I/O down each hardware path in turn to verify them. The monitor detects that the paths to the volume group have changed for some reason, and this activity is interpreted and reported as an error.

(3) Certain MC/ServiceGuard commands such as cmgetconf.

(4) When diagnostic utilities are run which probe paths to verify them. These utilities may be part of the OS. These will be seen to happen at a regular interval, usually 24 hours.

If the EMS messages are not accompanied by "Powerfail" messages or LVM "Switch" messages then they can be ignored.

If desired, the EMS configuration can be modified to screen out these messages - please consult HP for details.

If EMS is not installed or is not active, only Diagnostic System Messages may be seen in the syslog file during boot as paths are verified.

[Vladimir Berezhnoy]

Thanks to all for the replies. Fixed this as suggested on EMC kb.
The messages started to appear device paths has changed due to upgrade on EMC storage.


----------------
Fix: HP EMS does not support the EMC disks and therefore cannot handle the information received.

In all cases when such a message is reported, is it recommended that you verify the system log from the host and the storage system. If nothing is found in the host logs or in the CLARiiON logs, then this message can be ignored.

Disable paths to Clariion

EMS Notifications can be removed for each disk by add these to the disabled_instances file. You can use /etc/opt/resmon/lbin/moncheck to list all devices monitored. Then add entries to your /var/stm/data/tools/monitor/disabled_instances file for the devices that you do not want monitored.

For example :

/storage/events/disks/default/0_12_0_0.8.0.0.0.0.1

or

/storage/events/disks/default/0_12_0_0.8*
/storage/events/disks/default/1_4_0_0.8*

After that file is edited, then re-enable monitoring using

/etc/opt/resmon/lbin/monconfig , select "E" to Enable Monitoring.

###############################