Excessive I/O warnings in Syslog

[E. Vodegel]

Hi,

I've got a N4000 server combined with 2 FC10 disc cabinets with HP-ux 11.0 on it. Everything is fibre channel connected. I get several messages in my syslog.

A part of the syslog.log (16/01):
Jan 16 23:55:37 osloprod vmunix: DIAGNOSTIC SYSTEM WARNING:
Jan 16 23:55:37 osloprod vmunix: The diagnostic logging facility has started re
ceiving excessive
Jan 16 23:55:37 osloprod vmunix: errors from the I/O subsystem. I/O error entr
ies will be lost
....
Jan 16 23:58:03 osloprod EMS [2241]: ------ EMS Event Notification ------ Value:
"0.89552" for Resource: "/system/kernel_resource/system_v_ipc/semaphore/semmni"
(Threshold: != "0") Execute the following command to obtain event details:
/opt/resmon/bin/resdata -R 1068957699 -r /system/kernel_resource/system_v_ipc/se
maphore/semmni -a
Jan 16 23:58:03 osloprod EMS [2241]: ------ EMS Event Notification ------ Value:
"10.398" for Resource: "/system/kernel_resource/system_v_ipc/semaphore/semmns"
(Threshold: != "0") Execute the following command to obtain event details:
/opt/resmon/bin/resdata -R 1068957700 -r /system/kernel_resource/system_v_ipc/sem
aphore/semmns -a
Jan 16 23:57:14 osloprod : su : + tty?? root-oracle
Jan 16 23:58:03 osloprod EMS [2241]: ------ EMS Event Notification ------ Value:
"0.0993" for Resource: "/system/kernel_resource/system_v_ipc/message/msgmni"
(Threshold: != "0")
Jan 16 23:58:03 osloprod EMS [2241]: ----- EMS Monitor Restart due to system reboo
t ----- Title: Kernel Resource Monitor Command: /etc/opt/resmon/lbin/krmond
Vendor: Hewlett-Packard Co. Version: A.11.00.04 To obtain a list of curr
ently monitored resources, execute the following: /opt/resmon/bin/resdata -M 3690
108990
Jan 17 00:00:25 osloprod vmunix: DIAGNOSTIC SYSTEM WARNING:
Jan 17 00:00:25 osloprod vmunix: The diagnostic logging facility is no longer r
eceiving excessive
Jan 17 00:00:25 osloprod vmunix: errors from the I/O subsystem. 44 I/O error e
ntries were lost.


A part of the syslog (17/01):
Jan 17 04:53:44 osloprod vmunix: DIAGNOSTIC SYSTEM WARNING:
Jan 17 04:53:44 osloprod vmunix: The diagnostic logging facility has started re
ceiving excessive
Jan 17 04:53:44 osloprod vmunix: errors from the I/O subsystem. I/O error entr
ies will be lost
Jan 17 04:53:44 osloprod vmunix: until the cause of the excessive I/O logging i
s corrected.
Jan 17 04:53:44 osloprod vmunix: If the diaglogd daemon is not active, use the
Daemon Startup command
Jan 17 04:53:44 osloprod vmunix: in stm to start it.
Jan 17 04:53:44 osloprod vmunix: If the diaglogd daemon is active, use the logt
ool utility in stm
Jan 17 04:53:44 osloprod vmunix: to determine which I/O subsystem is logging ex
cessive errors.
Jan 17 04:53:44 osloprod vmunix: DIAGNOSTIC SYSTEM WARNING:
Jan 17 04:53:44 osloprod vmunix: The diagnostic logging facility is no longer r
eceiving excessive
Jan 17 04:53:44 osloprod vmunix: errors from the I/O subsystem. 10 I/O error e
ntries were lost.

The last error messages appear every second.

Yesterday a HP engineer arrived and has changed the Fibre card of the FC10 disc cabinet (There are 2 on every disc cabinet). Now the system boots, but give the same error messages.

Then I installed 1 patch (PHK_23939). This wasn't the solution.

Then I installed the september 2001 Quality Pack and the December 2001 Hardware Critical Patches). But all of this does not seem to work.

Does somebody know a solution.

I have attached an output of the syslog and an ioscan. I see there are "no_hw" messages in the ioscan output.

Kind regards,
Erwin

[Stefan Farrelly · Accepted solution]

Hi,

if youre getting excessive I/O warnings from the online diags (STM) then you definitely have a hardware problem. Youre getting so many errors that it cant log them all.

If you can see NO_HW messages from ioscan this simply confirms you have a hardware connection problem to whatever is coming up as NO_HW. Call your HP engineers and get them back onsite and dont let them leave until its fixed.

[Albert Smith_1]

Are you receiving any LBOLTS in your syslog? If so you will need to contact HP for a resolve. Also check your DMESG it too can give you some valuable information.

I would also suggest running diagnostics on your bus's and your disk's then running logtool on those devices.

Once you run logtool it will give you a better view of which device is causing the problem.

[Albert Smith_1]

Erwin,

I read your syslog the first thing I have to say is upgrade your diagnostics and your EMS. You are at a back leveld version.

Second - What kind of disk array are you attaching to?

Are you in a Service Guard Config?

[E. Vodegel]

Hello,

What we have done:

First get a HP engineer.

We have installed patch PHSS_23996
mv /etc/lvmtab /etc/lvmtab.old
vgscan -v
vgchange -a y
shutdown -ry 0

All the LCC's (Fibre adapters)firmware updated. It seemed there was 1 fibre cable not inserted well. This was probably the one that was giving excessive errors.

Thanks