HP-UX 11.0 I/O error

[Cindy Huang]

I just reboot the HP-UX 11.0 box over the weekend, and it's showing The diagnostic logging facility has started receiving excessive errors from the I/O subsystem.

/var/adm/syslog/syslog.log
May 31 10:48:59 scmk01 vmunix: DIAGNOSTIC SYSTEM WARNING:
May 31 10:48:59 scmk01 vmunix:
May 31 10:48:59 scmk01 vmunix: The diagnostic logging facility has started receiving excessive
May 31 10:48:59 scmk01 vmunix: errors from the I/O subsystem. I/O error entries will be lost
May 31 10:48:59 scmk01 vmunix: until the cause of the excessive I/O logging is corrected.
May 31 10:48:59 scmk01 vmunix: If the diaglogd daemon is not active, use the Daemon Startup command
May 31 10:48:59 scmk01 vmunix: in stm to start it.
May 31 10:48:59 scmk01 vmunix: If the diaglogd daemon is active, use the logtool utility in stm
May 31 10:48:59 scmk01 vmunix: to determine which I/O subsystem is logging excessive errors.

The box seems funcation ok so far. Is there anything I should check, or any patch I could apply for this warning?

Thank you,
Cindy Huang

[Michael Tully]

Are there any other messages being received, like scsi lbolt, resets etc?

You could install a later set of 'STM' patches for your system, but overall perhaps you should look at the entire patching scenario. When was the last time you patched your system?

[Mel Burslan]

looks like your server either does not have stm installed or configured not to run at startup.

if I were you I would make it run one way or the other.

IO subsystem errors are usually associated with SCSI or Fibre Channel adapters. So, you may be having a problem with storage.

run
ioscan -funC disk
and see if anything reporting as NO_HW

and if you have a maintenence contract, have an HP Tech check your server just to make sure.

[Cindy Huang]

I don't see messages like scsi lbolt or resets in syslog.log. However, I have attached the log file to this message.

The last time we have patched this system was September last year.

What are the new patches I should get to solve this problem?

[Victor Fridyev]

Hi,

Install the newest version of DIAGNOSTIC. Sometimes old versions give wrong messages. Additionally check the existing diagnostic settings via SAM.

HTH

[Steven E. Protter]

You lan card has recently been disconnected.

This can be due to cable failure or switch failure or even failure of the NIC.

All of these things should be checked.

Your fiber connected disk may have had a problem due to disruption of your fiber network or a problem on your disk array.

I'd check with the disk/san admin and see if any events have occured.

Fiber connected disks are treated as scsi devices.

Thats where I'd look.

SEP

[Andrew Merritt_2]

From the syslog, it looks as though the problem is caused by the lan connection:

May 31 10:48:59 scmk01 vmunix: btlan: NOTE: MII Link Status Not OK - Check Cable Connection to Hub/Switch at 0/8/0/0/4/0....
May 31 10:48:59 scmk01 vmunix: btlan: NOTE: MII Link Status Not OK - Check Cable Connection to Hub/Switch at 0/8/0/0/5/0....
May 31 10:48:59 scmk01 vmunix: btlan: NOTE: MII Link Status Not OK - Check Cable Connection to Hub/Switch at 0/8/0/0/6/0....
May 31 10:48:59 scmk01 vmunix: btlan: NOTE: MII Link Status Not OK - Check Cable Connection to Hub/Switch at 0/8/0/0/7/0....
May 31 10:48:59 scmk01 vmunix: DIAGNOSTIC SYSTEM WARNING:
...

These errors have lead to the Excessive I/O error message, because at this stage the OnlineDiags have not yet started up. See a bit later in the file diagmond is started:

May 31 10:49:27 scmk01 diagmond[1589]: started

Before that, the diaglogd process is not ready to process the events from the device drivers, and so the fairly short pipeline gets filled up, leading to the warning. diagmond starts diaglogd.

The events that were in the pipeline should get processed once diaglogd is running, so you should see them when you run 'logtool' in STM (as the warning suggests you should).

The message that says the logging is now working is not output until the next time an event is generated by a device driver, so if the system is behaving itself you might not see it for some time.