diaglogd daemon utilize high CPU

[sam_25_1]

diaglogd daemon is showing 96% CPU utilization in TOP. stopped and started the service but no effect.

dmesg shows
=========================================
sysmap: rmap ovflo, lost [44182l,44184l)
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 9301 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the DEMLOG daemon is not active, use the DIAGSYSTEM command
in SYSDIAG to start it.
If the DEMLOG daemon is active, use the LOGTOOL utility in SYSDIAG
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 474 I/O error entries were lost.

STM shows the following output

Path Product Active Tool Status

10/4/8 NIO LAN Interface (4) Information Incomplete
10/4/16 NIO LAN Interface (4) Information Incomplete
10/12 Core I/O Adapter (44) Information Successful


in the Activity log it shows

Fri Feb 8 11:31:55 2008: Information tool (dlpi) starting on path (10/4/8).

Fri Feb 8 11:31:56 2008: The name of the driver for the network interface card
is not recognized.
Possible Cause(s)/Recommended Action(s):

Internal application error.

Fri Feb 8 11:31:56 2008: Attempt to get access to device 10/4/8 failed.

Fri Feb 8 11:31:56 2008: The DLPI driver responded to a 2 request with an
error acknowledgement implying that the request could
not be completed.
The DLPI driver returned a dlpi error of 3 and an
HP-UX errno of 0
Possible Cause(s)/Recommended Action(s):

DLPI communication was not initiated with a call
to tl_dlpi_init. Call tl_dlpi_init.
The hardware path sent to tl_dlpi_init is not
associated with a DLPI device. Run ioscan -fn to
find the correct hardware path for the Ethernet,
Token Ring, or FDDI device or run the appropriate
(non-DLPI) tool against the hardware.
Refer to the DL_ERROR_ACK section of for
more information about dlpi error 3. Refer to
for more information about HP-UX errno
0
Internal application error.

Fri Feb 8 11:31:56 2008: An attempt to deactivate the stream by unbinding it
from the DLSAP (Data Link Service Access Point)
failed.

Fri Feb 8 11:31:56 2008: Attempt to release the device failed. Device could
be left in an invalid state.

Fri Feb 8 11:31:56 2008: Tool completed with exit_status MOD_INCOMPLETE (5)
indicating the tool was started but could not
properly complete execution.
======================================

please help me out.

Thanks everyone for your ongoing support till now.

SAM.

[Steven E. Protter · Accepted solution]

Shalom SAM,

I'd say your system has a pretty major problem.

Some questions to clarify:
1) Is the system pingable from the network.
2) Is the systems network switch or switch port on.

Attempt to get access to device 10/4/8 failed.
ioscan
What device is on this path. I think it failed.

SEP

[sam_25_1]

Hi Sep,

Thanks a lot for your valuable inputs,

I have checked the path specified are for the LAN cards..


10/4/8 NIO LAN Interface (4) Information Incomplete
10/4/16 NIO LAN Interface (4) Information Incomplete

The lan cards are not working if believe. The system is up and running fine,its a production system and there are 80 plus users logged on the server. The server is having no performance issues at the moment.. Please let me know what we could do to get this resolved.. I appreciate your help & guidance..

[Steve Steel]

Hi

sysmap: rmap ovflo, lost [44182l,44184l)

See
http://docs.hp.com/en/B3921-60631/nsysmap.5.html

When Should the Value of This Tunable Be Raised?
The following message will appear on the console when the resource map overflow occurs: sysmap{32|64}: rmap ovflo, lost [X,Y).

If this happens only occasionally, no action is necessary. If this happens with any sort of frequency, the tunable should be increased, or the following panic message may occur:

kalloc: out of kernel virtual space.When increasing nsysmap{32|64}, doubling the tunable value is a reasonable rule of thumb. If the problem persists after doubling the tunable several times from the default, there is likely to be a more serious kernel problem, and you should contact HP support to investigate.


Steve Steel

[Michael Steele_2]

Hi Sam:

This is probably an event associated to diagmond and EMS that can be turned of with EMS. Verifty this in /etc/opt/resmon/log/*. You're looking for large logs like registar. Paste in anything that looks fishy.

[Andrew Merritt_2]

Hi Sam,
The most likely cause is that there is a hardware problem, probably storage related, and diaglogd is spending its time dealing with the events passed to it by the device driver.

I say this because there are 9301 I/O error entries missed, which indicates that there are a lot being reported.

You should also check that you have a current version of the OnlineDiags installed.

If you're not getting any EMS events reported in /var/opt/resmon/log/event.log which tell you what the problem is, I suggest running STM and from there run 'logtool' and look to see which devices are reporting IO problems.

Andrew

[Andrew Merritt_2]

I'd suggest that you actually have a very old version of OnlineDiags installed, now I've noticed the references to "DEMLOG" and "SYSDIAG". Your first priority should be to get a supported version installed.

Check versions: http://www.docs.hp.com/hpux/onlinedocs/diag/stm/stm_upd.htm#table
Install: http://www.software.hp.com/portal/swdepot/displayProductInfo.do?productNumber=B6191AAE
Andrew


P.S. I've just noticed you have other threads going for the same problems, and that you mention this is HP-UX 10.20. The last version of OnlineDiags for that was A.29.00, if you can find that somewhere (it won't be on hp.com). You should still check out whether logtool shows you what is causing the problem. You should also look at replacing the system with a supported one.

[sam_25_1]

Hello Everyone,

Thanks a lot for your valuable suggestions & guidance I sincerly appreciate it. Andrew I am using a older version of os 10:20 so the stm version is not compatiable with the teh latest release, when I did a lanscan -v the hardware path 10/4/8 which showed in stmoutput using logutility tool is running fine and the nic cards are up, I am able to ping the servers.

Thanks once again to everyone for your valuable suggestiosn, I would check it and revert back to you'll. I am lucky to have some many ppl to help me out.

Have a nice day Gentleman..

Cheers

Sam