topic Diagmond stopped due to system port Failure in Operating System - HP-UX

Diagmond stopped due to system port Failure

Roland Rebstock_1 — Fri, 09 Dec 2005 10:40:30 GMT

All, we have had a strange issue with diagmond/stm on all of our HPUX 11.0 servers. For some reason diagmond stopped at the same time 3 days in a row and the same time on 10 L/N Classes. Could this be a bug with Veritas Netbackup? there seems to be some correlation. There error states

Server 1
Dec 7 08:11:02 msuaow01 diagmond[761]: Exit due to system port failure
Dec 7 08:11:02 msuaow01 inetd[13821]: execv /usr/openv/netbackup/bin/vopied: No
such file or directory

Server 2
Dec 7 08:29:08 msuart01 diagmond[18607]: Exit due to system port failure
Dec 7 08:29:13 msuart01 inetd[560]: vopied/tcp: accept: No buffer space availab
le

Sever 3
Dec 7 08:23:56 msuapd09 diagmond[593]: Exit due to system port failure

Server 4
Dec 7 08:10:58 msuapd07 diagmond[738]: Exit due to system port failure
Dec 7 08:11:03 msuapd07 inetd[7461]: execv /usr/openv/netbackup/bin/vopied: No
such file or directory

Re: Diagmond stopped due to system port Failure

Sameer_Nirmal — Fri, 09 Dec 2005 12:04:51 GMT

Hi,

The port used by "diagmond" is tcp/1508 . This is system port and always need to be available at all time.
The messages shows that the deamon exit on account of this port failure.
To know the cause of diagmond exit, you need to check its logs using
# cstm
cstm > sal

Are you trying to say such errors are flashed when Veritas netbackup is used at a specified time?
If yes , then I suspect port 1508 is being used by Netbackup services. Check Netbackup documentation as well as /etc/services file for Netbackup entries. Port 1508 needs to be spare-out from Netbackup context.

Which version of STM you have on the system?
# swlist -l bundle OnlineDiag

Re: Diagmond stopped due to system port Failure

Roland Rebstock_1 — Fri, 09 Dec 2005 12:30:30 GMT

ed Dec 7 10:14:52 2005: Daemon process (diaglogd) with process identifier
(6058) exited.

Wed Dec 7 10:14:52 2005: Daemon process completed with exit_status
SYS_ACCESS_IOE_FAILED (121) indicating process exited
due to failure to get access to system error driver
in the OS.

Possible Causes/Recommended Action:

Process internal error.

Wed Dec 7 10:14:52 2005: Launching daemon executable
(/usr/sbin/stm/uut/bin/sys/diaglogd).
Wed Dec 7 10:14:54 2005: Daemon process (diaglogd) with process identifier
(6136) exited.

Wed Dec 7 10:14:54 2005: Daemon process completed with exit_status
SYS_ACCESS_IOE_FAILED (121) indicating process exited
due to failure to get access to system error driver
in the OS.

Possible Causes/Recommended Action:

Process internal error.

Wed Dec 7 10:14:54 2005: Launching daemon executable
(/usr/sbin/stm/uut/bin/sys/diaglogd).

Wed Dec 7 10:14:55 2005: Daemon process (diaglogd) with process identifier

Wed Dec 7 10:14:56 2005: Launching daemon executable
(/usr/sbin/stm/uut/bin/sys/diaglogd).

Wed Dec 7 10:14:57 2005: Daemon process (diaglogd) with process identifier
(6155) exited.

Wed Dec 7 10:14:57 2005: Daemon process completed with exit_status
SYS_ACCESS_IOE_FAILED (121) indicating process exited
due to failure to get access to system error driver
in the OS.

Possible Causes/Recommended Action:

Process internal error.

Wed Dec 7 10:14:57 2005: Launching daemon executable
(/usr/sbin/stm/uut/bin/sys/diaglogd).

Wed Dec 7 10:14:58 2005: Daemon process (diaglogd) with process identifier
(6157) exited.
Wed Dec 7 10:14:58 2005: Daemon process completed with exit_status
SYS_ACCESS_IOE_FAILED (121) indicating process exited
due to failure to get access to system error driver
in the OS.

Possible Causes/Recommended Action:

Process internal error.

Wed Dec 7 10:14:58 2005: Daemon process (diaglogd) will not be restarted as
restart attempts have exceeded the maximum allowed
(5). Start daemon process manually using User
Interface.

Thu Dec 8 08:27:00 2005: Read system call failed with errno (239), when
attempting to read data from file descriptor (6).

The errno returned from the read system call was
unexpected.

Thu Dec 8 08:27:00 2005: The attempt to read IPC message data failed.

Thu Dec 8 08:27:00 2005: Attempt to read the IPC header from a port failed.
The port is on system name (msuart01) at IP address
(172.25.0.199) and has system port (1508).

Thu Dec 8 08:27:00 2005: Attempt to receive message from system port failed.

Thu Dec 8 08:27:00 2005: Attempt to retrieve a message from the Support Tools
System Manager system port failed due to unexpected
errors.

Possible Causes/Recommended Action:

Internal application error. Support Tools System
Manager will abort.

Thu Dec 8 09:17:20 2005: Diagmond daemon started.

Re: Diagmond stopped due to system port Failure

Sameer_Nirmal — Fri, 09 Dec 2005 15:52:44 GMT

Hi,

Diagnostics checklist is given here. Verify all those systems as per this checklist.

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000065556789

Re: Diagmond stopped due to system port Failure

Andrew Merritt_2 — Tue, 13 Dec 2005 06:38:57 GMT

Hi Roland,
As Sameer asks, what version of OnlineDiags do you have installed?

There were some problems with diagmond exiting due to problems with being probed on its port, but those should be fixed in the current versions of OnlineDiags. However, it does look as though the Netbackup is triggering this; why is it accessing port 1508?

Regarding the 'lsal' output from cstm; what's actually logged at the time that the diagmond process exits for the first time (as logged in syslog)?

(I couldn't follow the link Sameer gave, btw.)

Andrew

Re: Diagmond stopped due to system port Failure

Roland Rebstock_1 — Tue, 13 Dec 2005 10:30:22 GMT

OnlineDiag B.11.00.18.09 HPUX 11.0 Support Tools Bundle, Sep 2001

Re: Diagmond stopped due to system port Failure

Andrew Merritt_2 — Tue, 13 Dec 2005 10:47:03 GMT

Oh, gosh, that's OLD! There have certainly been some fixes in that area since that release to deal with port scanners.

http://www.docs.hp.com/en/diag/stm/stm_upd.htm#table

Install HWE0403 (A.47.00) and PHSS_33917, then see if the problem persists.

The link to the OnlineDiags is:

http://www.software.hp.com/portal/swdepot/displayProductInfo.do?productNumber=B6191AAE

You can also go to http://www.software.hp.com and then type "B6191AAE" in the search box.

Andrew