topic Re: ServiceGuard causes kernel process table to fill! in Operating System - HP-UX

ServiceGuard causes kernel process table to fill!

mpua — Wed, 17 Nov 2010 09:14:00 GMT

Hi,

I've experienced a strange issue with one of my HP-UX 11.23 servers.
We have ServiceGuard A.11.18.00 installed in both nodes... with four defined packages: two failower pckages (Control-M and Control-EM) and two Oracle multinode packages.

The process table filled completely yesterday at 23:15 roughly but its been a gradual process, it all began at november 15th i think...

The last "ps -ef" log we collected shows thousands of ServiceGuard processes (cmclconfd...)

The last ServiceGuard log lines in syslog.log say the following:

Nov 15 21:49:31 server2 cmclconfd[11164]: cl_msg_tcp_post_accept failed: Invalid argument
Nov 15 21:49:31 server2 cmclconfd[11164]: Failed to TCP listen.
Nov 15 21:49:31 server2 cmclconfd[11164]: Failed setting up daemon.
Nov 15 22:01:28 server2 cmclconfd[26255]: Unable to retrieve peer socket address: Invalid argument
Nov 15 22:01:28 server2 cmclconfd[26255]: cl_msg_tcp_post_accept failed: Invalid argument
Nov 15 22:01:28 server2 cmclconfd[26255]: Failed to TCP listen.
Nov 15 22:01:28 server2 cmclconfd[26255]: Failed setting up daemon.
Nov 15 22:53:45 server2 cmclconfd[20250]: Unable to retrieve peer socket address: Invalid argument
Nov 15 22:53:45 server2 cmclconfd[20250]: cl_msg_tcp_post_accept failed: Invalid argument
Nov 15 22:53:45 server2 cmclconfd[20250]: Failed to TCP listen.
Nov 15 22:53:45 server2 cmclconfd[20250]: Failed setting up daemon.
Nov 15 23:01:49 server2 cmclconfd[1567]: Unable to retrieve peer socket address: Invalid argument
Nov 15 23:01:49 server2 cmclconfd[1567]: cl_msg_tcp_post_accept failed: Invalid argument
Nov 15 23:01:49 server2 cmclconfd[1567]: Failed to TCP listen.
Nov 15 23:01:49 server2 cmclconfd[1567]: Failed setting up daemon.

Any hints?

Re: ServiceGuard causes kernel process table to fill!

Raj Briden — Wed, 17 Nov 2010 09:43:22 GMT

what is the status of thousands of cmclconfd process in ps -ef .. it is still running/dead/sleep

Re: ServiceGuard causes kernel process table to fill!

mpua — Wed, 17 Nov 2010 10:05:12 GMT

I had to reset the partition (this box is a nPar in a SuperDome complex) so the process table usage is fine now:

11:03:25 text-sz ov proc-sz ov inod-sz ov file-sz ov
11:03:27 N/A N/A 368/4200 0 1374/793063 0 3028/92000 0
11:03:29 N/A N/A 368/4200 0 1374/793063 0 3029/92000 0
11:03:31 N/A N/A 371/4200 0 1379/793063 0 3045/92000 0
11:03:33 N/A N/A 370/4200 0 1376/793063 0 3045/92000 0

Re: ServiceGuard causes kernel process table to fill!

Raj Briden — Wed, 17 Nov 2010 10:11:51 GMT

looks good :-)

Re: ServiceGuard causes kernel process table to fill!

mpua — Wed, 17 Nov 2010 11:17:15 GMT

Anyway, I've opened a support case with HP in order to get this not happen again...