swagentd and rcpd processes used almost all CPU resources

[Dmitriy_16]

HP-UX scp1 B.11.11 U 9000/800 1116474605 unlimited-user license

System was overloaded because of 2 processes swagentd and rcpd
System: scp1 Mon Aug 22 10:40:11 2005
Load averages: 0.74, 0.70, 0.68
143 processes: 133 sleeping, 10 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 1.04 17.0% 0.0% 43.8% 39.2% 0.0% 0.0% 0.0% 0.0%
1 0.69 13.9% 0.0% 80.2% 6.0% 0.0% 0.0% 0.0% 0.0%
2 0.49 13.7% 0.0% 51.6% 34.7% 0.0% 0.0% 0.0% 0.0%
3 0.73 19.0% 0.0% 21.4% 59.5% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.74 16.0% 0.0% 49.1% 34.9% 0.0% 0.0% 0.0% 0.0%

Memory: 1508868K (1478488K) real, 2067404K (2024444K) virtual, 1685496K free Page# 1/5

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 1201 root 241 20 10156K 2796K run 464:53 98.55 98.37 swagentd
2 ? 1039 root 241 20 11148K 3388K run 469:35 98.41 98.24 rpcd
3 ? 16256 scu 154 20 432M 122M sleep 549:26 8.45 8.44 scf
1 ? 16257 scu 154 20 432M 121M sleep 565:09 8.41 8.39 scf
1 ? 16255 scu 154 20 432M 121M sleep 565:34 8.31 8.29 scf
3 ? 16251 scu 154 20 35440K 25192K sleep 204:26 3.41 3.40 manager
1 ? 1519 root 152 20 318M 123M

Attempt for restarting of swagentd process failed
#swagentd -r
ERROR: swagentd exiting due to initialization error in background
process. Check the logfile "/var/adm/sw/swagentd.log" for
more information.
# vi /var/adm/sw/swagentd.log
"/var/adm/sw/swagentd.log" 911 lines, 31951 characters

======= 03/28/03 10:38:15 SST BEGIN swagentd (pid = 248).

* Checking job queue at 1 minute intervals.
* Started install agent on "/" for root@svu20612, pid=261,
03/28/03 10:38:21 SST
* Agent pid=261 completed. 03/28/03 10:42:33 SST
* Started install agent on "/" for root@svu20612, pid=1537,
03/28/03 10:42:40 SST
* Agent pid=1537 completed. 03/28/03 10:42:57 SST
* Started install agent on "/" for root@svu20612, pid=1587,
03/28/03 10:43:03 SST
* Agent pid=1587 completed. 03/28/03 10:43:51 SST
* Started install agent on "/" for root@svu20612, pid=1865,
03/28/03 10:43:57 SST
* Agent pid=1865 completed. 03/28/03 10:44:13 SST
* Started install agent on "/" for root@svu20612, pid=1987,
03/28/03 10:44:17 SST
* Agent pid=1987 completed. 03/28/03 10:50:08 SST
* Started install agent on "/" for root@svu20612, pid=3418,
03/28/03 10:50:16 SST
* Agent pid=3418 completed. 03/28/03 10:50:39 SST
* Started install agent on "/" for root@svu20612, pid=3940,
03/28/03 10:50:46 SST
* Agent pid=3940 completed. 03/28/03 10:51:26 SST

======= 03/28/03 10:57:16 SST BEGIN swagentd (pid = 97).

* Checking job queue at 1 minute intervals.
* Started configure agent on "/" for root@svu20612, pid=100,
03/28/03 10:57:16 SST
* Agent pid=100 completed. 03/28/03 10:58:23 SST
* Started configure agent on "/" for root@svu20612, pid=1554,
03/28/03 10:58:26 SST
* Agent pid=1554 completed. 03/28/03 10:58:36 SST
* Started configure agent on "/" for root@svu20612, pid=1749,
03/28/03 10:58:38 SST
* Agent pid=1749 completed. 03/28/03 10:58:59 SST
* Started configure agent on "/" for root@svu20612, pid=2017,
03/28/03 10:59:04 SST
* Agent pid=2017 completed. 03/28/03 10:59:08 SST
* Started configure agent on "/" for root@svu20612, pid=2116,
03/28/03 10:59:12 SST
* Agent pid=2116 completed. 03/28/03 10:59:27 SST
* Started configure agent on "/" for root@svu20612, pid=2427,
03/28/03 10:59:29 SST
* Agent pid=2427 completed. 03/28/03 10:59:58 SST
* Started configure agent on "/" for root@svu20612, pid=2965,
"/var/adm/sw/swagentd.log" 911 lines, 31951 characters
* Checking job queue at 1 minute intervals.

======= 08/22/05 11:08:09 GMT BEGIN swagentd (pid = 14497).

WARNING: Failure registering as Remote Procedure Call server using
protocol sequence "ncacn_ip_tcp" with endpoint "2121". The
endpoint you have specified is in use by another process.
Either another copy of the SD daemon is running or there is a
conflict in choice of endpoints with another application.
Endpoints are specified within square brackets [] in the
"rpc_binding_info=" option.
WARNING: Failure registering as Remote Procedure Call server using
protocol sequence "ncadg_ip_udp" with endpoint "2121". The
endpoint you have specified is in use by another process.
Either another copy of the SD daemon is running or there is a
conflict in choice of endpoints with another application.
Endpoints are specified within square brackets [] in the
"rpc_binding_info=" option.
ERROR: Terminating prematurely - cannot serve RPC requests with any
binding.

======= 08/22/05 11:08:09 GMT END swagentd (pid = 14497, startup
error).


I want mention that I have found unclosed connection by 2121 port in
HP server from another PC 10.11.11.18
# netstat -an | grep 2121
tcp 0 0 10.11.11.30.2121 *.* LISTEN
tcp 0 0 scp1.2121 10.11.8.18.44386 CLOSE_WAIT
udp 0 0 *.2121 *.*
48d90800 stream 0 0 48d6e280
0 0 0
/opt/dcelocal/var/rpc/local/s-0/2121
48d90b00 dgram 0 0 48d5cf40
0 0 0
/opt/dcelocal/var/rpc/local/s-3/2121

I suppose that unclosed connection between HP server and 10.11.8.18 host was the
reason of swagentd and rcpd proceses overloaded CPU.

I have restarted swagend daemon by using
kill -9
/usr/sbin/swagentd -r
But after that I also had to stop using kill -9 rpcd process and then
restarted it /opt/dce/sbin/rpcd
ps -ef|grep rpcd
scu 17229 3201 1 12:10:04 pts/tc 0:00 grep rpcd
root 10512 1 0 11:57:47 ? 0:00 /opt/dce/sbin/rpcd
ps -ef|grep swagentd
scu 17685 3201 1 12:11:01 pts/tc 0:00 grep swagentd
root 27299 1 0 11:32:13 ? 0:00 swagentd -r

Current system load is normal

I want to know how can appeare such connection? and how it is possible to prevent such situation in future?

[RAC_1]

I ran into very similar once. This is caused by hanged swagentd (not exactly hung, but waiting for ncadg_ip_udp call. Basically a swinstall/swremove/swlist )sd commands) hang and in turn locks up swagentds and rpcd.

The only measure, I had done was restart swagentd.

To avoid this be sure to patch your system regularly. This does not seem to be a known issue with swagentd.