sshd takes high cpu

[Vijay Chakravarthy]

Hi

On Hpux 11.11 I found that the sshd is taking up high cpu. Its not the actual daemon /opt/ssh/sbin/sshd but the cpu consuming process is shown up as '/opt/ssh/sbin/sshd -R'.

The recent changes to my system is that I installed IP filter.

Below is the output from ps.

root 23066 6190 255 10:15:21 ? 852:44 /opt/ssh/sbin/sshd -R
root 6190 1 0 Sep 6 ? 0:04 /opt/ssh/sbin/sshd
root 15051 6190 255 Sep 8 ? 2122:24 /opt/ssh/sbin/sshd -R

It can be seen that both processes 15051 and 24066 have taken up lot of cpu time.
Any help on this is highly appreciated.

Thanks
Vijay.

[Matti_Kurkela]

Do you have the "lsof" command installed (a very good tool available from the Porting Archive)?
You could use "lsof -p 23066" and "lsof -p 15051" to see what files and/or network sockets these processes are accessing.

Note that the problem processes are children of the main sshd process. Sshd forks a child process when it's receiving a connection. If the "privilege separation" mechanism is in use, it will actually fork two processes.
See the description here:
http://www.citi.umich.edu/u/provos/ssh/privsep.html

As these problem processes are still running as root, it suggests that whatever has caused them to be stuck in their current state has happened almost immediately after the fork.

Does your IPfilter set-up block TCP Reset or ICMP destination-unreachable packets? Blocking off these might cause sshd to be unaware that the client-side has lost the connection. Some versions of SSH seem to get stuck at that point, eating CPU.

(I think I've seen this or very similar behaviour before, but I don't remember enough details. In that case, there was a very strictly-configured firewall between the client and the server.)

MK

[rick jones]

First step would be to fire-up Glance and see in which state (User, System etc) the processes are consuming CPU cycles. If user, then you might try profiling with Prospect: http://www.hp.com/go/prospect If kernel, you might take a tusc trace to see which system calls it is making, or use the suitable screen in Glance. You might also get some interesting stuff from Prospect in that case as well.

[Pedro Cirne]

Hi,

Curious...I'm having exactly the same problem:

System: porsx013 Tue Oct 9 10:21:14 2007
Load averages: 1.74, 1.91, 1.83
374 processes: 298 sleeping, 52 running, 24 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 1.71 5.4% 89.6% 5.0% 0.0% 0.0% 0.0% 0.0% 0.0%
1 1.77 20.1% 42.1% 37.6% 0.2% 0.0% 0.0% 0.0% 0.0%
2 1.77 2.2% 93.8% 4.0% 0.0% 0.0% 0.0% 0.0% 0.0%
3 1.72 1.4% 97.4% 1.0% 0.2% 0.0% 0.0% 0.0% 0.0%
4 1.55 14.1% 71.2% 14.7% 0.0% 0.0% 0.0% 0.0% 0.0%
5 1.75 2.0% 96.0% 2.0% 0.0% 0.0% 0.0% 0.0% 0.0%
6 1.79 2.6% 93.2% 4.2% 0.0% 0.0% 0.0% 0.0% 0.0%
7 1.82 5.4% 88.9% 5.8% 0.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 1.74 6.6% 84.3% 9.2% 0.0% 0.0% 0.0% 0.0% 0.0%

Memory: 4717528K (4114612K) real, 8630196K (7659064K) virtual, 175272K free Page# 1/9

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 22406 root 241 20 5416K 60K run 965:54 78.91 78.77 sshd
5 ? 21327 ebs_jm 245 22 7008K 3412K run 0:26 64.76 59.17 map

root@porsx013:/usr/bin# lsof -p 22406
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 22406 root cwd DIR 64,0x3 8192 2 /
sshd 22406 root txt REG 64,0x6 1351680 40769 /opt (/dev/vg00/lvol6)
sshd 22406 root mem REG 64,0x7 36864 5164 /usr/lib/libpam.1
sshd 22406 root mem REG 64,0x7 749568 9152 /usr/lib/libnsl.1
sshd 22406 root mem REG 64,0x6 135168 18091 /opt/star-ncf-prod/ep_patch/usr/lib/libxti.2
sshd 22406 root mem REG 64,0x7 28672 5504 /usr/lib/libxnet.2
sshd 22406 root mem REG 64,0x7 282624 28483 /usr/lib/libm.2
sshd 22406 root mem REG 64,0x7 155648 27764 /usr/lib/libsec.2
sshd 22406 root mem REG 64,0x7 253952 28715 /usr (/dev/vg00/lvol7)
sshd 22406 root mem REG 64,0x7 163840 7263 /usr/lib/libk5crypto.sl
sshd 22406 root mem REG 64,0x7 20480 7261 /usr/lib/libcom_err.sl
sshd 22406 root mem REG 64,0x7 1110016 28716 /usr/lib/libkrb5.sl
sshd 22406 root mem REG 64,0x7 159744 139 /usr/lib/libpthread.1
sshd 22406 root mem REG 64,0x7 24576 45450 /usr/lib/libdld.2
sshd 22406 root mem REG 64,0x7 1826816 45150 /usr/lib/libc.2
sshd 22406 root mem REG 64,0x7 270336 45448 /usr/lib/dld.sl
sshd 22406 root 0u IPv4 0x52972580 0t0 TCP porsx013:22->porsx013:62164 (ESTABLISHED)
sshd 22406 root 1u IPv4 0x52972580 0t0 TCP porsx013:22->porsx013:62164 (ESTABLISHED)
sshd 22406 root 2u CHR 3,0x2 0t0 66 /dev/null
sshd 22406 root 3u unix 0x4d59c4c0 0t2
sshd 22406 root 4w FIFO 0x8c14db08 0t0 653373571
sshd 22406 root 5u unix 0x50efe7c0 0t0

[RaviKumarB]

Hi I also have the same issue. If any one come across the solution, please let me know.

Load averages: 3.49, 2.23, 2.82
396 processes: 369 sleeping, 26 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 3.62 30.5% 0.0% 63.9% 5.6% 0.0% 0.0% 0.0% 0.0%
1 3.36 48.0% 3.0% 16.7% 32.3% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 3.49 39.2% 1.4% 40.4% 18.9% 0.0% 0.0% 0.0% 0.0%

Memory: 2259096K (1111892K) real, 3413572K (1903632K) virtual, 219020K free Pag
e# 1/36

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 2409 root 241 20 5416K 428K run 5135:23 62.49 62.38 sshd
1 ? 4697 root -16 20 43392K 6448K run 39231:00 4.36 4.35 midaemon
1 ? 10982 ctmagent 154 20 5816K 1028K sleep 0:00 4.15 2.46 ssh
0 ? 26320 ctmagent 158 20 3160K 1352K sleep 3:32 1.01 1.01 sh
1 ? 4715 root 127 20 82932K 34844K sleep 2008:13 0.55 0.55 scopeux
1 ? 11066 ctmagent 154 20 5372K 524K sleep 0:00 5.64 0.54 ssh
1 ? 16400 ctmagent 152 20 272M 53344K run 550:37 0.46 0.46 java
1 ? 37 root 152 20 8032K 8032K run 1128:02 0.41 0.41 vxfsd
1 ? 4758 root 152 20 312M 81092K run 678:38 0.37 0.37 java
root@hapinfp3:/># lsof -p 2409
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 2409 root cwd DIR 64,0x3 8192 2 /
sshd 2409 root txt REG 64,0x4 1351680 217 /opt/ssh/PA-RISC2.0/sbin/sshd
sshd 2409 root mem REG 64,0x6 36864 6301 /usr/lib/libpam.1
sshd 2409 root mem REG 64,0x6 753664 14589 /usr/lib/libnsl.1
sshd 2409 root mem REG 64,0x4 135168 43658 /opt/star-ncf-prod/ep_patch/usr/lib/libxti.2
sshd 2409 root mem REG 64,0x6 28672 6665 /usr/lib/libxnet.2
sshd 2409 root mem REG 64,0x6 282624 54 /usr/lib/libm.2
sshd 2409 root mem REG 64,0x6 155648 58 /usr/lib/libsec.2
sshd 2409 root mem REG 64,0x6 253952 9480 /usr/lib/gss/libgssapi_krb5.sl
sshd 2409 root mem REG 64,0x6 163840 9483 /usr/lib/libk5crypto.sl
sshd 2409 root mem REG 64,0x6 20480 9481 /usr/lib/libcom_err.sl
sshd 2409 root mem REG 64,0x6 1105920 9484 /usr/lib/libkrb5.sl
sshd 2409 root mem REG 64,0x6 159744 56 /usr/lib/libpthread.1
sshd 2409 root mem REG 64,0x6 24576 6068 /usr/lib/libdld.2
sshd 2409 root mem REG 64,0x6 1826816 6339 /usr/lib/libc.2
sshd 2409 root mem REG 64,0x6 274432 65 /usr/lib/dld.sl
sshd 2409 root 0u STSO 21,0x80000e 0t0 86 /dev/tcp->tcpm->ip
sshd 2409 root 1u STSO 21,0x80000e 0t0 86 /dev/tcp->tcpm->ip
sshd 2409 root 2u CHR 3,0x2 0t0 68 /dev/null
sshd 2409 root 3u unix 0x6d7f7580 0t2
sshd 2409 root 4w FIFO 0x527cdc88 0t0 1720162046
sshd 2409 root 5u unix 0x6cf13880 0t0

[ranganath ramachandra]

Sounds like this: HP-UX - What Are the sshd "-R" and "-r" Options?