SSH Very Loaded

[AZayed]

Dears,
Good day,

Recently we force our users to switch to SFTP instead of FTP. Now, every time I login to the system using SSH it takes 10 seconds after entering the password. I think this is related to the last action. I tried to reduce the logging of sftp-server to QUIET but this is didn't help.

So, can you help me tune this issue?

Thanks

[Mel Burslan]

try posting the full output (screen capture) from command :

ssh -vvv my_hostname_here

and mark the line in this output where it hangs for that 10 seconds on this output

[AZayed]

Hi Mel,
Hope you doing well.

It stops at the end of the ssh login. After the last line.

wtmpx@wtmpx-laptop:~$ ssh -vvv -l root XXX.XXX.XXX.YYY
OpenSSH_5.5p1 Debian-4ubuntu4, OpenSSL 0.9.8o 01 Jun 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to XXX.XXX.XXX.YYY [XXX.XXX.XXX.YYY] port 22.
debug1: Connection established.
debug1: identity file /home/wtmpx/.ssh/id_rsa type -1
debug1: identity file /home/wtmpx/.ssh/id_rsa-cert type -1
debug1: identity file /home/wtmpx/.ssh/id_dsa type -1
debug1: identity file /home/wtmpx/.ssh/id_dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.6p1+sftpfilecontrol-v1.3-hpn13v7
debug1: match: OpenSSH_5.6p1+sftpfilecontrol-v1.3-hpn13v7 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.5p1 Debian-4ubuntu4
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 139/256
debug2: bits set: 519/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: host XXX.XXX.XXX.YYY filename /home/wtmpx/.ssh/known_hosts
debug3: check_host_in_hostfile: host XXX.XXX.XXX.YYY filename /home/wtmpx/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 5
debug1: Host 'XXX.XXX.XXX.YYY' is known and matches the RSA host key.
debug1: Found key in /home/wtmpx/.ssh/known_hosts:5
debug2: bits set: 508/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/wtmpx/.ssh/id_rsa ((nil))
debug2: key: /home/wtmpx/.ssh/id_dsa ((nil))
debug1: Authentications that can continue: publickey,password,keyboard-interactive,hostbased
debug3: start over, passed a different list publickey,password,keyboard-interactive,hostbased
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/wtmpx/.ssh/id_rsa
debug3: no such identity: /home/wtmpx/.ssh/id_rsa
debug1: Trying private key: /home/wtmpx/.ssh/id_dsa
debug3: no such identity: /home/wtmpx/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: packet_send2: adding 32 (len 23 padlen 9 extra_pad 64)
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 0
debug3: packet_send2: adding 48 (len 10 padlen 6 extra_pad 64)
debug1: Authentication succeeded (keyboard-interactive).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env ORBIT_SOCKETDIR
debug3: Ignored env SSH_AGENT_PID
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env XDG_SESSION_COOKIE
debug3: Ignored env WINDOWID
debug3: Ignored env GNOME_KEYRING_CONTROL
debug3: Ignored env GTK_MODULES
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env DEFAULTS_PATH
debug3: Ignored env SESSION_MANAGER
debug3: Ignored env USERNAME
debug3: Ignored env XDG_CONFIG_DIRS
debug3: Ignored env DESKTOP_SESSION
debug3: Ignored env PATH
debug3: Ignored env PWD
debug3: Ignored env GDM_KEYBOARD_LAYOUT
debug1: Sending env LANG = en_US.utf8
debug2: channel 0: request env confirm 0
debug3: Ignored env GNOME_KEYRING_PID
debug3: Ignored env MANDATORY_PATH
debug3: Ignored env GDM_LANG
debug3: Ignored env GDMSESSION
debug3: Ignored env SPEECHD_PORT
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env GNOME_DESKTOP_SESSION_ID
debug3: Ignored env LOGNAME
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env DBUS_SESSION_BUS_ADDRESS
debug3: Ignored env LESSOPEN
debug3: Ignored env WINDOWPATH
debug3: Ignored env DISPLAY
debug3: Ignored env LESSCLOSE
debug3: Ignored env XAUTHORITY
debug3: Ignored env COLORTERM
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: fd 3 setting TCP_NODELAY
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 65536
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0

[Earl_Crowder · Accepted solution]

Check the size of /var/adm/wtmps, if it's large, and you don't need it, truncate it:
>/var/adm/wtmps

See this thread :

http://h30499.www3.hp.com/t5/System-Administration/SSH-Authentification-very-slow-on-HPUX-11-31/m-p/4361622#M346130

[Mel Burslan]

I am assuming you do not get any delay before you enter the password and after entering the password, everything scrolls up fast until it hits the last line on this log. I that is right, your ssh is working fine whereas you system might be too busy to handle your shell request.
how is the system load ? and since you are suspecting too many ssh processes, check to see how many sshd processes are running at any given time by running

ps -ef | grep ssh | wc -l

at random intervals. Unless it is an out-of this world high number like in the order of thousands, your sshd is good for all intents and purposes and you need to look elsewhere for this problem

[Bill Hassell]

What version of HP-UX are you running? Did you install the KRNG (Kernel Random Number Generator) product:

https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=KRNG11I

Without this add-on, the random number generator code is based on several commands that can sometimes take a very long time to complete. The RNG is called during the key exchange during authentication.

[AZayed]

Hi Folks,
Hope you doing well.

Yes, maybe my SSH is working fine but my system is overloaded.

Now I grep for SSH and there is the output :

# uname -r
B.11.23

08:33 : Number of SSH is 23 & Number of sftp is 22
08:34 : Number of SSH is 2 & Number of sftp is 1
08:35 : Number of SSH is 6 & Number of sftp is 5
08:36 : Number of SSH is 2 & Number of sftp is 1
08:37 : Number of SSH is 6 & Number of sftp is 5
08:38 : Number of SSH is 22 & Number of sftp is 21
08:39 : Number of SSH is 6 & Number of sftp is 5
08:40 : Number of SSH is 10 & Number of sftp is 9
08:41 : Number of SSH is 2 & Number of sftp is 1
08:42 : Number of SSH is 6 & Number of sftp is 5
08:43 : Number of SSH is 22 & Number of sftp is 21
08:44 : Number of SSH is 6 & Number of sftp is 5
08:45 : Number of SSH is 3 & Number of sftp is 2
08:46 : Number of SSH is 2 & Number of sftp is 1
08:47 : Number of SSH is 7 & Number of sftp is 6
08:48 : Number of SSH is 22 & Number of sftp is 21
08:49 : Number of SSH is 6 & Number of sftp is 5
08:50 : Number of SSH is 6 & Number of sftp is 5
08:51 : Number of SSH is 2 & Number of sftp is 1
08:52 : Number of SSH is 6 & Number of sftp is 5
08:53 : Number of SSH is 22 & Number of sftp is 21
08:54 : Number of SSH is 10 & Number of sftp is 9
08:55 : Number of SSH is 2 & Number of sftp is 1
08:56 : Number of SSH is 2 & Number of sftp is 1
08:57 : Number of SSH is 6 & Number of sftp is 5
08:58 : Number of SSH is 22 & Number of sftp is 21
08:59 : Number of SSH is 6 & Number of sftp is 5
09:00 : Number of SSH is 10 & Number of sftp is 9
09:01 : Number of SSH is 2 & Number of sftp is 1
09:02 : Number of SSH is 2 & Number of sftp is 1

[AZayed]

Hi,
Also, this is the output for wtmpx

# ls -l /var/adm/w*
-rw-rw-rw- 1 adm adm 263520 Dec 24 02:45 /var/adm/wtmp
-rw-rw-r-- 1 adm adm 183060 Jun 15 2008 /var/adm/wtmp.Jun15-08
-rw-rw-rw- 1 adm adm 432815856 Jan 6 09:22 /var/adm/wtmps
-rw-rw-r-- 1 adm adm 6877184 Jun 15 2008 /var/adm/wtmps.Jun15-08
-rw-r--r-- 1 root root 280 Dec 24 02:45 /var/adm/wtmpx
#
# du -sk /var/adm/w*
264 /var/adm/wtmp
184 /var/adm/wtmp.Jun15-08
436704 /var/adm/wtmps
6728 /var/adm/wtmps.Jun15-08
8 /var/adm/wtmpx

[AZayed]

Also,
This is the OpenSSH version information. I think it's the latest one.

OpenSSH_5.6p1+sftpfilecontrol-v1.3-hpn13v7, OpenSSL 0.9.8o 01 Jun 2010
HP-UX Secure Shell-A.05.60.002, HP-UX Secure Shell version

[AZayed]

Thanks folks,

SSH reads wtmps file and it seems I have a big file. around half gig.

Now I trim the file it's working very well.

/usr/sbin/acct/fwtmp < /var/adm/wtmps > /var/adm/wtmps-Jan-06-2011.ASCII

cat /dev/null > /var/adm/wtmps