SSh disconnecting very frequently

[Grayh]

Hi All,

I have noticed that SSH is being disconnecting very frequently for which I have to kill the processes or reboot the server...

I see the following in syslog:-

/usr/sbin/nfsd[1530]: nfsd 1 3 sock 4
/usr/sbin/nfsd[1547]: nfsd 0 1 sock 4
/usr/sbin/nfsd[1529]: nfsd 0 3 sock 4
/usr/sbin/nfsd[1548]: nfsd 0 2 sock 4
/usr/sbin/nfsd[1546]: nfsd 0 0 sock 4
/usr/sbin/nfsd[1540]: nfsd 2 1 sock 4
/usr/sbin/nfsd[1542]: nfsd 2 2 sock 4

krsd[1615]: Delay time is 300 seconds
sshd[1692]: SSH: Server;Ltype: Version;Remote: 10.3.99.34-1862;Protocol: 1.99;Client: 3.2.9 SSH Secure Shell for Windows
Nov 24 08:18:04 HPBOX01 sshd[1692]: Accepted keyboard-interactive/pam for root fr
om 10.3.99.34 port 1862 ssh2
Nov 24 08:20:56 HPBOX01 sshd[1692]: SSH: Server;LType: Throughput;Remote: 10.3.99.34-1862;IN: 18016;OUT: 5808;Duration: 172.2;tPut_in: 104.6;tPut_out: 33.7
Nov 24 08:25:05 HPBOX01 sshd[3251]: SSH: Server;Ltype: Version;Remote: 10.3.99.34
-1887;Protocol: 1.99;Client: 3.2.9 SSH Secure Shell for Windows
Nov 24 08:25:11 HPBOX01 sshd[3251]: error: PAM: Authentication failed for root from 10.3.99.34
rt 1887 ssh2


Does the above mean something....

[Johnson Punniyalingam]

Hi Grayh,

how about your logout & timeout settings ...?

set autologout=60

Thanks,
Johnson

[OldSchool]

Nov 24 08:25:11 HPBOX01 sshd[3251]: error: PAM: Authentication failed for root from 10.3.99.34
rt 1887 ssh2

well, I didn't see anything related to disconnects, only that when prompted for the password, it wasn't entered correctly.

What process "do you have to kill"? sshd or the the connected session?

if its the connected session, then you have network issues between the client and the server, as the actual session was still running.

Posting any related message seen at the client might be instructive.

[Grayh]

What does this mean


# ps -aef | grep defunc
root 4587 2103 0 09:28:38 ? 0:00
root 4589 4453 0 09:28:45 console 0:00 grep defunc
# ps -aef | grep 2103
root 4587 2103 0 09:28:38 ? 0:00
root 4591 4453 0 09:28:56 console 0:00 grep 2103
root 2103 1 0 08:17:47 ? 0:02 /usr/sbin/stm/uut/bin/tools/moni
tor/ia64_corehw

Last time some 20+ defunct processes were there

[Ganesan R]

Hi Grayth,

There are many things to cause the session got disconnected.

1.It could be a pure network issue. I would first suspect this.

2.There is a shell variable TMOUT (autologout for csh) that controls session timeout.
#echo $TMOUT
0
If it returns the value 0 it means it's value is indefinite and it is the default value. Sessions will not get disconnected at all.
If TMOUT value is 0 and if still sessions gets disconnected, then there could be network settings. Need to contact network administrator.
If TMOUT value is set low increase it.

# export TMOUT=180

To make the change permanent set it in your .profile or /etc/profile.

3.It is not unusual for network administrators to forcefully disconnect telnet/ssh sessions that are idle for long periods. If you do not see the 1 minute warning at 59 minutes on your screen(if TMOUT is 60min), then HP-UX is not the problem. Contact your network administrator.
Network administrator may have set the settings in such a way that if any session is idle for more than a defined time then the session will be disconnected at router level.

[Grayh]

System Model Number.............: ia64 hp server BL860c
EMS Version.....................: A.04.20
STM Version.....................: C.58.00


OS: HP-UX 11iV2 .... Model: BL860C

I was thinking where else i should check at...

[Ganesan R]

Hi,

defunc process are zombie processes which is just sitting in the process table but does not consume any resources. These processes cannot be killed since it's parent process are already died.

But this is not the cause for the ssh session disconnections.

[Grayh]

Thanks Ganesn ... My time ouyt value is set to 0

# echo $TMOUT
0

Is there anything else I should Specifically Check for... Any other logs or anything

[Grayh]

Again the ssh got disconnected and it do not give any errors


And I do not see any errors as well

[Dennis Handly]

>What does this mean?
root 2103 1 0 08:17:47 ? 0:02 /usr/sbin/stm/uut/bin/tools/monitor/ia64_corehw
root 4587 2103 0 09:28:38 ? 0:00

This is a known problem in ia64_corehw. I don't know if there is a patch for it?

>Last time some 20+ defunct processes were there

With the same zombie master ia64_corehw?

>Ganesan: These processes cannot be killed since it's parent process are already died.

This is incorrect. zombies can be killed by killing the zombie master, which is very much alive.
If the parent process died, these are orphans and not defunct and are adopted by init.

[Grayh]

This time it is very strange...

I was checking the MP logs and other event and syslogs...

I can still ping the server and connect it through the MP

And all of a sudden telnet was working for a minute or two after which the console got disconnected... Could not connect with ssh at all..



Then rebooted the system... it worked..

Both ssh & telnet both worked...

Either a reboot of the server or bringing the interface down and then back UP is a Temperory solutions for this issue...

netstat shows

Ipkts Ierrs Opkts Oerrs Coll
1333 0 1006 0 0
331 0 331 0 0

[Grayh]

SSH: Server;Type: Version;Remote: -2500;Protocol: 1.99;Client: 3.2.9 SSH Secure Shell for Windows + HPUX

Accepted keyboard-interactive/pam for root from port 2500 ssh2

[Grayh]

I have found a bit more which might mean something here:-

In the passwd file

sshd:*:211:206::/var/empty:/usr/bin/ksh

[Grayh]

Normally this should look something like this....

sshd privsep:/var/empty:/bin/false

[OldSchool]

"Either a reboot of the server or bringing the interface down and then back UP is a Temperory solutions for this issue..."

the network interface? you are pointing right back at network / hardware issues. esp if telnet worked then didn't.....

Potential issues:
cabling, switches, routers, hubs and the network i/o cards in the server itself...not the ssh / sshd software.

[Grayh]

How can I start/Stop the ssh service ..

# /sbin/init.d/secsh stop
ksh: /sbin/init.d/secsh: not found.

I have the following two files on my box:

-->/usr/lpp/openssh.base/inst_root/etc/ssh/sshd_config
-->/etc/ssh/sshd_config

[vz7r1x]

To start ssh procs, do the following:

/usr/local/sbin/sshd start
or
/sbin/init.d/ssh start

Thanks

[Grayh]

Now this is acting very weird...

This minute I can telnet but not ssh
again I cannot telnet and no ssh

again I am able to telnet for some time... and Cant telnet or ssh or ftp again

... This has become quite serious now...

[Matti_Kurkela]

The path /usr/lpp/openssh.base/inst_root/etc/ssh/sshd_config is NOT the standard path of a HP-supported SSH package.

The standard path would be /opt/ssh/etc/sshd_config.

Is this a locally-compiled version, installed by a current or previous sysadmin of your site?

A quick Googling reveals that the path /usr/lpp/... seems to be an AIXism. Compiling a sshd for HP-UX using the configuration settings for AIX might cause all kinds of strange behaviour.

MK

[Grayh]

Sorry that was a typo... I now have the following

# find . -name sshd_config
./opt/ssh/newconfig/opt/ssh/etc/sshd_config
./opt/ssh/src/ssh/sshd_config
./opt/ssh/etc/sshd_config

[Patrick Wallek]

Run this command:

# ndd -get /dev/ip ip_ire_gw_probe

If it returns a value of '1' then that means that the IP stack will occasionally poll the gateway to see if it is alive. It something happens and it thinks the gateway is dead, then you can see behavior just like you see now.

So, if the value returned by the command above is '1', run this command:

# ndd -set /dev/ip ip_ire_gw_probe 0

Then see how long you telnet and ssh connections last.

If this appears to fix the problem, you will need to make this change permanent by editing the /etc/rc.config.d/nddconf file and adding:

TRANSPORT_NAME[0]=ip
NDD_NAME[0]=ip_ire_gw_probe
NDD_VALUE[0]=0

NOTE: If you already have some entries in this file, you will need to change the '[0]' above to whatever the next available number is. Be sure the you use the same number for all 3 lines above as they are a set. All configuration items in the nddconf file should be in sets of 3.

There are good comments in the nddconf file itself.

[Grayh]

Thanks Patrick...

I am now going to monitor the server for sometime ... do network tuning in /etc/rc.config.d/nddconf...

Currently I do not have any entries other than the default comments there..

But I was thinking why only this one blade server in the entire enclosure is behaving like this...

Did something go wrong on this Blade Server...

[OldSchool]

-->/usr/lpp/openssh.base/inst_root/etc/ssh/sshd_config

is the location on AIX....which explains why the start file wasn't there...they use a different mechanism to start the subsystems

[Grayh]

Sorry -->/usr/lpp/openssh.base/inst_root/etc/ssh/sshd_config was a typo error... As I was having multiple consoles open...

Looks like Patrics Solution has solved the issue.. I will have to check that tomorrow...

But I know he might be Off shift... If you Gurus can tell me as the signifance of ndd being able to resolve the issue when disabled from 1 to 0....

I am a bit curious to know

[Grayh]

Now I have some errors...

ssh_exchange_identification: read: Connection reset by peer
ssh: connect to host port 22: Connection refused