rsh connection times out occassionally

[Kevin Bingham]

Hi

Have 9000/800, recently upgraded to HP-UX 11.0 (32bit).

Since the upgrade, remote shell calls occasionally timeout and fail. Remote shell is being invoke from Win NT SBS, and message that NT gets is as follows:
c:\>rsh hp1 -l userid command
-> hp1:Connection timed out
rsh: can't establish connection

While NT is waiting for rsh to respond, I have looked at netstat output, and got the following while it is waiting:
hp1:/#netstat -a |grep WAIT
tcp 0 0 hp1.1023 ntsrv.1022 TIME_WAIT
tcp 0 0 hp1.shell ntsrv.1023 TIME_WAIT

We use the rsh functioanlity to port our software to HP-UX, and the pattern seems to be that the first rsh call works fine, but subsequent ones may or may not work.

Any help greatly rewarded/appreciated...
Thanks in advance.
Kevin

[Pedro Sousa]

Hi Kevin!
I'm not aware of that problem, but here is a patch that you should have installed: PHNE_21731 s700_800 11.00 r-commands cumulative mega-patch

It fixes some errors relating remsh hangs.
hope it helps.
good luck

[Steven Sim Kok Leong]

Hi,

To aid troubleshooting, have you checked whether:
1) multiple remsh to localhost on the HP-UX server works?
2) remsh from a non-NT client to the HP-UX server works?
3) rsh from another NT client to the HP-UX server works?
4) ftp and telnet works from your NT client to the HP-UX server?

Do you have a firewall or filter between your NT client and your HP-UX server?

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Kevin Bingham]

Answers to the questions so far...

I am reviewing the documentation for PHNE_21731 to determine whether this is an applicable patch, we do not have it installed yet.

1)Multiple remsh calls to local host work fine
2)Our office uses NT, no non-NT clients available, single HP box for porting work
3)rsh from another NT machine works fine to HP box
4)ftp and telnet work fine from the NT SBS Server to the HP box.

No firewall or filters in place, really simple network, NT.SBS->hub<-HP

[Kevin Bingham]

Hi again
I read the notes for PHNE_21731 and it looked promising so I installed the patch.

Unfortunately the problem still exists after a reboot of the machine.

Any more ideas?

Kevin

[Pedro Sousa]

Hi again.

I found this doc relating a similar problem.
It's not a recent one, but maybe it helps you dealing with it:
http://europe-support2.external.hp.com/cki/bin/doc.pl/sid=44d0b16809c4a2926f/screen=ckiDisplayDocument?docId=200000018438359

Did you change your domain name?
Do you use DNS? Have you changed the UNIX user account name?

good luck.

[Kevin Bingham]

Hi,

Thanks for the reply. I cannot view the link that you posted, I get an MSIE "The page cannot be found" error message when I try to go there. Can you send it to me? kbingham@excite.co.uk

I changed the hosts file to add another domain-qualified alias for the machine to solve a problem with the sendmail program. I tried backing out this change, but the problem persists. We have not changed the Unix user account explicitly ( I am not sure what the HP-UX 11.0 upgrade may have done). As far as I know, we don't use DNS.

Kevin

[Pedro Sousa]

Kevin, I sent you the email with the link.

Telnet the HPUX system and then execute "who -R"
Check if the domain is correct.

[Kevin Bingham]

My applogoes, I have fat-fingure-trouble today, my email address is k_bingham@excite.co.uk, please could you re-send that link content. Thanks

I ran the "who -R" command and got back my own NT machine name and the HP box name with out any domain qualification :
hp1:/#who -R
root console May 22 13:11 (hp1)
root pts/ta May 22 13:19 (kbingham)

I am not sure what you wanted from this output?

Kevin

[Pedro Sousa]

this "kbingham" is the name of your NT system, correct?
Do you have it on the user's home .rhosts file?
I think so...
check the doc and then say something.

[Kevin Bingham]

Hi,
Thanks for the email. Unfortunately it doesn't help much. I have all of the relevant entries in hosts and .rhosts. (I double checked everything mentioned in the email) I recreated the problem, and while it was waiting to time out, I ran the "who -R" command, it did NOT give an entry for the rsh session which was busy timing out. Those entries that were returned reflected the active telnet sessions. the host names in () were as per the /etc/hosts file without domain qualification.

I have a hunch that it is related to the socket not being released by the HP box, as per my original netstat output in the original posting. How can I force the release of the sockets in use. ntsrv is the hostname of the NT SBS machine and hp1 is the HP box.

I will have a look on the MS support pages for something related to this problem, but I am not confident of finding anything useful there.

Thanks so far,
Kevin

[rick jones]

Possibly a few things interacting...

Remsh is constrained to using "priviledged ports" below 1024.

The side which initiates connection close is supposed to enter TIME_WAIT for 2*MSL (MSL == Maximum Segment Lifetime). This is to preserve TCP connection state to the point where there is a statistical certainty that any segments from that connection that were delayed or duplicated on the net have all timed-out and died. Otherwise, it is possible for those segments to be accepted by the new connection of the same name (local/remote IP, local/remote port) and cause silent data corruption.

If you try to do remshes "too fast" you can have all the four-tuples between the two machines in TIME_WAIT. If the connection timeout is shorter than TIME_WAIT, your connections can timeout.

There are ways to go from TIME_WAIT to established, but they rely on the new connection's Initial Sequence Number being in just the right range. As people push for increasingly random ISN's that becomes increasingly rare.

TIME_WAIT under HP-UX 11 defaults to 60 seconds, which is about as low as I would consider making I for anything other than a fully crash and burn environment. If you really really feel you must alter the length of TIME_WAIT, use ndd to set tcp_time_wait_interval in units of milliseconds (ftp://ftp.cup.hp.com/dist/networking/briefs/)

In SPECweb96 testing, it became known that the TCP stack in NT would, well, cheat TIME_WAIT. NT would have a limit to the number of TCP Protocol Control Blocks. (user configurable). If that limit was reached, and a new connection needed to be made, and if there were a connection in TIME_WAIT, the NT TCP would steal that PCB and effectively short-circuit TIME_WAIT. Doubleplusungood. But it means that if NT were the side to initiate the close, it might not get the pauses. Sadly, crime does sometimes pay :) In its favor (?) NT did default to a fully RFC compliant two minute TIME_WAIT length. You can find the NT ways to tune both those things burried in SPECweb96 submittals using NT - check the tuning settings used.

You might also check to make sure that for subsequent remsh commands, the client side (NT in this case?) is not using the same port number immediately and that it is indeed at least cycling through the reserved port range.

[Kevin Bingham]

Hi Rick,

This sounds like it describes the problem. We use an automated script based mechanism for invoking compiles and links of our HP code from NT using rsh on NT. Where the size of the code being compiled and linked is small, the turn around time would be rather quick, meaning the next rsh call could be invoked inside the minimum timeout period.

What is strange to me is that the behaviour is not consistent. I wrote a simple NT batch file (rsh_hp1.bat) containing the following commands:
===============================
rsh hp1 -l userid set
rsh hp1 -l userid env
rsh_hp1.bat
===============================

When I invoked this from a command prompt on NT, the first rsh call goes through fine, the second one times out, then the bat file calls itself, and the subsequent rsh calls go through fine, with a failure rate of perhaps 10%. What I don't understand is the inconsistent behaviour. During the first cycle of the bat file, the failure rate (timeouts) is 50%, after that it varies. While this test was running, the HP box had just been cold started and was waiting with only one telnet session running (besides system processes), and the NT machine was also under no stress.

You made reference to checking that the subsequent remsh commands on the NT side are not using the same port number.
How would I check this? (It seems highly possible that this might be the case.)

Thanks in advance
Kevin