Operating System - HP-UX
1833541 Members
2724 Online
110061 Solutions
New Discussion

tcp open/close and shonky ip configuration

 
SOLVED
Go to solution
steven Burgess_2
Honored Contributor

tcp open/close and shonky ip configuration

Hi everyone

It has become apparent that a couple of our middleware servers have been configured rather shonkily from an ip point of view with regard to resolution of addresses between each other

I am going to try and explain as best I can

On our systems we have 2 hostnames, cards etc. 1 configured for the production LAN the 2nd for backup LAN

I am going to use dummy ip addresses for this example

on server1 (nslookup - using local hosts files)

server1=164.39.23.5
server2=101.15.11.3

on server2 (nslookup - using local hosts files)

server1=101.15.11.4
server2=164.39.23.6

If i try to rcp a file from server1 to server2 it hangs at the end of the copy.

Here's what I think is happening, but would like someone, if possible to go into fine detail with regard to how tcp/ip handles it ACK's etc and tell me if i'm wrong.

So...

Connection initiated from server1 out of 101 address to default gateway. gateway looks up route and passes connection to server via 101 interface.

Connection comes in, tcp handshake initiated.
(heres where the problem starts) I come in on one address (101), yet when the server does a lookup for me, he see's me on another address (164), what does he then do ? initiate the session back to the server, but passes the data connection out of the 164 address as opposed to the incoming 101 address ? it's then passed through the router and back to server1 on it's 164 address ? We then essentially have a data transmission across 2 subnets......

It's not working, I will ammend the config so they match, I just want to understand exactly how the boxes acknowledge eachother etc

Chaars

Steve



take your time and think things through
17 REPLIES 17
Steven E. Protter
Exalted Contributor

Re: tcp open/close and shonky ip configuration

Steve,

I'm not lecturing on security, but openssh does have a -v mode.

That might give you some insite in how this process works.

I see no -v mode for rcp, which I barely remember how to use.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
steven Burgess_2
Honored Contributor

Re: tcp open/close and shonky ip configuration

thanks
take your time and think things through
steven Burgess_2
Honored Contributor

Re: tcp open/close and shonky ip configuration

boing
take your time and think things through
Sridhar Bhaskarla
Honored Contributor

Re: tcp open/close and shonky ip configuration

Steve,

Can you post your netstat -rn and netstat -in on both the boxes?.

Basically 101.15.11.4 and 101.15.11.3 are in the same broadcast domain. They should communicate directory without going through the default gateway anywhere.

I would first check the subnetmasks defined on both the boxes corresponding to 101 interfaces. Make sure they are consistent. It is a quite common error that one is configured with 255.255.255.0 and the other with the default mask.

-Sri
You may be disappointed if you fail, but you are doomed if you don't try
steven Burgess_2
Honored Contributor

Re: tcp open/close and shonky ip configuration

Hi Shridhar

sorry, i have used bad examples. I can't post netstat output

Consider the addresses to be in different classes , on A and the other B

I really wanted to know what happens from the tcp/ip when it receives something it doesn't expect

Rgds
take your time and think things through
Ron Kinner
Honored Contributor
Solution

Re: tcp open/close and shonky ip configuration

The initial TCP/IP connection is based on the results of nslookup on the name of the remote host. This gives an IP address which is used as a destination. The routing table (netstat -rn) is then consulted to determine which interface to use and which is the next hop (gateway). The packet is then sent on to the gateway which forwards it on to the destination. The packet is normally addressed to port 514 since rcp rides on the rsh protocol. If the host is listening on 514 then the threeway handshake completes (return packets being sent to the source address using the local routing table). The TCP/IP stack then passes the ball to the rshd.

From
http://www.private.org.il/mini-tcpip.faq.html#4.%20Of%20the%20rsh%20protocol.


"The rshd listens on TCP port #514. The following info is from the unix rshd man pages :

Service Request Protocol

When the rshd daemon receives a service request, it initiates the following protocol:



The rshd daemon checks the source port number for the request. If the port number is not in the range 0 through 1023, the rshd daemon terminates the connection.

The rshd daemon reads characters from the socket up to a null byte. The string read is interpreted as an ASCII number (base 10). If this number is nonzero, the rshd daemon interprets it as the port number of a secondary stream to be used as standard error. A second connection is created to the specified port on the client host. The source port on the local host is in the range 0 through 1023."

Note: We still have not done any reverse lookup so this optional new connection is made to the original source ip using the requested port and follows the same return path as the replies to the initial connection. This is sort of similar to what ftp does so it may have trouble passing a firewall. This inability to create the stderr back channel connection could cause the thing to hang. It might be interesting to do a
netstat -an |grep a.b.c.d
where a.b.c.d is the ip address of the remote system (and the remote system is on the local subnet to eliminate firewall or router problems) to see if we use one or two connections to do an rcp.

"The rshd daemon uses the source address of the initial connection request to determine the name of the client host. If the name cannot be determined, the rshd daemon uses the dotted decimal representation of the client host's address."

Note: Here we do a reverse lookup of the source IP address to see who started the session.

nslookup a.b.c.d

where a.b.c.d is the source address of the tcp/ip connection.

If this fails we will use the source ip address when we try to authenticate. You should try this on the remote system to see what result you get.

"The rshd daemon retrieves the following information from the initial socket:

A null-terminated string of at most 16 bytes interpreted as the user name of the user on the client host.
A null-terminated string of at most 16 bytes interpreted as the user name to be used on the local server host.
Another null-terminated string interpreted as a command line to be passed to a shell on the local server host. "

Note: In the case of rcp the two user names are the same tho rsh has provisions for using two different names and the command line will do the actual copying. I've never looked at it on a sniffer but I expect it is really something like
cat localhost:localfile1 > remotehost:remotefile1.


"The rshd daemon attempts to validate the user using the following steps:

The rshd daemon looks up the local user name in the /etc/passwd file and tries to switch to the home directory (using the chdir subroutine). If either the lookup or the directory change fails, the rshd daemon terminates the connection."

Note: Here is another thing you can check. Does the user name exist in the /etc/password file and is there a home directory for the user name? Can we write in it if logged in as the user?

"If the local user ID is a nonzero value, the rshd daemon searches the /etc/hosts.equiv file to see if the name of the client workstation is listed. If the client workstation is listed as an equivalent host, the rshd daemon validates the user.

If the $HOME/.rhosts file exists, the rshd daemon tries to authenticate the user by checking the .rhosts file."

Note: Immediately above is the first use of the info from the nslookup a.b.c.d. Is the name we got back from nslookup a.b.c.d in the $HOME/.rhosts? Does the file even exist?

"If either the $HOME/.rhosts authentication fails or the client host is not an equivalent host, the rshd daemon terminates the connection.

Once rshd validates the user, the rshd daemon returns a null byte on the initial connection and passes the command line to the user's local login shell. The shell then inherits the network connections established by the rshd daemon."


So you see it really does not care what you have in /etc/hosts for the name since we do a reverse lookup of the IP address. If it does not appear in /etc/hosts then we ask the DNS server who hopefully has the right answer. Even if the command succeeds there may still be a message sent back to the stderr and it is possible depending on how the software is written that we may suddenly discover that there is no stderr channel open and hang here with the real job done but unable to complete because it can't send the "I made it" message. Near as I can tell the data passes over the original connection and the second connection is only used for stderr.

Ron
steven Burgess_2
Honored Contributor

Re: tcp open/close and shonky ip configuration

Thanks Ron

Digesting...
take your time and think things through
Sridhar Bhaskarla
Honored Contributor

Re: tcp open/close and shonky ip configuration

Hi Steve,

I didn't read the message completely. I didn't notice you said "at the end of copy". I concentrated on "It's not working". So you did have a successful connection.

Looks like Ron threw a lot of light on the issue.

The issue lies in how the IP addressses of one server are configured in DNS on the other server but not the other way.

Reverse lookup is all that matters. Once server1 initiates the connection to server2 through 101 address, server2 will reverse lookup the incoming address to see if it can get the name. It depends on how the IP address (101.15.11.3
) is configured in DNS on server2. It's not in /etc/hosts and I believe you don't have it in DNS. Then it will simply be the IP address. If this is the case, you might have configured your .rhosts to look at the IP address. It matches the incoming address and the rhosts authentication is done. Rest of the flow will follow fin_wait, fin_wait2, time_wait and closed on server1 side and close_wait, last_ack and closed on server2 side which again has nothing to do with the nslookup stuff.

Now, the question is - is it happening with only rcp? How about ftp, telnet, remsh etc.,?

One quick thing you may want to check is the speed on both the interfaces. Make sure they are set to full duplex 100MBPS if that is the speed you are setting it to.

-Sri
You may be disappointed if you fail, but you are doomed if you don't try
Brian Hackley
Honored Contributor

Re: tcp open/close and shonky ip configuration

Steve,

To add more confusion without sniffing/traceing the packets:

Its POSSBILE that you want to use remshd with -s flag in the /etc/inetd.conf file on the target. inetd -c after editing the file. man remshd (looking at my 11.11 box)

You may also want to ensure the hostname lookups on both hosts are set up the way you want the "primary" connection to the target to appear. Lookup with "files" first in /etc/nsswitch.conf and edit /etc/hosts...

Hope that helps.
-> Brian Hackley
Ask me about telecommuting!
Sridhar Bhaskarla
Honored Contributor

Re: tcp open/close and shonky ip configuration

Steve,

Another thing I wanted to post but forgot is to setup ethereal on one of the boxes if you really want to see the handshake.

Once ethereal is loaded (from the porting site, make note of the dependencies), you can start it and capture only packets that go to/from server2. Once the data is captured, you can select "follow tcp stream" to look at the tcp stream of your rcp. You can get the details of each packet with syn and ack numbers. You can drill further down into each packet and check the TCP, IP, Ethenet and Frame headers.

-Sri
You may be disappointed if you fail, but you are doomed if you don't try
rick jones
Honored Contributor

Re: tcp open/close and shonky ip configuration

Just to make sure it comes out - TCP and ACK's care nothing about name to IP address resolution. TCP deals only with IP addresses and TCP port numbers.

If your connections are appearing to "hang" at the end of the transfer, (I take it the files are created, at least partially, at the destination?) it suggests something other than application-level authentication - which one would presume took place at the start of the session.

Definitely try for the packet trace. Don't really worry too much about the "real" IP addresses - quite frankly, if you are basing your security on the obscurity of your IP addresses, well...
there is no rest for the wicked yet the virtuous have no pillows
Jeff Schussele
Honored Contributor

Re: tcp open/close and shonky ip configuration

Hi Steve,

I know this is off-topic, but I'm curious.
Would "shonky" be equivalent to the American "squirrely".

I'm always amused at how the "Queen's English" can be so different throughout the world.

Cheers (Or should I say cheerio?),
Jeff
PERSEVERANCE -- Remember, whatever does not kill you only makes you stronger!
steven Burgess_2
Honored Contributor

Re: tcp open/close and shonky ip configuration

Thanks for the replies everyone.

I'm going to emulate this environment in test and work through to understand all the responses. This is something I have always wanted to understand so will take this opportunity

Jeff...hehehe

Yes , shonky could be translated to "squirrely"

what what dear boy

Chaars

Steven
take your time and think things through
Ron Kinner
Honored Contributor

Re: tcp open/close and shonky ip configuration

Jeff,

Something shonky about your spelling of "squirrely." Down here in the Sinkhole State we spell that "squirrelly" (with two L's.)

Steve,

Is there a firewall involved? If so, look for it to block the stderr connection and cause the failure you are seeing. I take it that the file does get copied?

Ron
Jeff Schussele
Honored Contributor

Re: tcp open/close and shonky ip configuration

Hi Ron,

And you probably spell chilli with just one l too, huh?
Nah - you're right 2 ls for squirelly it is. I stand corrected. It was indeed a shonky spelling my Floridian friend.

Cheers,
Jeff
PERSEVERANCE -- Remember, whatever does not kill you only makes you stronger!
steven Burgess_2
Honored Contributor

Re: tcp open/close and shonky ip configuration

Hi Ron

Will check and keep you posted.

Would you mind explaining this part of your reply

The rshd daemon reads characters from the socket up to a null byte. The string read is interpreted as an ASCII number (base 10). If this number is nonzero, the rshd daemon interprets it as the port number of a secondary stream to be used as standard error. A second connection is created to the specified port on the client host. The source port on the local host is in the range 0 through 1023."


Thanks
take your time and think things through
Ron Kinner
Honored Contributor

Re: tcp open/close and shonky ip configuration

I didn't write the stuff in the quotation marks but my interpretation is as follows:

"The rshd daemon reads characters from the socket up to a null byte. The string read is interpreted as an ASCII number (base 10). If this number is nonzero, the rshd daemon interprets it as the port number of a secondary stream to be used as standard error. A second connection is created to the specified port on the client host. The source port on the local host is in the range 0 through 1023."

Once we strip off the TCP/IP headers we are left with the rsh data packet. This is parsed by looking for "null bytes" which since we are speaking ASCII are probably nothing more than a (or maybe 3) byte(s) with eight 0's. The first string is read from the start of the packet to the first null byte. If the string is not equal to ASCII 0 (060 in octal) but works out to a decimal number x between 1 and 1023 then the creation of the stderr back channel is being requested and the number is interpreted as the port to which the connection should be made. The rshd then asks the tcp/ip stack to make a second connection back toward the source of the original connection to port x. This connection is used to send back error messages which would normally appear on the console.


A list of possible error messages is given at the bottom of:

http://docs.sun.com/db/doc/816-0211/6m6nc66sv?a=view

Note that if the command executes correctly a "0" is returned over the stderr channel. I think the rcp command is waiting for this 0 to know that the remote copy completed successfully so it can return cleanly. If it doesn't get it I expect it just hangs or times out which sounds to me like your problem.

So in summary the rsh command can ask for the creation of a stderr back channel on port x which would mean we would have two tcp/ip connections active which we could see with netstat -an:

tcp 0 0 A:y ==> B:514 ESTABLISHED

tcp 0 0 A:x <== B:z ESTABLISHED

where A is the source address of the PC where you started the rsh or rcp and B is the source address of the remote PC. x and y have to be >0 but < 1023 and z can probably be any port.

Ron