TCP connections time out after moving to new datacentre

[Geever Paul]

I have 2 HP9000 L Class boxes one in our office and other one recently moved to a datacentre.

This server is connected to a cisco switch and cisco PIX firewall at the datacentre for secutity filtering.

The server at Datacentre timesout on SQL,telnet,etc.. seesions in 20- 30 minutes to local Windows connections (within datacentre) and remote PC connections

I have DEC2002 patch set installed on the server. I have noticed the VNC sessions works fine but any other sessions from a windows PC/Server times out...

Any ideas... do you think "ndd" should be used ??


Thanks
G.Paul

[Michael Steele_2]

Sounds like your firewall closed some ports. You can verify this with netstat and lsof. I believe your vnc runs on port 5902.

'netstat ???an | grep ???i ???e vnc ???e 5902'

'lsof -i udp:5902'

???lsof ???i tcp:5902???

Try telnet'ing to the port to see if its open:

# telnet xxx.xxx.xxx.xxx 5902
Trying...
Connected to `hostname`
Escape character is '^]'.

Ports in HP-UX are commented out in /etc/services to be closed, but a separate application like your firewall will be on a router. Check with your network administrator. Common ports that should be open are 'telnet' on port 23, ftp on 21, ntp on 123, http on 80: If these don't show up in netstat or lsof then its a firewall blockage.

Here's a download of 'lsof' if you don't have it yet:

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.64/

[Steven E. Protter · Accepted solution]

Another less likely possibility:

The Cisco switch is messing with you.

According to HP there are known problems with Cisco switches set on auto negotiate with HP boxes.

I had the problem myself, lanadmin -x 1 showed 100 BaseT full-duplex, yet I was having all kinds of problems with stability doing Ignite tranfers.

I took the following steps.

1) Set the Cisco switch to 100 BaseT Full-Duplex Manual(hard code) on the switch port. Okay the switch guy did that.

2) Booted the box. lanadmin now showed half duplex.

hmmmm

3) Hard coded the proper settings into the file /etc/rc.config.d/hpbtlanconf

booted again.

Problem solved.

These were L2000/rp5450 boxes with built in and add in 100 BaseT NIC's

My suggestion is probably worth looking into.

SEP

[Ron Kinner]

While some firewalls will drop a connection which lasts "too long" if the problem is also being seen on local connection which have nothing to do with the firewall then it's probably not the firewall. I like Steven's idea with the duplex mismatch creating errors. Some switches will reset a port which has too many errors and this could be what is happening. There should be something in the switch's log files if this is what is happening or you should see a lot of errors and resets on the switch port. It's always best to hard code the duplex setting on both ends of a circuit to a server. (Steven, the reason your Lanadmin showed half duplex after you had the network guy set the switch side to full is because that is the way autonegotiate is supposed to work. If the other side is not set to autonegotiate it has to go to half. I think the reason they did it that way is because they figured a accidental full duplex connection on a hub would mess up the whole hub so by going to half they are playing it safe. Obviously this was before the widespread use of switches.) You should be able to influence how fast this disconnect occurs by sending a lot of traffic in one direction or the other. (One way will work fine. The other will run slow and cause a lot of errors.)

Ron

[Geever Paul]

Thanks for the responses...guyz

I am gonna change the switch to Manual 100 Full DUX and see if the problem goes away!..

the behaviour doesn't seems to be a full port reset.. the connections that are not active for 20-30 minutes get disconnected... while other active connections works fine.

There's no timeout on the profiles or on the switch..

Say if I open an SQL Plus from a windows PC and leave it for 20 minutes..without touching...it will disconnect from the database.. the same with Telnet...and other applications...

When I mentioned the local PCs...I ment the PC/Servers at the datacentre which are on the other side of the firewall...( Yes, it's passing trhu the filtering on the firewall before reaching the PC..

Thanks again, and let me know if you have any suggesions ??? patches ?? (I have DEC 2002) or specific NDD commands ??

I will let you know the Manual settings make a diffrence


G.Paul

[Todd Whitcher]

Also something to consider is shell auto timeouts.

Check for the following in your profiles:

ITRC DOC KBRC00000118
C-SHELL (/bin/csh)

In the HP-UX environment, the C Shell has an automatic logout feature, which automatically logs the user out of their session after one hour of inactivity.

This feature can be overwritten either locally (for a single-user) or globally (for all users):

LOCALLY: Set the following resource line in the user's $HOME/.cshrc file:

set autologout = 0

NOTE: Setting the autologout resource to 0 will set the effective timeout to infinity; forcing the window to stay up as long as the user is logged in.

GLOBALLY: Set the following resource line in the system-wide /etc/csh.login file:

set autologout = 120

NOTE: Setting the autologout resource to 120 will set the effective timeout to 2 hours (or 120 minutes).

NOTE: The autologout resource in the C Shell is set in minutes. Any number of minutes can be specified to this resource's value.

KORN SHELL (/bin/ksh)

In the HP-UX environment, the Korn Shell has a timeout feature called TMOUT. Therefore, if windows are disappearing in your user's Korn Shell environment, you may want to use the env command to look at the environment variable set for this user:

/usr/bin/env

If the TMOUT= environment variable is set to any value greater than 0, the Korn Shell windows will timeout at this specified time.

This feature can be set either locally (for a single-user) or globally (for all users):

LOCALLY: Set the following resource line in the user's $HOME/.profile file:

TMOUT=0

NOTE: Setting the autologout resource to 0 will set the effective timeout to infinity; forcing the window to stay up as long as the user is logged in.

GLOBALLY: Set the following resource line in the system-wide /etc/profile file:

TMOUT=3600

NOTE: Setting the autologout resource to 3600 will set the effective timeout to 1 hour (or 3600 seconds).

NOTE: The TMOUT resource in the Korn Shell is set in seconds. Any number of seconds can be specified to this resource's value

[Michael Steele_2]

I believe one of the important comments being made by one of the other responders is that manual settings made to one node in the network should be followed up by manual settings to all the nodes in the network.

Consequently, nothing should be left automatic anywhere in the network(?).

Please correct me if I'm wrong though, I've also never encountered this 'auto sense' configuration problem to this degree.

[Bill Douglass]

That 20-minute time-out sounds familiar. I get that on idle SSH connections to our DMZ boxes, which pass through a PIX.

BTW, none of these are HP-UX systems, so I suspect it is a PIX issue.

Check with your firewall admin. I suspect that is a default in the PIX configuration.

The auto-config problem manifests itself most noticably during heavy-traffic periods. You will notice that throughput on such connections is very slow, and your switches may be logging errors on the specific links. I've also had boxes (Sun HME interfaces) flap so badly that the Catalyst switches would disable the port.

[Ron Kinner]

OK, I misunderstood the reference to local windows connections. If everything is passing through the PIX and these are idle connections that are dropping I would definitely look at the PIX. Default idle connection timeout on a PIX with the latest software is 1 hour but this could have been changed by the admin or it could have been different on an earlier version of the PIX software. It also varies depending on the type of connection which would explain why the VNC works. See

http://www.cisco.com/univercd/cc/td/doc/product/iaabu/pix/pix_sw/v_63/cmdref/tz.htm#1026093

Ron

[Geever Paul]

Thanks Every one for your valuable replys.

The problem was a combination of Switch and the Firewall. After changing the switch and the server to fixed 100BT Full duplex the time out problem in the middle of SQL and telnet sessions gone away.
We also seems to fix other issues we had with Netbackup backups timing out with Error 41 (network timeout) when it backsup local drives..

The telnet inactive session time out is still occuring, but we know it's exactly after 1hr which points to the PIX timeout on idle sessions:..We are still investigating the exact parameter within PIX (pigs)

thanks again for your contributions...Fourms Rules....!

[Kevin Wright]

the Pix parameter is conn if I remember right.

timeout conn hour/min/sec.

[A. Clay Stephenson]

TIMEOUT XLATE HH:MM:SS CONN HH:MM:SS

(Timeout for PIX VPN translate and idle connections).