Slow opening up a connection to HP-UX

[echong]

Hi,

I have a server that takes a long time to establish a network connection. It is a HPUX 11i server running Openview Operation and Oracle. When I try to do anything related to network, such as telnet, nslookup or a OVO command, it will hang for awhile before it returns to normal output. But for connection that is already open, such as my X session, or telnet that is already opened, no delay at all. There is no error on dmesg or syslog. nfile and nproc looks good from "sar -v". No network error from lanadmin. The problem only happens intermittently.

Do you have any suggestion where else I can look for problem?

Thanks

[RAC_1]

How is server utilization. and not just cpu utilization. how about disk, network and memory??

[DCE]

This sounds like a dns issue. Check your dns settings

[Florian Heigl (new acc)]

I could imagine this to be a DNS issue, like when the primary server sometimes is unavailable or needs a longer time to answer (most commonly seen with w2k/w2k3 DNS or configuration problems in bind)

if You want to look into this, I'd recommend to use a simple script like i.e.


#!/bin/sh
# this script will check Your every 10 seconds for one hour
i=0
while [ 360 -ge $i ]
do
sleep 10
timex nslookup #some server > /tmp/lookups.log
i=$(( $i + 1 ))
done

after one hour, look through the log file and check if every lookup finished within reasonable times (well below two or three seconds, that is)

[Alex Lavrov.]

Couple of things to check.

1:
nslookup
nslookup

nslookup
nslookup

On both,client machine and server.

2:
check network settings on your server and on the switch that they are configured to the same values. 100 or 1000 and also Half-duplex or full-duplex.

lanadmin -x (or maybe it's capital X ... to be honest, I don't remember, one is used to view, another to set)

[echong]

I just use DNS as an example, so I am sure it is not a DNS problem. In fact, DNS server is running on the same box.

The system resouce is not a problem. Checking from glance, everything looks good.

No collision error.

There is no problem once the connection is established.

Thanks.

[echong]

Forgot to mention that it happens on both in-bound or out-bound connections.

[Mel Burslan]

Eric,

It really does not matter if DNS is running on the same box or not. It mainly depends how your server searches or tries to resolve the server names.

if you happen to look at your /etc/nasswitch.conf and /etc/resolv.conf you can figure out a lot. If you can not tell what those files mean to you, come back and post them here and myself or someone else most probably can tell you what may be going wrong with your system, because all the symptoms you are telling on your original post, points to a DNS related issue which has been discussed many time over in these forums

[Bill Hassell]

Your delay is roughly 20 seconds per failed DNS server. So when you telnet to the box, start counting. About 15-20 seconds means that the first DNS server specified in /etc/resolv.conf is not responding. If the delay is about 30-40 seconds, then the first 2 entries are not responding, and so on.

To prove this, edit the /etc/nsswitch.conf file and fix the (very unreliable) entry for hosts: to read as follows:

hosts: files [NOTFOUND=continue UNAVAIL=continue] dns [NOTFOUND=return UNAVAIL=continue TRYAGAIN=return]

Now edit /etc/hosts to add the IP address and name of the box you're on (the client) to the hosts file. Now try logging on. You should see instant response, indicating that you have DNS problems. One of the classic DNS failures is thatthe server does not keep reverse DNS records and/or does not know it's own name.

[John Waller]

It could also be a DNS problem relating to the client you are logging in from. We use VPN through the internet to connect from home, as the IP address we connect from is not known to the company's DNS server we get a delay of about 30 seconds when we telnet to the server. To confirm this, you could temporily rename /etc/resolv.conf and try to connect again. By renaming resolv.conf you are forcing the server to only use the /etc/hosts file so if it can't find the entry it will give up imediatly.

[Ranjith_5]

Hi,

can you have a look at your name resolution..check your /etc/nsswitch.conf file for the hosts: entry. Change the host entry as follows and try telnet.

hosts: files


Regards,
Syam

[Sudeesh]

try to telnet using IP address... if u see a considerable diffrence in response, you can confirm its a dns issue.


Sudeesh

[echong]

Ok. Seems everyone agree this is a DNS problem, I will investigate on this.

I am running forward DNS on the same server, and using three other servers as forwarders. I will start the tracing to see how it perform. Does anyone know a good source on DNS performance tuning??

Thanks for everyone's input.

[Bill Hassell]

This is very likely not a performance issue. The server is dead or not replying to your HP-UX system's requests. No need to check the DNS server load, just use nslookup and post the error messages. If you see an error about the server, it is likely not configured to know it's own name (real common mistake in PC DNS servers). Test for forward and reverse lookups by using nslookup with the name of your client and then again with the IP address of your client. That's what telnetd is doing.

[Ranjith_5]

Hi Eric,

Exactly..there shouldn't be a performance problem of DNA. Mostly a local name resolution problem..look at /etc/nsswitch.conf first, if ok then go ahead and check the /etc/resolv.conf and other things. I had also faced the same problem previously..After correcting the /etc/nsswitch.conf file it go resolved. In my case the nsswitch.conf was pointing to a NIS server which was unavailable.

If it takes lon time to execute SAM/swinstall mostly the problem is 99% with name resolution. My problem got resolved by doing #cp /etc/nsswitch.files /etc/nsswitch.conf

Regards,
Syam

[Ranjith_5]

sorry last post had spelling errors..i was too hurry...

=======
Exactly..this shouldn't be a performance problem of DNS. Mostly a its a local name resolution issue..look at /etc/nsswitch.conf first, if ok then go ahead and check the /etc/resolv.conf and other things. I had also faced the same problem previously..After correcting the /etc/nsswitch.conf file it got resolved. In my case the nsswitch.conf was pointing to a NIS server which was unavailable.

If it takes long time to execute SAM/swinstall mostly the problem is 99% with name resolution. My problem got resolved by doing #cp /etc/nsswitch.files /etc/nsswitch.conf

Regards,
Syam

[echong]

Since this problem happen intermittently, so I assume it is not a configuration issue, otherwise I will have this problem all the time. Right?

[Mel Burslan]

Hypotetical scenario:
let's say your /etc/nsswitch.conf looks like this:

hosts: dns [NOTFOUND=continue UNAVAIL=continue] files [NOTFOUND=continue UNAVAIL=continue TRYAGAIN=return]

and your /etc/resolv.conf looks like this:

domain mycompany.com
nameserver 10.11.12.200
nameserver 10.11.12.201
nameserver 10.11.12.202

under these circumstances, if nameserver 10.11.12.200 goes down, whereas nameserver 10.11.12.201 and nameserver 10.11.12.202 are up, you will experience about a 20 seconds delay as Bill indicated in his first post in this thread.

if nameserver 10.11.12.200 and nameserver 10.11.12.201 go down together whereas nameserver 10.11.12.202 staying up, you will experience about a 40 seconds delay as your server will wait about this much time to get an answer from each server.

I think you've got the idea. So, if you guarantee that, when you are experiencing this intermittent delay, these dns servers are up and serving addresses without any problems, yes I must agree with you, your problem is not with DNS, but I found it, 99 times out of 100 that it is related to DNS servers. The box may be up but the DNS admin maybe making a massive address change, which will knock down your main DNS server offline while the zone files get generated, or, somebody maybe attacking your DNS server for vulnerability testing (this is true only if your internal and external DNS server is the same one). Give any reason that you can think of which can knock its performance down or knock it totally offline here.

So, before ruling out of DNS issue, first try Bill's suggestion and change the search order in the nsswitch.conf file as he indicated to files first dns second. Then put the ip addresses of the frequently accessed hosts into /etc/hosts file, then see if you get the same latency problems. I can almost assure you that you will not.

[Tim D Fulford]

try this for a test

timex netstat -rn

timex netstat -r

If netstat -r takes a while it is almost definitely a resolution (DNS tyoe) problem.

I get the feeling you want something other than "DNS is the problem"... so check out your network

% netfmt -t 20 -f /var/adm/nettl.LOG000
% lanadmin .. this is menu driven so check out ecah interface
% lanadmin -x .. check if all the interfaces are correct, best to be auto detect.
% lanscan
Do some lanscans on other HP hardware & note the mac addresses. this wil check out physical connectivity. then do
% linkloop -i

Also try ftping 10MB files and checking the transfere speed is acceptable.

Regards

Tim

[Ranjith_5]

Hi Eric,

My opinion is before you go for an advanced troubleshooting, take 2 mins to heck the basic configuration files like nsswitch.conf. Any way you are not loosing anything. If this is ok then Agree you go ahead and troubleshoot your DNS. Why dont you try changing the search order of nsswitch.conf. I have already mentioned in my last post abt this. This takes only negligible time ad you can be confident when doing the next level of troubleshooting.

Regards,
Syam

[Roberto_146]

Hi all from Spain.

I am from the network team, and we are implementing a network with HP-UX servers (Oracle).

We had the same slow connection problem with those machines.

Reading this document (thanks to all), I told the Unix team to add the source IP to the HP-UX hosts file. Everything was solved since that point!!!!

So... this is the solution for all the people who have the same problem. Maybe they can add ranges o pools to solve this issue in a global mode instead of individually.

Thanks to all.

Roberto.
dospuntosd@telefonica.net