topic Re: Slow opening up a connection to HP-UX in Operating System - HP-UX

Slow opening up a connection to HP-UX

echong — Tue, 23 Aug 2005 13:09:38 GMT

Hi,

I have a server that takes a long time to establish a network connection. It is a HPUX 11i server running Openview Operation and Oracle. When I try to do anything related to network, such as telnet, nslookup or a OVO command, it will hang for awhile before it returns to normal output. But for connection that is already open, such as my X session, or telnet that is already opened, no delay at all. There is no error on dmesg or syslog. nfile and nproc looks good from "sar -v". No network error from lanadmin. The problem only happens intermittently.

Do you have any suggestion where else I can look for problem?

Thanks

Re: Slow opening up a connection to HP-UX

RAC_1 — Tue, 23 Aug 2005 13:11:25 GMT

How is server utilization. and not just cpu utilization. how about disk, network and memory??

Re: Slow opening up a connection to HP-UX

DCE — Tue, 23 Aug 2005 13:14:55 GMT

This sounds like a dns issue. Check your dns settings

Re: Slow opening up a connection to HP-UX

Florian Heigl (new acc) — Tue, 23 Aug 2005 13:18:06 GMT

I could imagine this to be a DNS issue, like when the primary server sometimes is unavailable or needs a longer time to answer (most commonly seen with w2k/w2k3 DNS or configuration problems in bind)

if You want to look into this, I'd recommend to use a simple script like i.e.

#!/bin/sh
# this script will check Your every 10 seconds for one hour
i=0
while [ 360 -ge $i ]
do
sleep 10
timex nslookup #some server > /tmp/lookups.log
i=$(( $i + 1 ))
done

after one hour, look through the log file and check if every lookup finished within reasonable times (well below two or three seconds, that is)

Re: Slow opening up a connection to HP-UX

Alex Lavrov. — Tue, 23 Aug 2005 13:19:43 GMT

Couple of things to check.

1:
nslookup
nslookup

nslookup
nslookup

On both,client machine and server.

2:
check network settings on your server and on the switch that they are configured to the same values. 100 or 1000 and also Half-duplex or full-duplex.

lanadmin -x (or maybe it's capital X ... to be honest, I don't remember, one is used to view, another to set)

Re: Slow opening up a connection to HP-UX

echong — Tue, 23 Aug 2005 13:38:59 GMT

I just use DNS as an example, so I am sure it is not a DNS problem. In fact, DNS server is running on the same box.

The system resouce is not a problem. Checking from glance, everything looks good.

No collision error.

There is no problem once the connection is established.

Thanks.

Re: Slow opening up a connection to HP-UX

echong — Tue, 23 Aug 2005 13:41:20 GMT

Forgot to mention that it happens on both in-bound or out-bound connections.

Re: Slow opening up a connection to HP-UX

Mel Burslan — Tue, 23 Aug 2005 14:29:45 GMT

Eric,

It really does not matter if DNS is running on the same box or not. It mainly depends how your server searches or tries to resolve the server names.

if you happen to look at your /etc/nasswitch.conf and /etc/resolv.conf you can figure out a lot. If you can not tell what those files mean to you, come back and post them here and myself or someone else most probably can tell you what may be going wrong with your system, because all the symptoms you are telling on your original post, points to a DNS related issue which has been discussed many time over in these forums

Re: Slow opening up a connection to HP-UX

Bill Hassell — Tue, 23 Aug 2005 15:59:57 GMT

Your delay is roughly 20 seconds per failed DNS server. So when you telnet to the box, start counting. About 15-20 seconds means that the first DNS server specified in /etc/resolv.conf is not responding. If the delay is about 30-40 seconds, then the first 2 entries are not responding, and so on.

To prove this, edit the /etc/nsswitch.conf file and fix the (very unreliable) entry for hosts: to read as follows:

hosts: files [NOTFOUND=continue UNAVAIL=continue] dns [NOTFOUND=return UNAVAIL=continue TRYAGAIN=return]

Now edit /etc/hosts to add the IP address and name of the box you're on (the client) to the hosts file. Now try logging on. You should see instant response, indicating that you have DNS problems. One of the classic DNS failures is thatthe server does not keep reverse DNS records and/or does not know it's own name.

Re: Slow opening up a connection to HP-UX

John Waller — Wed, 24 Aug 2005 01:54:07 GMT

It could also be a DNS problem relating to the client you are logging in from. We use VPN through the internet to connect from home, as the IP address we connect from is not known to the company's DNS server we get a delay of about 30 seconds when we telnet to the server. To confirm this, you could temporily rename /etc/resolv.conf and try to connect again. By renaming resolv.conf you are forcing the server to only use the /etc/hosts file so if it can't find the entry it will give up imediatly.

Re: Slow opening up a connection to HP-UX

Ranjith_5 — Wed, 24 Aug 2005 02:12:33 GMT

Hi,

can you have a look at your name resolution..check your /etc/nsswitch.conf file for the hosts: entry. Change the host entry as follows and try telnet.

hosts: files

Regards,
Syam

Re: Slow opening up a connection to HP-UX

Sudeesh — Wed, 24 Aug 2005 02:19:07 GMT

try to telnet using IP address... if u see a considerable diffrence in response, you can confirm its a dns issue.

Sudeesh

Re: Slow opening up a connection to HP-UX

echong — Wed, 24 Aug 2005 10:13:45 GMT

Ok. Seems everyone agree this is a DNS problem, I will investigate on this.

I am running forward DNS on the same server, and using three other servers as forwarders. I will start the tracing to see how it perform. Does anyone know a good source on DNS performance tuning??

Thanks for everyone's input.

Re: Slow opening up a connection to HP-UX

Bill Hassell — Wed, 24 Aug 2005 10:39:01 GMT

This is very likely not a performance issue. The server is dead or not replying to your HP-UX system's requests. No need to check the DNS server load, just use nslookup and post the error messages. If you see an error about the server, it is likely not configured to know it's own name (real common mistake in PC DNS servers). Test for forward and reverse lookups by using nslookup with the name of your client and then again with the IP address of your client. That's what telnetd is doing.

Re: Slow opening up a connection to HP-UX

Ranjith_5 — Wed, 24 Aug 2005 10:47:51 GMT

Hi Eric,

Exactly..there shouldn't be a performance problem of DNA. Mostly a local name resolution problem..look at /etc/nsswitch.conf first, if ok then go ahead and check the /etc/resolv.conf and other things. I had also faced the same problem previously..After correcting the /etc/nsswitch.conf file it go resolved. In my case the nsswitch.conf was pointing to a NIS server which was unavailable.

If it takes lon time to execute SAM/swinstall mostly the problem is 99% with name resolution. My problem got resolved by doing #cp /etc/nsswitch.files /etc/nsswitch.conf

Regards,
Syam

Re: Slow opening up a connection to HP-UX

Ranjith_5 — Wed, 24 Aug 2005 10:50:10 GMT

sorry last post had spelling errors..i was too hurry...

=======
Exactly..this shouldn't be a performance problem of DNS. Mostly a its a local name resolution issue..look at /etc/nsswitch.conf first, if ok then go ahead and check the /etc/resolv.conf and other things. I had also faced the same problem previously..After correcting the /etc/nsswitch.conf file it got resolved. In my case the nsswitch.conf was pointing to a NIS server which was unavailable.

If it takes long time to execute SAM/swinstall mostly the problem is 99% with name resolution. My problem got resolved by doing #cp /etc/nsswitch.files /etc/nsswitch.conf

Regards,
Syam

Re: Slow opening up a connection to HP-UX

echong — Wed, 24 Aug 2005 14:52:11 GMT

Since this problem happen intermittently, so I assume it is not a configuration issue, otherwise I will have this problem all the time. Right?

Re: Slow opening up a connection to HP-UX

Mel Burslan — Wed, 24 Aug 2005 15:04:25 GMT

Hypotetical scenario:
let's say your /etc/nsswitch.conf looks like this:

hosts: dns [NOTFOUND=continue UNAVAIL=continue] files [NOTFOUND=continue UNAVAIL=continue TRYAGAIN=return]

and your /etc/resolv.conf looks like this:

domain mycompany.com
nameserver 10.11.12.200
nameserver 10.11.12.201
nameserver 10.11.12.202

under these circumstances, if nameserver 10.11.12.200 goes down, whereas nameserver 10.11.12.201 and nameserver 10.11.12.202 are up, you will experience about a 20 seconds delay as Bill indicated in his first post in this thread.

if nameserver 10.11.12.200 and nameserver 10.11.12.201 go down together whereas nameserver 10.11.12.202 staying up, you will experience about a 40 seconds delay as your server will wait about this much time to get an answer from each server.

I think you've got the idea. So, if you guarantee that, when you are experiencing this intermittent delay, these dns servers are up and serving addresses without any problems, yes I must agree with you, your problem is not with DNS, but I found it, 99 times out of 100 that it is related to DNS servers. The box may be up but the DNS admin maybe making a massive address change, which will knock down your main DNS server offline while the zone files get generated, or, somebody maybe attacking your DNS server for vulnerability testing (this is true only if your internal and external DNS server is the same one). Give any reason that you can think of which can knock its performance down or knock it totally offline here.

So, before ruling out of DNS issue, first try Bill's suggestion and change the search order in the nsswitch.conf file as he indicated to files first dns second. Then put the ip addresses of the frequently accessed hosts into /etc/hosts file, then see if you get the same latency problems. I can almost assure you that you will not.

Re: Slow opening up a connection to HP-UX

Tim D Fulford — Wed, 24 Aug 2005 16:12:40 GMT

try this for a test

timex netstat -rn

timex netstat -r

If netstat -r takes a while it is almost definitely a resolution (DNS tyoe) problem.

I get the feeling you want something other than "DNS is the problem"... so check out your network

% netfmt -t 20 -f /var/adm/nettl.LOG000
% lanadmin .. this is menu driven so check out ecah interface
% lanadmin -x .. check if all the interfaces are correct, best to be auto detect.
% lanscan
Do some lanscans on other HP hardware & note the mac addresses. this wil check out physical connectivity. then do
% linkloop -i

Also try ftping 10MB files and checking the transfere speed is acceptable.

Regards

Tim

Re: Slow opening up a connection to HP-UX

Ranjith_5 — Wed, 24 Aug 2005 22:08:52 GMT

Hi Eric,

My opinion is before you go for an advanced troubleshooting, take 2 mins to heck the basic configuration files like nsswitch.conf. Any way you are not loosing anything. If this is ok then Agree you go ahead and troubleshoot your DNS. Why dont you try changing the search order of nsswitch.conf. I have already mentioned in my last post abt this. This takes only negligible time ad you can be confident when doing the next level of troubleshooting.

Regards,
Syam