topic Re: Poor Network Performance in Operating System - HP-UX

Poor Network Performance

JCI IT Unix — Thu, 30 Oct 2003 19:38:57 GMT

I have a huge amount of data being copied from several servers to a ServiceGuard package file system. When I copy the data using rcp to the package name the performance is extremely slow. When I copy the same data using rcp to the server hostname instead of the package name the performance is extremely fast. What is going on?

Re: Poor Network Performance

Steven E. Protter — Thu, 30 Oct 2003 20:04:49 GMT

Note: rcp is not secure.

lanadmin -x 0

replace the 0 with the number from lanscan output.

lan1 is 1

If your network is not performing up to full duplex and card speed, there are steps to fix it. Make sure your switch is set to the exact NIC settings you want, without auto negotiate unless you have a 1000 BaseT NIC.

You can then hard code the network settings in /etc/rc.config.d/hpbtlanconf

Post anything unusual you find in your setup.

SEP

Re: Poor Network Performance

Con O'Kelly — Thu, 30 Oct 2003 20:05:20 GMT

Hi Burk

Firstly I have to admit I have no knowledge of MC/SG so could be way off the mark with the following suggestion!

The first thing I'd check is that the hostname may use a different Network Interface from the package name.
If so - check the setup of Interfaces, ie are they both at 100FD and have switch ports for each interface been set to 100FD rather than autonegotiate.

Cheers
Con

Re: Poor Network Performance

Bill Hassell — Thu, 30 Oct 2003 21:22:01 GMT

As mentioned, you are likely using a different interface card when you use the package name versus the hostname. The easiesyt way to check is to start the transfer and see which LAN card's light is flashing. As mentioned, 100Mbit LAN connections can cause severe slowdowns in performance if the duplex and bit rate are not matched. For 100Mbit connections, locking the connection on the switch AND the I/O card will solve the problem.

Re: Poor Network Performance

Mark Grant — Fri, 31 Oct 2003 01:23:29 GMT

It could also be a routing problem. I would suggest you try a "traceroute" from both machines to the other using the hostname and the package name and see if packages take the same route.

Re: Poor Network Performance

Brian Markus — Fri, 31 Oct 2003 02:20:16 GMT

The HPUX Porting site has a copy of ethereal-0.9.15 (18 Sep 2003)which is a network traffic analyzer (sniffer).

http://gatekeep.cs.utah.edu/

You can follow the packets and see what's going on. It's most likely routing or DNS. The Ethereal site has a great tutorial on how to use the tool.

http://www.ethereal.com/

Hope this helps

-Brian.

Re: Poor Network Performance

JCI IT Unix — Fri, 31 Oct 2003 08:27:11 GMT

The network card is a 1000Base-SX and it is the same network card for both the hostname and the package name. A network sniffer shows that when the client servers send its data to the package name it will send a small amount of data and then stop for a second, send a little more and stop for a second. This happens over and over. When the client sends to the hostname instead of the package name, same network card, the client sends the data without delays.

Re: Poor Network Performance

Robert-Jan Goossens — Fri, 31 Oct 2003 08:41:51 GMT

Burk,

Check this doc,

Document description: Incorrect TCP and UDP checksum generated by 1000-Base cards*
Document id: KBRC00012966

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000070346466

Regards and a good weekend.

Robert-Jan

Re: Poor Network Performance

Mark Greene_1 — Fri, 31 Oct 2003 10:39:25 GMT

Are both the server host namd and the package name in the /etc/hosts file on the server? What is your /etc/nssswitch.conf file settings? The delay may be due to IP/name resolution problems.

mark

Re: Poor Network Performance

JCI IT Unix — Fri, 31 Oct 2003 10:46:56 GMT

The hostname and the package name are in both the /etc/hosts file and DNS. The nsswitch.conf file is set to look at files first and then DNS. Testing proves it is resolving the names.

Re: Poor Network Performance

Ron Kinner — Fri, 31 Oct 2003 13:45:15 GMT

Could you post a little bit of the sniffer output for each transfer? Might be able to see what is happening.

Ron

Re: Poor Network Performance

JCI IT Unix — Fri, 31 Oct 2003 14:00:40 GMT

Sniffer shows that when the server su58 sends data via rcp to the package, su58 will send a small amount of data and then pause for a second, send a little more and pause for a second, this is happening over and over until complete. When server su58 sends data to the hostname instead of the package name su58 (sender) does not pause at all.

Re: Poor Network Performance

Ron Kinner — Sat, 01 Nov 2003 07:56:19 GMT

Yes but I'm interested in the details in the headers. What about the window sizes? Any flags set? Are you getting regular ACKs? Tomeouts? Dropped packets?

If you want to scrub your IP addresses that's fine. Stick the printout in Word or other text editor and replace your first three octaves with A.B.C. Please leave the last octave so I can tell them apart.

Ron

Re: Poor Network Performance

Mark Greene_1 — Mon, 03 Nov 2003 12:55:45 GMT

Does netstat -r show different gateways for the package vs. the name? Is gated running? Are you running a Vlan on either system?

mark

Re: Poor Network Performance

rick jones — Mon, 03 Nov 2003 12:58:03 GMT

Indeed, getting some of the packet trace would be very helpful...if we are going to see the MSS (TCP data size) options and such, the trace will need to start from just before you initiate the rcp.

Also, if you could run netstat -p tcp before a transfer, and then again after a transfer, and then use beforeafter <> to subtract one from the other, that might help as well.

As for duplex mismatches, my personal view is that one should look at the lanadmin stats first before embarking on a campaign to hard-code 100BT settings (especially if there is gigabit invovled since one shoul d_never_ try to hardcode copper gigabit...)

If there is a duplex mismatch, then if the interface description field says full-duplex, there should also be FCS errors in the statistics.

If the interface description field says half-duplex, there should also be "late collisions" (not just collisions, but _late_ collisions) in the statistics.

Re: Poor Network Performance

Stuart Abramson_2 — Mon, 03 Nov 2003 13:24:51 GMT

Here's a shot: Does the package mount the file system from an NFS mount from the server? If so, then you are going through NFS to use the package name, and going direct to disk when you use the hostname.

We had this problem one time.