Poor Network Performance

[JCI IT Unix]

I have a huge amount of data being copied from several servers to a ServiceGuard package file system. When I copy the data using rcp to the package name the performance is extremely slow. When I copy the same data using rcp to the server hostname instead of the package name the performance is extremely fast. What is going on?

[Steven E. Protter]

Note: rcp is not secure.

lanadmin -x 0

replace the 0 with the number from lanscan output.

lan1 is 1

If your network is not performing up to full duplex and card speed, there are steps to fix it. Make sure your switch is set to the exact NIC settings you want, without auto negotiate unless you have a 1000 BaseT NIC.

You can then hard code the network settings in /etc/rc.config.d/hpbtlanconf

Post anything unusual you find in your setup.

SEP

[Con O'Kelly]

Hi Burk

Firstly I have to admit I have no knowledge of MC/SG so could be way off the mark with the following suggestion!

The first thing I'd check is that the hostname may use a different Network Interface from the package name.
If so - check the setup of Interfaces, ie are they both at 100FD and have switch ports for each interface been set to 100FD rather than autonegotiate.

Cheers
Con

[Bill Hassell]

As mentioned, you are likely using a different interface card when you use the package name versus the hostname. The easiesyt way to check is to start the transfer and see which LAN card's light is flashing. As mentioned, 100Mbit LAN connections can cause severe slowdowns in performance if the duplex and bit rate are not matched. For 100Mbit connections, locking the connection on the switch AND the I/O card will solve the problem.

[Mark Grant]

It could also be a routing problem. I would suggest you try a "traceroute" from both machines to the other using the hostname and the package name and see if packages take the same route.

[Brian Markus]

The HPUX Porting site has a copy of ethereal-0.9.15 (18 Sep 2003)which is a network traffic analyzer (sniffer).

http://gatekeep.cs.utah.edu/

You can follow the packets and see what's going on. It's most likely routing or DNS. The Ethereal site has a great tutorial on how to use the tool.

http://www.ethereal.com/

Hope this helps

-Brian.

[JCI IT Unix]

The network card is a 1000Base-SX and it is the same network card for both the hostname and the package name. A network sniffer shows that when the client servers send its data to the package name it will send a small amount of data and then stop for a second, send a little more and stop for a second. This happens over and over. When the client sends to the hostname instead of the package name, same network card, the client sends the data without delays.

[Robert-Jan Goossens]

Burk,

Check this doc,

Document description: Incorrect TCP and UDP checksum generated by 1000-Base cards*
Document id: KBRC00012966

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000070346466

Regards and a good weekend.

Robert-Jan

[Mark Greene_1]

Are both the server host namd and the package name in the /etc/hosts file on the server? What is your /etc/nssswitch.conf file settings? The delay may be due to IP/name resolution problems.

mark

[JCI IT Unix]

The hostname and the package name are in both the /etc/hosts file and DNS. The nsswitch.conf file is set to look at files first and then DNS. Testing proves it is resolving the names.

[Ron Kinner]

Could you post a little bit of the sniffer output for each transfer? Might be able to see what is happening.

Ron

[JCI IT Unix]

Sniffer shows that when the server su58 sends data via rcp to the package, su58 will send a small amount of data and then pause for a second, send a little more and pause for a second, this is happening over and over until complete. When server su58 sends data to the hostname instead of the package name su58 (sender) does not pause at all.

[Ron Kinner]

Yes but I'm interested in the details in the headers. What about the window sizes? Any flags set? Are you getting regular ACKs? Tomeouts? Dropped packets?

If you want to scrub your IP addresses that's fine. Stick the printout in Word or other text editor and replace your first three octaves with A.B.C. Please leave the last octave so I can tell them apart.

Ron

[Mark Greene_1]

Does netstat -r show different gateways for the package vs. the name? Is gated running? Are you running a Vlan on either system?

mark

[rick jones]

Indeed, getting some of the packet trace would be very helpful...if we are going to see the MSS (TCP data size) options and such, the trace will need to start from just before you initiate the rcp.

Also, if you could run netstat -p tcp before a transfer, and then again after a transfer, and then use beforeafter <> to subtract one from the other, that might help as well.

As for duplex mismatches, my personal view is that one should look at the lanadmin stats first before embarking on a campaign to hard-code 100BT settings (especially if there is gigabit invovled since one shoul d_never_ try to hardcode copper gigabit...)

If there is a duplex mismatch, then if the interface description field says full-duplex, there should also be FCS errors in the statistics.

If the interface description field says half-duplex, there should also be "late collisions" (not just collisions, but _late_ collisions) in the statistics.

[Stuart Abramson_2]

Here's a shot: Does the package mount the file system from an NFS mount from the server? If so, then you are going through NFS to use the package name, and going direct to disk when you use the hostname.

We had this problem one time.