network card rate limiting

[slydmin]

Greetings,
Is there a way to rate limit network traffic on a certain NIC on hpux? The network graph shows that the system tops off at 20mbps for all incoming and outgoing traffic. If indeed it is rate limited, is there a way to remove this limit? Use lanadmin, perhaps?

ioscan gives this output
lan 4 0/12/0/0 igelan CLAIMED INTERFACE HP A6825-60101 PCI 1000Base-T Adapter

This an HPUX B11.11 OS on rp5470 hardware.

Any help is appreciated.
Thanks,
-S

[Bill Hassell]

No, there is no rate limiter on the card or in the driver. The first thing to do is to look at lanadmin to see what speed it is actually set to and most important, the duplex setting. If it is auto-negotiating, your switch must do the same thing. The stats on the second page should all be zeros. If you see FCS or alignment errors, or most important, collisions, then your switch is not set correctly at all. If you have a GigaBit switch, it must be set to auto-negotiate, the same as your LAN card. If this is a 100 Mbit switch, make sure lanadmin reports full-duplex setting. Otherwise, the switch and card did not auto-negotiate correctly and you'll have to hardcode 100 Mbit and full duplex.

[slydmin]

There are no FCS or Alignment errors and all counters are 0 for Errors/collisions

Here is the output for the lanadmin
PPA Number = 3
Description = lan3 HP PCI 1000Base-T Release B.11.11.15
Type (value) = ethernet-csmacd(6)
MTU Size = 1500
Speed = 1000000000
Station Address = 0x306ef40cfe
Administration Status (value) = up(1)
Operation Status (value) = up(1)
Last Change = 326
Inbound Octets = 3768091933
Inbound Unicast Packets = 20194077
Inbound Non-Unicast Packets = 1384
Inbound Discards = 0
Inbound Errors = 0
Inbound Unknown Protocols = 18
Outbound Octets = 2796030486
Outbound Unicast Packets = 20568098
Outbound Non-Unicast Packets = 18
Outbound Discards = 0
Outbound Errors = 0
Outbound Queue Length = 0
Specific = 655367
Ethernet-like Statistics Group

Index = 4
Alignment Errors = 0
FCS Errors = 0
Single Collision Frames = 0
Multiple Collision Frames = 0
Deferred Transmissions = 0
Late Collisions = 0
Excessive Collisions = 0
Internal MAC Transmit Errors = 0
Carrier Sense Errors = 0
Frames Too Long = 0
Internal MAC Receive Errors = 0

So why would the server/network limit to 20mbps(which is shown on the MRTG for the switch that it is connected to)?

[Patrick Wallek]

Maybe its not limited and that's all the traffic that is going to this server.

[Bill Hassell]

SInce MTRG is just showing activity, you'll need to test it with a high performance data transfer program, namely ftp. Pick another system with a Gbit card, ideally on the same switch, then start ftp for several hundred megs on each server at the same time. Then check your stats. Note that throughput always depends on both ends of the connection.

[rick jones · Accepted solution]

HP-UX does not by default place any limits on NICs. It will drive them as fast as it can or as fast as it is asked, whichever is less.

Assuming the lanadmin stats are good and don't suffer from "rollover" on the octet (byte) counts, it looks like your average packet size is ~186 bytes in one direction and ~135 in the other.

In addition to testing with something that can do high-throughput (I'd suggest netperf rather than ftp, but I am biased :), and that requires an install step) you might also check the CPU utilization of all the CPUs in your system. Make certain that none of them are (close) to saturation, especially with interrupt time.

Also, at ftp://ftp.cup.hp.com/dist/networking/tools/ you will find a utility called "beforeafter" which you can use to subtract on set of lanadmin stats from another. You might use that to measure statistics over an interval rather than the "from boot" you get with just a single snapshot of the figures.

[slydmin]

I appreciate everyone's input. I will be doing some tests as suggested in your posts and report back.

Thanks,
-S

[slydmin]

Patrick,
I was actually trying to transfer about 650GBytes of data from this server to distant destination. The MRTG graphed the switch at 20Mbps, like something was limiting the transfer. BTW, it took about 6 odd days to complete the transfer!

I have linux boxes that accomplish this task in 15 hours, with some TCP tuning. (I am not trying to start a flame war, just observing). I would like to see my hpux boxes do the same or very close to that.

I should also metion I was using rcp, which is a little better than scp/ssh.

This is a leased line that we a pumping data through, 200mbps, of which only about 20 mbps is being seen as utilized by the switch connected to this server.

Thanks,
-S

[Bill Hassell]

rcp is not a great protocol for large data transfers and as Rick pointed out, the packets are extremely small. And since this is a WAN, you might have very large line response times. Try a ping to the remote system for a while. If the response time are dozens to hundreds of ms, then rcp would be the worst protocol to use as you need the largest packet size. ftp will automatically adapt to the line characteristics as well as allow overlapping acknowledgements, particularly useful for lines that have long turnaround times.

[rick jones]

If it is a single file being transferred, rcp shouldn't be appreciably worse than ftp for the transfer, modulo socket buffer settings.

If this is a long distance leased line, then you almost certainly need a socket buffer/window size larger than HP-UX defaults.

One limit to the performance of a TCP connection for bulk transfer is:

Tput <= Window/RoundTripTime

So, when your 200Mbit/s leased line is at least mildly loaded, run a ping and note the time. Then using the formula above, plug-in 200Mbit/s for Tput, the RTT you get from ping for RoundTripTime and then solve for the _minimum_ required window. Add some slop to that.

Then you will need to examin the manpages for rcp and/or ftp and ftpd on your UX system(s) to see how one gets them using larger socket buffers and so larger window sizes. I am pretty sure they don't take the defaults (although IMO passing "0" to the option to set the socket buffer size should, and if not folks should file ER's against those apps...).

ssh/scp is another kettle of fish since it has an application-layer flow control mechanism (or at least something that ends-up acting like that) and you need some later ssh/scp patches from their "upstream" sources which may or may not be in an HP-UX patch - check with the RC on that one. It also introduces the issue of crypto performance, which takes us back to issues of CPU util.