Questions on HP-UX Networking for the Gurus, Gurulettes, etc!

[Alzhy]

1.) On a GigE connected HP-UX system, would a packet rate of 1500/sec. be considered light, medium or heavy?

2.) What are the benefits of Trunking (APA)?. Would it mean higher packets per second capability or just higher "bandwidth" - bytes/second?

3.) For high-bandwidth/high-packets-per-second interserver connectvity requirements - what kernel tunables should be closely watched and possibly tweaked?

Thanks.

[Steven E. Protter]

Shalom,

1) Depends on your network hardware and configuration. Count on less than half theoretical maximum in real life.

2) APA with two identical cards and a correctly configured switch will get you higher bandwidth. I don't see how number of packets factors in. It depends on the size of the packets.

3) Hardly worth playing with. If you still want to take a look at the man page for ndd.

In general, in a proper networking environment there are better things to do than play with this. APA provides tangible benefits if the switch is configured to handle a single IP address spanning two switch ports.

SEP

[Sundar_7]

Nelson,

Benefits of trunking - You get a much bigger pipe to push the data through. As far as my understanding goes, a packet is nothing but a specified number of bytes. So, a bigger pipe would effectively mean more bytes/second (thus inturn more packets per second).

To take advantage of the bigger pipe, you will need to use the TCP parameters. Kernel parameters are not of much help.

There are of the parameters that define the high and low water marks.

tcp_xmit_hiwater_def (read and write)
tcp_xmit_lowater_def (read and write)
tcp_recv_hiwater_def (read and write)
tcp_xmit_hiwater_lfp (read and write)
tcp_xmit_lowater_lfp (read and write)
tcp_recv_hiwater_lfp (read and write)
tcp_xmit_hiwater_lnp (read and write)
tcp_xmit_lowater_lnp (read and write)
tcp_recv_hiwater_lnp (read and write)

You could also increase the TCP window size.

You can use ndd to get/set the value of these parameters. /etc/rc.config.d/nddconf can be used to make the setting permanent.

Sundar.

[Matti_Kurkela]

1.) When judging that, it would be important to know at least the average packet size. With all minimum-size packets, this would be a light load. If you're using jumbo frames and transferring mostly maximum-size packets, this would probably be a medium-to-heavy load.

2.) More bandwidth, but latency stays the same. Trunking won't help if you have to push just one packet to one destination; but if there are lots of packets to be moved, it's useful.

3.) Before tweaking, I think you must decide on several points.
- Is latency more important than throughput, or vice versa?
- Is it desirable to keep retrying to transfer each packet without error, or can you afford to lose some packets when there is congestion or errors on the line?
- Is it important that the packets arrive in the correct order, or at least approximately so?

Consider real-time streaming video: you'll want the network latency to stay stable (and preferably small), so that the stream of packets stays mostly in order. That helps to minimize the amount of buffering and rearranging required at the receiving end, which keeps the total latency (experienced by the user) small. A package that arrives after a video frame it belongs to is already displayed is useless: better to stop retrying early and just plunge ahead.

MK

[Alzhy]

Amigos, this inquiry is a result of our plans to enforce physical server separation of our Databases (Oracle) from the applications (home-grown C, Cobol, lots of Java, etc..) in a 3-tier (Browser, Weblogic, UNIX-DB/App) environment.

This so we can tune the DB Server environment better and so the DB Engine/Instance(s) are not swamped when the non-Oracle processes ramp up.

The environment is Batch by night and hybrid Online-DSS/Batc by day. Network packet sizes vary heavily (any way to find out? exactly) between the current mixed APP-DB server and our WebLogic middle-tiers. Range during Online ours is 800 to 5000 packets/second.

In terms of throughput -- it's anywhere between a few KB/sec to no more than 8MBytes/sec..

On the UNIX DB Backend.. when we separate those Java, C, Cobol processes (which all use SQL connect and JDBC, etc..) - we're facing an unknown in terms of performance (network-wise) if we proceeed with the total separation. We'd like to end up with our UNIX backend as 2 pieces -- a compute farm (exclusively running those C, Cobol, Java processes) and a DB Farm -- server(s) that will purely server Databases. So we are thining of preparing for such an eventuality by using APA trunks - if indeed there will be a huge pipe requirement (packet/sec or bandwidth between the separated environments.

[Mel Burslan]

Nelson,

This is not from the hpux side of my shop but rather IBM/AIX side. We have a comparable setup for one of our mission critical applications. Our setup is, a pair of DB servers (active/passive cluster conf) connected to G-bit switches and two ports trunked to make up a bigger pipe and another two ports trunked as a passive failover on each of the db nodes. Application layer in my case is a little different from you as we do not have any in-house hodge-podge of apps but mainly a weblogic and some java apps doing heavy SQL queries to the Oracle back-end. Each app server talks to the db layer over a single g-bit port. And so far we didi not observe any network meltdowns.

Having said that, 800-5000 packets per second range is drastically wide. I am sure, if you do just the g-bit connectivity between db and app layers properly, you should not need any APA'd ports on the lower end of the spectrum you are talking about. Should the traffic go higher, it mainly depends on the nature of the data transfer. Before the APA option, I'd consider investigating jumbo frames if your packet payloads are large. But if it is just few bytes of data in a packet, jumbo packets will not be an option and you may have to seriously consider introducing APA to your environment.

This is basically an adaptive control case. Apply load, watch network load, tweak ! Then, lather-rinse-repeat cycle. As a wise man said, "life is what happens when you have other plans". This situation in my opinion, is a perfect example. While your are investigating APA, you might be ok just upgrading to a g-bit network. Sorry about the vague answer but this is all I can say.

[rick jones]

1500 packets per second isn't all that high.

APA increases the aggregate performance available under an IP address. However any _single_ flow will go no faster than what can be carried on a single link.

There aren't many kernel tunables to tweak - you might use ndd to tweak some stack settings such as default socket/window size.

For high packet per second situations, you want to keep an eye on the per-CPU utilization of the CPUs.

[Alzhy]

Mel,Thanks.. we're actually already running GigE in APA Failover mode and plan to do trunking if there is a need. There is apprehension that once we separate apps from the DB, the GigE pipe may not be enough, as it will be an additional network burden on the DB servers which will also be hit hard with network connections from the Weblogic machines.

Rick, any tip on how to actually see if how large our packets are? Also, will implementing TCP.Offload (Gigether-Enh) help?

[rick jones]

If you want to get the average packet size over an interval, take snapshots of either netstat -s -p tcp or lanadmin -g mibstats or both, and then run them through ftp://ftp.cup.hp.com/dist/networking/tools/beforeafter.c

Then you divide the byte/octet counts by the segment/frame count and you have the average size.

If you want to get a distribution of the sizes, you should take a tcpdump trace and then run it through some sort of post-processing.

I've never played with it, but I suppose that there may be some ways to abuse ipfilter to get something here too. Generally if I say "ipfilter" an ipfilter guru pops-out of the woodwork, maybe that will happen here too :)

[Ben Dehner]

It depends on your packet size, but I'd consider 1500/sec packets to be a light load for GigE. Unless you're using jumbo packets, typical ethernet MTU size is about 1500 bytes, so a good guestimate on average packet size is 1000 bytes. (This is somewhat high when you consider things like ARP, DNS, syslog, spanning tree, etc, which are all tiny packets.) So you're tralking about a sustained transfer rate of about 11 Mb/sec -- the 1 Gb interface is still pretty bored.

For trunking APA, there is a difference between bandwidth and bandspeed. Others have pointed out that your transfer rate to a single remote destination will not exceed the 1Gb/s speed of the underlying interfaces. However, multiple remote destinations have a better chance of getting the full 1 Gb/s transfer rate.

I'm not sure about the kernel paramters. However, apparently you are seperating the database server and application server. If bandwidth is a concern, you might consider a dedicated interface on both systems solely for this connection. Put it on its own seperate non-routable VLAN where the other network stuff won't interfer with it. If the sustained bandwidth requirement starts to top 100 Mb/sec, you might want to reconsider moving some of these applications seperate from the database server.

[Bill Hassell]

Packets per second is not a useful metric by itself. A 100 byte packet carries tremendous overhead compared to a 1000 byte or 9000 byte packet. Some of the overhead is all the extra bytes around each packet and some of it is the system overhead (driver, application,). So as mentioned before, 1500/sec could be a very heavy for a 200 Mhz computer, nothing for a 1200 Mhz box.

Also as mentioned, APA is the same as opening new lanes on a freeway -- with an enforced speed limit. A single stream of cars travels exactly the same speed (and throughput) with one lane or with 4 lanes. So a single threaded application will zero improvement with APA no matter how hard it tries to push more packets -- it's a single stream. Change the system to use multiple streams (threads or additional apps) then APA makes sense and will show significant improvement, ignoring the driver and system overhead on both sides. Database applications may show some improvement with jumbo frames -- it all depends on the the actual data size. Large ftp transfers will benefit greatly for example.

So benefits depend on enough fast CPUs, the latest OS (11.23 or 11.31) and driver patches, and plenty of independent processes on all the computers using the APA connection.