Tangosol UDP tuning

[Mr Neil Wilson]

Hi all,

Has anybody done any UDP/Socket tuning on HP-UX 11.23 for using the Tangosol Coherence caching product? They have a datagram test tool which tests the network you are using to see the sort of throughput you can achieve by firing UDP packets between 2 servers (1 being a sender & the other a receiver).
However, when we try this on a dedicated gigabit network we are not obtaining the throughput they say we should be able to achieve on a Gigabit network (Around 100Mbps).We are getting about 70Mbps before we start getting constant socket overflows as seen by netstat -p udp command.

The servers are 2 x CPU's rx4640 (during the test we are using about 80% of a single cpu) & the interfaces cards are set to 1000/FD (autoneg) on both servers & switch interfaces.

We have changed the socket_buf_max & socket_udp_rcvbuf_default to 2MB (2097152) as recommended but we are still not getting above the 70Mps.

Has anybody any other recommendations to get more throughput? BTW, the switch is a single Cisco WS-C4948 wire-speed Gigabit Switch.

[Bill Hassell]

UDP packets (such as ping) are usually quite small so the overhead in both source and destination may come into play with high packets rates. If the packets are large (several Kbytes) then change the interface to use jumbo frames using lanadmin -M 9000. Assuminh that lan0 is the interface card:

lanadmin -M 9000 0

and check the new value:

netstat -in

This assumes that the other side also supports jumbo frames. This setting can be made permanent by editing the appropriate hpigelanconf or hpgelanconf file in /etc/rc.config.d

Jumbo frames will mean a 6x reduction in packet count for the same data as long as the messages are more than 1500 bytes long.

[Mr Neil Wilson]

Hi Bill - Apologises for the late reply due to Xmas break & sickness!

The test tool is sending packets of 1468 bytes in size so they make full use of the 1500 MTU & we can specify the rate at which to transmit these messages on, specified in megabytes. This is where we get to specifiying 70Mbps before seeing missing packets in the test tool & socket overflows using netstat -p udp. Anything above this rate & we get a severe downturn in throughput & success rates.

What i've noticed is that the sending server has no socket overflows, it is just the receiving server which is reporting them.

[rick jones]

Given that UDP has no flow-control one's options are a triffle limited. However, based on my experience with netperf's UDP_STREAM test, you _may_ be able to address the socket buffer overflows on the receiver by increasing the SO_RCVBUF size. However, that will only help if the periods of overflow are simply spikes and not a sustained situation.

When you made the ndd changes, if you rebooted without putting them into nddconf they were lost. Also, if the Tangosol SW made any explicit setsockopt calls your udp settings may have been overridden.

On the sending side, there wouldn't by definition be socket buffer overflows, but there could be transmit buffer overflows for which you would use lanadmin -g mibstats to check.

You may want to try some netperf UDP_STREAM stuff just to get a second opinion. With it you can also try binding the receiver to a CPU other than the one taking interrupts from the NIC.

A final nit - Mbps would be mega _bits_ per second, not megabytes, and if you are indeed using gigabit ethernet, its max would be ~100MBps or 9XX Mbps.

[Mr Neil Wilson]

I increased the socket_udp_rcvbuf_default parameter just to see if it made a difference but it didn't. The test tool sends the load in a number of bursts per second according to the realtime output but i reckon the outcome of this is a sustained load.

Regarding the sending server, the lanadmin command showed no errors/collisions or deferred transamissions, so this looks all ok.

I'll have to look at some other test tools, like netperf, to verify the throughput rather than just believe the Tangosol tool. I've seen in other forums posts that this is a recommended one to use. Is there a 'ready to use' executeable for HP-UX rather than having to complie one?

Thanks,
Neil.

[rick jones]

While it is not an HP product, the HP support types may have pre-compiled netperf binaries. I don't distribute any via netperf.org myself though.

It would be good to know the specifics of the burst size and inter-burst time being used by their test tool.

In netperf-speak, that sounds like the ./configure --enable-burst compilation path, which will enable the -w and -b options for inter burst interval and burst size (number of sends). That can lead to some nasty burstiness - at the very least one would want the SO_RCVBUF size on the receiver to be, oh, 2X the product of burst size and send size just to be paranoid about it.

Since not all platforms have nice granulatiry on the interval timer, netperf includes a "sit and spin" option to allow much finer interburst intervals and so lower burst sizes for a given bitrate. This comes at the expense of much higher CPU utilization on the sending side. One adds a --enable-spin to the configure command line before compiling.

The netperf manual can be seen at:

http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html

[Mr Neil Wilson]

Thanks Rick,

We've downloaded netperf & run the configure & install scripts & it works successfully.

The output from the UDP_STREAM test shows that we are indeed getting the 100MBps throughput on the gigabit network (the 114.60 figure on the receiver) if i understand the output correctly:

[root@brsux064] # ./netperf -t UDP_STREAM -H 10.201.25.65 -f M
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.201.25.65 (10.201.25.65) port 0 AF_INET
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # MBytes/sec

65535 65507 10.01 79893 0 498.39
2097152 10.01 18370 114.60

What i don't understand is why we are getting messages lost?

The only info i can gather regarding the burst details is from the Tangosol sender output when the test is running - see attchement.

There are more details on their site regarding the test utility & the packet sizes in the test for reference:

http://wiki.tangosol.com/display/COH33UG/Datagram+Test

[Mr Neil Wilson]

Thanks Rick,

We've downloaded netperf & run the configure & install scripts & it works successfully.

The output from the UDP_STREAM test shows that we are indeed getting the 100MBps throughput on the gigabit network (the 114.60 figure on the receiver) if i understand the output correctly:

[root@brsux064] # ./netperf -t UDP_STREAM -H 10.201.25.65 -f M
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.201.25.65 (10.201.25.65) port 0 AF_INET
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # MBytes/sec

65535 65507 10.01 79893 0 498.39
2097152 10.01 18370 114.60

What i don't understand is why we are getting messages lost?

The only info i can gather regarding the burst details is from the Tangosol sender output when the test is running - see attchement.

There are more details on their site regarding the test utility & the packet sizes in the test for reference:

http://wiki.tangosol.com/display/COH33UG/Datagram+Test

[rick jones]

Well, in the case of your netperf run(s) the lost traffic is from the sender being much faster than the network.

You might compare the send sizes netperf is making by default with those made by the tangosol tool and then use the test-specific -m option to cause netperf to make sends of the same size.

[Mr Neil Wilson]

I'll have a play around with netperf to send the same size packets as the tangosol tool & try the netperf 'burst' options you mentioned to try and replicate the test tool behaviour with similar amounts of traffic & bursts. I'll post back any issues, if any, but thanks for all the info provided so far.

Neil.

[H Niska]

Just a small comment to Bill, ping isnt UDP it is ICMP, even if they both are fire and forget packets they are not the same.