topic Re: How to inprove NIC performance? in Operating System - Linux

How to inprove NIC performance?

Vitaly Karasik_1 — Wed, 05 Dec 2007 08:50:05 GMT

I'm running RHEL/CentOS 3 (2.4.21-47.ELsmp) with Broadcom NIC and BASP bonding (bcm5700 v8.1.55) When network traffic is ~12MB/sec (== ~120K packets/sec) CPU system load is growing up to 50-60%%.
Yes, I know, that small packets (~100 bytes/packet) are not optimal for network throughput, but in the short-middle term I cannot re-write our application.

Are there any network driver / kernel parameters which can help me to increase network throughput?
Replacing bcm5700 to tg3? Upgrading to RHEL 4 or 5 with kernel 2.6?
Other ideas?

Thanks,
Vitaly

Re: How to inprove NIC performance?

Steven E. Protter — Wed, 05 Dec 2007 09:12:58 GMT

Shalom Vitaly,

Among the options presented, I've seen the biggest improvement with the 2.6 kernel. They totally re-wrote how they handled TCP and its more configurable.

I don't consider Centos5.1 stable enough for important systems. Going centos 4.5 with all patches before update 6 is stable and I'm very pleased with the performance bump I get when migrating Red Hat and Centos 3 systems.

SEP

Re: How to inprove NIC performance?

Vitaly Karasik_1 — Wed, 05 Dec 2007 09:36:18 GMT

Thank you!
In fact, our application doesn't use TCP/UDP, but just raw sockets. So tons of TCP params aren't relevant for us.

Re: How to inprove NIC performance?

Rob Leadbeater — Wed, 05 Dec 2007 14:05:38 GMT

Hi,

Maybe you could tweak the MTU (Maximum Transmission Unit) size to be in line with the size of the data that your application is sending...

I think you do this by setting the mtu option in modules.conf.

Hope this helps,

Regards,

Rob

Re: How to inprove NIC performance?

Vitaly Karasik_1 — Wed, 05 Dec 2007 14:40:24 GMT

Rob, thank you!
But I don't sure that tuning MTU can help in my case: my packets (~100bytes) is smaller than MTU (1500 bytes).
I don't believe that decreasing MTU make sense.

Re: How to inprove NIC performance?

Heironimus — Thu, 06 Dec 2007 14:57:37 GMT

I have no idea if it will help, but one thing to try is switching to the generic Linux bonding driver instead of BASP. I would also try tg3 instead of bcm5700. Broadcom no longer supports bcm5700 (and, by extension, BASP) and says to use tg3, and HP's certified drivers for Proliants have also moved from bcm5700 to tg3.

I've slowly been switching Proliants from bcm5700 to tg3 without any problems. The only real change is that tg3 uses ethtool instead of module options for forcing speed and duplex. I haven't quite decided if I want to use HP's tg3 or Red Hat's tg3, both seem to work equally well.

Re: How to inprove NIC performance?

Vitaly Karasik_1 — Thu, 06 Dec 2007 15:01:58 GMT

Can you point me to some URL about this?
Does it mean that bcm5700 unsupported for kernels 2.4 (RHEL3) too?

Re: How to inprove NIC performance?

Heironimus — Thu, 06 Dec 2007 16:21:40 GMT

http://www.broadcom.com/support/ethernet_nic/faq_drivers.php#97

"For the NetXtreme and NetLink product lines, the tg3 driver is now the only Linux driver that Broadcom supports. Accordingly, Broadcom has discontinued support for the bcm5700 driver and no longer provides updates."

Broadcom's FAQ says that kernels newer than 2.4.24 include the tg3 driver, but RHEL/CentOS 3 uses a 2.4.21 derivative that also has a fully-functional tg3 driver (at least in the recent updates, I'm not sure about the original release). HP's certified drivers for RHEL3 on Proliant are tg3 now, and their NIC firmware update utility wants tg3.

I haven't had any problems with the bcm5700 driver - I still have lots of machines using it and I haven't felt any need to make the switch on existing systems. However, I can't ignore it when both Broadcom and HP say that tg3 is the way to go.