performance problems with TOE

[schmitt jo]

Hi,

when i active TCP offloading in my server proliant DL 360 g5 (netcard HP NC37
i driver version 3.0.5.0), the performance are very slow with tcp services (http, https) in windows 2003 server.

Much appreciated if you could help ..

Thanks

[Morten Dalgaard]

I'm having issues with TOE as well, both with 3.0.5.0, and older versions. Network activity drops to zero about once a minute for a few seconds, then resumes again.
Seems like a bad driver, or the hardware itself, i can only advise turning TOE engine off.

[rick jones]

While I cannot say this is happening, I will point-out that in the past (early 1990s) there were a host of "NFS offload cards" (think TOE for NFS over UDP) which were there to offload all the 'heavy' NFS protocol processing. Very soon thereafter the host CPUs had a spike in performance, and the CPUs down in the offloading I/O cards could not keep-up. The pendulum swung back in favor of the host CPU(s).

Given the somewhat limited power budgets in an I/O card (IIRC O(25Watts)) and some of the recent spikes in Intel host CPU performance, knuth only knows where the tipping point lies these days.

It might be interesting to try some simple benchmarking with something like netperf. The natural inclination is to just run a bulk transfer test (eg TCP_STREAM, the netperf default) but in this case I think I would suggest a TCP_RR test comparing TOE on to off, and also some aggregate TCP_RR testing - either via the configure --enable-burst or/and with multiple netperfs running. Bulk transfers (particularly when/if coupled with large MTUs) can hide a multitude of performance ills. http://www.netperf.org/

[MUGEN]

I have issue about TOE on my BL460c with Windows 2003 SP2.

After I installed SP2, network performance become extremely slow. I then unchecked to disable TOE, RSS, Chimney in the Network Utility. Also I modified the registry with values:

1. DisableLargeSendOffload = 1
2. DisableTaskOffload = 1
3. EnableRSS = 0
4. EnableTCPA = 0
5. EnableTCPChimney = 0

Afterward the network performance got a bit fast but still slow.

This happens when the traffic is thur a WAN Link. You may say that the problem is on the Router/Switch/WAN Link but there is no problem to the IBM servers with same OS.

Here is the reply from Microsoft Support:
I reviewed the netmon traces. The W2k3 SP2 computer still delayed 200~400ms to retransmit some packets. Based on our test, I suspected the problem is on the NIC driver or firmware side.

[rick jones]

A common problem in a WAN environment is insufficient TCP window to keep the pipe full - perhaps those 200-400 ms delays were waiting for ACKs to get back from the remote?