topic Poor network performance with ML 350's in ProLiant Servers (ML,DL,SL)

Poor network performance with ML 350's

David B Walsh — Tue, 23 Oct 2007 05:05:12 GMT

Hi,

I have two HP ML350 servers, as follows:

"S1" is an ML350 G4 with 3GHz CPU, 2GB RAM, Ultra320 15K disks in RAID 5 and built in gigabit NIC running SBS 2003 Premium with SP2.

"S2" is a brand new ML350 G5 with quad core CPU, 2GB RAM, SAS 10K disks in RAID 5 and built in gigabit NIC running Windows Server 2003 Standard R2 SP2.

I have been doing some testing on the network and the performance is nowhere near what I would hope for and expect it to be. I used the following configurations and tests:

CONFIG1: Both servers connected to unbranded 100Mbps switch at 100Mbps

CONFIG2: Both servers connected together with a cross-over cable at 1Gbps

CONFIG3: Both servers connected to a brand new Netgear 1Gbps switch at 1Gbps

TEST1: From the console of S1, use Windows Explorer to copy an i386 folder from S2 to S1.

TEST2: From the console of S2, use Windows Explorer to copy an i386 folder from S1 to S2.

CONFIG1 TEST1 throughput = 26 Mbps
CONFIG1 TEST2 throughput = 46 Mbps

CONFIG2 TEST1 throughput = 29 Mbps
CONFIG2 TEST2 throughput = 67 Mbps

CONFIG3 TEST1 throughput = 42Mbps
CONFIG3 TEST2 throughput = 72Mbps

The tests were performed after cold boots on both servers, with no other devices connected to the network. TCP offload engine is disabled on S2. Both NIC's and switch ports were set to "auto" for speed and duplex. With the gigabit switch and the crossover cable, both NIC's reported that they were running at 1Gbps.

I can live with the performance on the 100Mbps switch, but I can't understand why increasing the speed of the network to 1Gbps (a tenfold theoretical increase) results in a miserable performance increase.

Can anyone suggest where to start troubleshooting this?

Thanks,
Dave.

Re: Poor network performance with ML 350's

James ~ Happy Dude — Tue, 23 Oct 2007 13:40:00 GMT

Hello David,

Many are Baffled by this; TOE, RSS & NetDMA seems to do more harm & benefit.
Refer some of these:
http://support.microsoft.com/kb/912222
http://support.microsoft.com/kb/936594 (Important hotfix which in NOT included in SP2)

& Very interestingly :
http://www.microsoft.com/technet/community/columns/cableguy/cg0606.mspx

Which Read:
*To ensure that TCP Chimney Offload will not reduce the capabilities of existing and future Microsoft WindowsÂ® network stacks, TCP Chimney Offload will not offload a connection if the network adapter does not support a needed processing capability, such as Internet Protocol security (IPsec) cryptographic processing.

*If a network adapter supports Receive-side Scaling, the Scalable Networking Pack uses this capability across all TCP connections, including connections that are offloaded through TCP Chimney Offload.(I am not sure, how RSS reacts when TOE is MANUALLY Disabled)

*The Scalable Networking Pack invokes NetDMA when it detects supporting hardware. If the Scalable Networking Pack detects that the hardware can support both NetDMA and TCP Chimney Offload, NetDMA is disabled and TCP Chimney Offload remains enabled.(Same here)

Regards.

Re: Poor network performance with ML 350's

Mi6t0 — Tue, 23 Oct 2007 14:12:25 GMT

A nice test!

So...
First of all please disable also the RSS in the NCU (should be near the TOE).
Could you try copying larger Files (I mean a big .iso) to check how the speed is then.

Re: Poor network performance with ML 350's

KarloChacon — Wed, 24 Oct 2007 01:19:15 GMT

hi

TOE - RSS - TCP Chimney what a combination that is causing a lot of headaches.

- So you already disable TOE good.

- what about RSS
http://support.microsoft.com/default.aspx?scid=kb;EN-US;927695

- disable TCP Chimney

"Netsh int ip set chimney DISABLED"
http://msexchangeteam.com/archive/2007/07/18/446400.aspx

and check this thread
http://urlao.com/toe

regards

Re: Poor network performance with ML 350's

David B Walsh — Wed, 24 Oct 2007 12:04:47 GMT

Thanks for the replies and help so far.

Not sure of TOE settings and the Microsoft Scalable Networking Pack are to blame in this situation, although reading through the links provided this Scalable Networking Pack implementation seems to be an absolute disgrace (heads should roll!!!).

Going back to my original post, S1 is an ML350 G4 so I don't think the NIC in this server has the capability at the hardware level to utilise the TOE and SNP, cough, "enhancements" (there are no mentions of things like RSS or TOE in the HP Network Configuration Utility). Nevertheless, I have done the following on S1:

NETSH INT IP SET CHIMNEY DISABLED

Modified the following registry keys:

EnableTCPChimney=0
EnableTCPA=0
EnableRSS=0

The NIC in S1 is a "NC7761 Gigabit Server Adapter" (33MHz, 32-bit, driver version 7.103.0.0)

S2 is a brand new ML350 G5, so I have done the following:

NETSH INT IP SET CHIMNEY DISABLED

Modified the following registry keys:

EnableTCPChimney=0
EnableTCPA=0
EnableRSS=0

The NIC in S2 is a "NC373i Multifunction Gigabit Server Adapter" (133MHz, 64-bit, driver version 3.4.10.0)

After making these changes I repeated my test on the gigabit switch (the servers are now in production and connected to this switch). There was no improvement.

I also repeated the test using a circa 500MB ISO file instead of an i386 folder. The throughput increased to aroun 200Mbps for this test.

I would like to know the following:

1. How can I be sure that TOE and Microsoft SNP are fully disabled on these servers and as such can be ruled out of the equation.

2. What sort of throughput are other forum users seeing on their networks with similar servers running at 1Gbps?

3. Does between 40Mbps and 70Mbps throughput for copying an i386 folder between these servers using a 1Gbps managed switch seem acceptable? (my gut feeling is no).

4. Does around 200Mbps throughput for copying a 500MB ISO image between these servers using a 1Gbps managed switch seem acceptable? (my gut feeling is still no, but getting much closer to what we can probably expect in practice).

Thanks,
Dave.

Re: Poor network performance with ML 350's

Mi6t0 — Thu, 25 Oct 2007 13:44:40 GMT

Dave, I like networking very much and themes like theese are one of my favourite :)

So... I have to tell that I expected the better speed with the ISO like perhaps most of the users.
A point that we didn't discuss are the cables. Did you try with different cables? How long are yours, what kind of model?

About the TOE - to check if it is disabled, you can open the NCU and see the settings. In fact, when it is disabled, CPU should have a little more load.

I think that in this case 1 Gbit is not so easy to be reached. But a transfer about 300-400 mbit is in my opinion acceptable.

Regards, Mi6t0