Slow file copy speed in one direction on DL380 G5

[NJK-Work]

Hi,

We have about 15 DL380 G5 servers all acting in the same way. Copying files from it is very slow. However, if I am on the server and copy files to some other device, it runs at normal (fast) speed. In other words, a copy pull is slow, a copy push is fast. So if I am logged in to my Windows XP PC and copy files from one of the servers to my local drive it is slow. If I RDP to the server and copy those same files (from the server) to my PC, it is very quick.

Our environment:
DL380 G5 - all firmware and HP drivers at latest version.
Windows 2003 R2 SP2 (standard and enterprise)
NICs are configured for 1000 GB/s Full duplex hardcoded on both the server and the switches
Switches are Cisco 6509 chassis with WS-X6748-GE-TX blades (I am not a networking guy - this is the info they provided me).

We have older DL380 G4 servers running at GB Ethernet on the same switches with no problems.

Anyone have any ideas?
Thanks
Nelson

[James ~ Happy Dude]

Hello Nelson,

1) Run Firmware CD from
http://h18023.www1.hp.com/support/files/server/us/download/27783.html

2) Is there any NIC team ?
if so, dissolve NIC, update individual NIC firmware, Update NIC Drivers, PAIR NIC using Latest NIC Utility.

3) Apply Latest PSP:
http://h18023.www1.hp.com/support/files/server/us/locate/69_6213.html#0

4) In your PC, go to Device manager, select the network Card & goto Properties; then Advance TAB; Set "Speed & Duplex" to be AUTO;;; If its already AUTO, Force it to 100MBps & check for any difference.

Regards,
James.

[James Kennedy_5]

Generally for 1000 MB you should be doing Auto on the Gigabit switch and NICs (as apposed to forcing 100/FULL on the lower speed NICs) .. but forcing should be ok too.

Anyways, I've been having lots of Network issues with G5 servers and PSP 7.90. Try disableing TOE on the team and both NICs. See if that helps.

Also, since you're running SP2, you may want to install this patch from microsoft: http://support.microsoft.com/kb/936594

[NJK-Work]

Thanks for the responses.

I am already at the latest firmware.
I am already at Support Pack 7.9
The problem also exists with Support pack 7.8
I already have the correct speed and duplex configure on my pc, my server and all ports involved. But it is just not my PC anyways - it is any device I try to copy files from the problematic server including other (good)servers.

I will try AUTO/AUTO and see if that helps
I will investigate the MS article.

Thanks for the tips.
Nelson

[Hense, Klaus]

Hallo, try connect your server and your PC or other server with a cross Link cabel and then test the performance!
test with break the Teaming.
Good luck..

[NJK-Work]

I had our Network guy switch the port to 100 FULL and I hardcoded the NIC to 100 FULL and now it works fine. So it is something with running those NICs on the DL380 G5 at GB that breaks something.

Nelson

[Igor Karasik · Accepted solution]

G5 has newest NIC, therefore
try to disable RSS and TCPChimney:

http://support.microsoft.com/default.aspx?scid=kb;EN-US;927695
http://msexchangeteam.com/archive/2007/07/18/446400.aspx

[NJK-Work]

Disabling TCPChimney fixed the problem...thanks!

Netsh int ip set chimney DISABLED


But what does that break, or should I say, what does it limit by disabling it. Am I loosing some other features by fixing another?

Thanks
Nelson

[Igor Karasik]

>>But what does that break, or should I say, what does it limit by disabling it. Am I loosing some other features by fixing another?


TCP Chimney offloads networking tasks from CPU, leaving more power for applications - at least in theory :-)

AFAIK, it is relatively new feature, exists only on newest NIC's and has many problems (at least in Microsoft implementation)

Also, according this HP article - TCP Chimney will not work if HP virus throttle feature installed (part of PSP):
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=398225&prodTypeId=15351&prodSeriesId=398225&objectID=c00810667

[NJK-Work]

Thanks again!

Nelson

[Tom E Reynolds]

I also had to disable TCPChimney to fix my upload problems through a VPN to my new HP DL380G5 servers with the onboard HP NC373i NIC.

Netsh int ip set chimney DISABLED

in the Biz forum see:

http://forums11.itrc.hp.com/service/forums/bizsupport/questionanswer.do;HP-FORUMS-S-WPA-IDX=Hv8JY1vnT51pNhfT6YS10pHP1QTy02pnJHJpfpgTvJ4V0B9znlJM!-1885766594!-124372549?threadId=1153566&admit=109447626+1197653161555+28353475

[PinnacleCS]

We had this same issue. We found the best way to do this is to go into the HP Network Config utility, properties of the NIC or team, settings tab and disable TCP Offload. This fixed all of our slowness issues.

[Jimbo!]

For the benefit of the search, this can be caused by auto-negotiate at one end and fixed full duplex at the other.

What happens is that since one end is fixed, auto-negotiate fails and the port or NIC immediately drops to auto-sense. Again, since the other end is fixed, this succeeds so one end will be 100/full (or 1000/full) and the other 100/half (or 1000/half).

Say the switch is 100/full and the server 100/half. Client is correctly configured and 100/full and both ends.

Files are transferred with a sliding window, allowing multiple data packets to be acknowledge with a single ACK.

Consider copying to the server:

Client sends a load of packets, server starts discarding them once it's window is full and awaits a gap for transmission. The client now stops because of lack of ACKs, the server can then send the ACK (with some packets lost), and the process can continue. The overall throughput seems reasonable.

Consider copying a file from the server:

Server sends a load of packets, the very first ACK, soon after transmission starts, from the client is seen as a collision and the server stops sending. Client assumes packet is lost and re-sends ACK after maybe one second delay. Process repeats, resulting in only a few packets each second, hence very slow throughput.

Hope that helps!

[NJK-Work]

Well, the problem is back on a new DL380 G5 server I just built. Using the this command does not fix the problem anymore:

Netsh int ip set chimney DISABLED

I tried setting the server to AUTO and the switch port to AUTO, and I also tried setting the server to 1GB FULL and the switch port to 1GB FULL. I get the same results.

Pulling files to the server is fine. Pushing files to another host (from that server) is slow. We discovered the problem because backups (NetBackup) were super slow. And backups are a push - files going from the server to the backup host, which results in the slow copy. But I can simulate the problem by just using a simply file copy (so NetBackup is not the problem).

Arrrg!
Any other suggestions?
Thanks
Nelson

[Tech-To-Tech]

Here's what I've done so far as a temp soultion until HP releas a newer driver for the NIC.
1) On the current NIC card disabled the following:
- Large Send and Checksum Offload
- RSS
2) Registry:
** Disable RSS:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableRSS and setting it to 0
** DisabledTaskOffload: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\DisableTaskOffload and set it to 1

the driver I have is version 3.4.10.0 dated 5/25/07 so I didn't need to update that. Also, you don't want to disable the TCP Chimney so once HP release a new driver then you can take advantage of the new features offered by SP2. Plus, Windows Update might reset those registry keys.

[KarloChacon]

hi

Tech-To-Tech I want to ask something when you say disable:
- Large Send and Checksum Offload
- RSS

you did that in the NIC Cards

and also did these changes in register

** Disable RSS:
HKEY_LOCAL_MACHINE\......

** DisabledTaskOffload: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet....

I mean you did both changes or you disabled both using register

sorry for my silly question but this topic is been a long issue SP2 - RSS- TOE - chimney - ..

regards

[Tech-To-Tech]

Yes I disabled them on both places and I'm glad you asked since I forgot to say that you have to reboot your server(s) after you make these changes on the registry

[KarloChacon]

thanks a lot for clarify that

that is because I've seen that people say I disable RSS but that was only on the NIC cards settings not using register as well as other NIC properties only in NIC cards but not in registry

regards

[NJK-Work]

Thanks for the tips.

I made the changes you suggested verbatum and verified my NIC driver is the same as your; it is.

However, I still get the same problem.

The one difference between this server and the others that I fixed by disabling the chimney stuff is that this server is behind a Firewall. Is there anything special I need to have my FW admins set on thier end to make this work?

As a test, I tried the copy process between others servers also behind the FW. I actually have 4 servers, as part of an overall project, that are behind the same FW and configured exactly. I see the same problem from each when going to a device outside the FW. But between them, the copy is fine. So I guess I just answered my own question...it has to be the FW. But what?

Any suggestions?
Nelson

[NJK-Work]

I think I am making progess :)

I undid all the settings you suggested (so the server looks the same as day one with the normal settings). File copies between devices behind the firewall still work fine. So I think this has nothing to do with TOE and the Chimney stuff like the other servers had. It just happens to have the same symptoms.

That being said, I think I will take this up with my FW guys and see what's up.

Unless you guys have any more suggestions...
Thanks for all the help.
Nelson

[KarloChacon]

hi

let me link some MS articles that maybe can help to get a whole idea about these issues


Regarding RSS and disable it - I added here because it refers Firewall

http://support.microsoft.com/kb/927695/

this other one has different scenarios that I think you can try just to check one of them apply for those servers

http://support.microsoft.com/kb/936594

and there is another article that refers something you already did

"""Windows Server 2003 Scalable Networking Pack open issues Third-party firewall vendors cannot connect to the Windows networking stack by using undocumented APIs when TCP Chimney offload is enabled.

TCP/IP statistics that are obtained from offload-capable hardware will be double counted if the following conditions are true:
â ¢The TCP Chimney offload functionality is turned off by using the EnableTCPChimney registry entry.
â ¢The TCP Chimney offload functionality is turned off by using the Netsh int ip set chimney DISABLED command
"""

this is a long one
http://support.microsoft.com/kb/912222/


I think I'm going to print those out

[Tech-To-Tech]

Nelson, did you have any issue with the FW prior applying SP2? Also, make sure that you have the server in a Gigabit port on the switch as well as the FW. BTW, what kind of FW do you have?

[NJK-Work]

Karlo - thanks for the suggestions. I had already tried all those steps including installing the hotfix listed in the one MS KB article.

Tech-to-Tech - the servers are built from an image with SP2 already installed. Since we have a volume licensing agreement with MS, we have access to the 2003 CDs with SP2 slipstreamed into the OS - I don't even think I have the option to uninstall SP2 when it is slipstreamed.

I do not know what type of FW we have and I am not sure our security team would appreciate me broadcasting that info anyways - sorry :)

Thanks
Nelson

[NJK-Work]

DING DING DING - we have a solution!

External interface on the FW was misconfigured for 100/Half instead of 100/Full. Not sure why this would only affect traffic in one direction and not the other, but it did.

That's 8 hours of server troubleshooting I will never get back.

[NJK-Work]

First problem
Run this on the server:
Netsh int ip set chimney DISABLED


Second problem
Never assume coworkers configured their part of the equation properly AND never believe them the first two times you ASK them if everything is configured properly; ask them a THIRD time :)