ProLiant Servers (ML,DL,SL)
cancel
Showing results for 
Search instead for 
Did you mean: 

Slow file copy speed in one direction on DL380 G5

SOLVED
Go to solution
NJK-Work
Honored Contributor

Slow file copy speed in one direction on DL380 G5

Hi,

We have about 15 DL380 G5 servers all acting in the same way. Copying files from it is very slow. However, if I am on the server and copy files to some other device, it runs at normal (fast) speed. In other words, a copy pull is slow, a copy push is fast. So if I am logged in to my Windows XP PC and copy files from one of the servers to my local drive it is slow. If I RDP to the server and copy those same files (from the server) to my PC, it is very quick.

Our environment:
DL380 G5 - all firmware and HP drivers at latest version.
Windows 2003 R2 SP2 (standard and enterprise)
NICs are configured for 1000 GB/s Full duplex hardcoded on both the server and the switches
Switches are Cisco 6509 chassis with WS-X6748-GE-TX blades (I am not a networking guy - this is the info they provided me).

We have older DL380 G4 servers running at GB Ethernet on the same switches with no problems.

Anyone have any ideas?
Thanks
Nelson
24 REPLIES
James ~ Happy Dude
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

Hello Nelson,

1) Run Firmware CD from
http://h18023.www1.hp.com/support/files/server/us/download/27783.html

2) Is there any NIC team ?
if so, dissolve NIC, update individual NIC firmware, Update NIC Drivers, PAIR NIC using Latest NIC Utility.

3) Apply Latest PSP:
http://h18023.www1.hp.com/support/files/server/us/locate/69_6213.html#0

4) In your PC, go to Device manager, select the network Card & goto Properties; then Advance TAB; Set "Speed & Duplex" to be AUTO;;; If its already AUTO, Force it to 100MBps & check for any difference.

Regards,
James.
James Kennedy_5
Regular Advisor

Re: Slow file copy speed in one direction on DL380 G5

Generally for 1000 MB you should be doing Auto on the Gigabit switch and NICs (as apposed to forcing 100/FULL on the lower speed NICs) .. but forcing should be ok too.

Anyways, I've been having lots of Network issues with G5 servers and PSP 7.90. Try disableing TOE on the team and both NICs. See if that helps.

Also, since you're running SP2, you may want to install this patch from microsoft: http://support.microsoft.com/kb/936594
NJK-Work
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

Thanks for the responses.

I am already at the latest firmware.
I am already at Support Pack 7.9
The problem also exists with Support pack 7.8
I already have the correct speed and duplex configure on my pc, my server and all ports involved. But it is just not my PC anyways - it is any device I try to copy files from the problematic server including other (good)servers.

I will try AUTO/AUTO and see if that helps
I will investigate the MS article.

Thanks for the tips.
Nelson
Hense, Klaus
Valued Contributor

Re: Slow file copy speed in one direction on DL380 G5

Hallo, try connect your server and your PC or other server with a cross Link cabel and then test the performance!
test with break the Teaming.
Good luck..
klhe
NJK-Work
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

I had our Network guy switch the port to 100 FULL and I hardcoded the NIC to 100 FULL and now it works fine. So it is something with running those NICs on the DL380 G5 at GB that breaks something.

Nelson
Igor Karasik
Honored Contributor
Solution

Re: Slow file copy speed in one direction on DL380 G5

NJK-Work
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

Disabling TCPChimney fixed the problem...thanks!

Netsh int ip set chimney DISABLED


But what does that break, or should I say, what does it limit by disabling it. Am I loosing some other features by fixing another?

Thanks
Nelson
Igor Karasik
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

>>But what does that break, or should I say, what does it limit by disabling it. Am I loosing some other features by fixing another?


TCP Chimney offloads networking tasks from CPU, leaving more power for applications - at least in theory :-)

AFAIK, it is relatively new feature, exists only on newest NIC's and has many problems (at least in Microsoft implementation)

Also, according this HP article - TCP Chimney will not work if HP virus throttle feature installed (part of PSP):
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=398225&prodTypeId=15351&prodSeriesId=398225&objectID=c00810667
NJK-Work
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

Thanks again!

Nelson

Re: Slow file copy speed in one direction on DL380 G5

I also had to disable TCPChimney to fix my upload problems through a VPN to my new HP DL380G5 servers with the onboard HP NC373i NIC.

Netsh int ip set chimney DISABLED

in the Biz forum see:

http://forums11.itrc.hp.com/service/forums/bizsupport/questionanswer.do;HP-FORUMS-S-WPA-IDX=Hv8JY1vnT51pNhfT6YS10pHP1QTy02pnJHJpfpgTvJ4V0B9znlJM!-1885766594!-124372549?threadId=1153566&admit=109447626+1197653161555+28353475

PinnacleCS
Occasional Advisor

Re: Slow file copy speed in one direction on DL380 G5

We had this same issue. We found the best way to do this is to go into the HP Network Config utility, properties of the NIC or team, settings tab and disable TCP Offload. This fixed all of our slowness issues.
Jimbo!
Occasional Advisor

Re: Slow file copy speed in one direction on DL380 G5

For the benefit of the search, this can be caused by auto-negotiate at one end and fixed full duplex at the other.

What happens is that since one end is fixed, auto-negotiate fails and the port or NIC immediately drops to auto-sense. Again, since the other end is fixed, this succeeds so one end will be 100/full (or 1000/full) and the other 100/half (or 1000/half).

Say the switch is 100/full and the server 100/half. Client is correctly configured and 100/full and both ends.

Files are transferred with a sliding window, allowing multiple data packets to be acknowledge with a single ACK.

Consider copying to the server:

Client sends a load of packets, server starts discarding them once it's window is full and awaits a gap for transmission. The client now stops because of lack of ACKs, the server can then send the ACK (with some packets lost), and the process can continue. The overall throughput seems reasonable.

Consider copying a file from the server:

Server sends a load of packets, the very first ACK, soon after transmission starts, from the client is seen as a collision and the server stops sending. Client assumes packet is lost and re-sends ACK after maybe one second delay. Process repeats, resulting in only a few packets each second, hence very slow throughput.

Hope that helps!
NJK-Work
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

Well, the problem is back on a new DL380 G5 server I just built. Using the this command does not fix the problem anymore:

Netsh int ip set chimney DISABLED

I tried setting the server to AUTO and the switch port to AUTO, and I also tried setting the server to 1GB FULL and the switch port to 1GB FULL. I get the same results.

Pulling files to the server is fine. Pushing files to another host (from that server) is slow. We discovered the problem because backups (NetBackup) were super slow. And backups are a push - files going from the server to the backup host, which results in the slow copy. But I can simulate the problem by just using a simply file copy (so NetBackup is not the problem).

Arrrg!
Any other suggestions?
Thanks
Nelson
Tech-To-Tech
Occasional Visitor

Re: Slow file copy speed in one direction on DL380 G5

Here's what I've done so far as a temp soultion until HP releas a newer driver for the NIC.
1) On the current NIC card disabled the following:
- Large Send and Checksum Offload
- RSS
2) Registry:
** Disable RSS:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableRSS and setting it to 0
** DisabledTaskOffload: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\DisableTaskOffload and set it to 1

the driver I have is version 3.4.10.0 dated 5/25/07 so I didn't need to update that. Also, you don't want to disable the TCP Chimney so once HP release a new driver then you can take advantage of the new features offered by SP2. Plus, Windows Update might reset those registry keys.
KarloChacon
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

hi

Tech-To-Tech I want to ask something when you say disable:
- Large Send and Checksum Offload
- RSS

you did that in the NIC Cards

and also did these changes in register

** Disable RSS:
HKEY_LOCAL_MACHINE\......

** DisabledTaskOffload: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet....

I mean you did both changes or you disabled both using register

sorry for my silly question but this topic is been a long issue SP2 - RSS- TOE - chimney - ..

regards
Didn't your momma teach you to say thanks!
Tech-To-Tech
Occasional Visitor

Re: Slow file copy speed in one direction on DL380 G5

Yes I disabled them on both places and I'm glad you asked since I forgot to say that you have to reboot your server(s) after you make these changes on the registry
KarloChacon
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

thanks a lot for clarify that

that is because I've seen that people say I disable RSS but that was only on the NIC cards settings not using register as well as other NIC properties only in NIC cards but not in registry

regards
Didn't your momma teach you to say thanks!
NJK-Work
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

Thanks for the tips.

I made the changes you suggested verbatum and verified my NIC driver is the same as your; it is.

However, I still get the same problem.

The one difference between this server and the others that I fixed by disabling the chimney stuff is that this server is behind a Firewall. Is there anything special I need to have my FW admins set on thier end to make this work?

As a test, I tried the copy process between others servers also behind the FW. I actually have 4 servers, as part of an overall project, that are behind the same FW and configured exactly. I see the same problem from each when going to a device outside the FW. But between them, the copy is fine. So I guess I just answered my own question...it has to be the FW. But what?

Any suggestions?
Nelson
NJK-Work
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

I think I am making progess :)

I undid all the settings you suggested (so the server looks the same as day one with the normal settings). File copies between devices behind the firewall still work fine. So I think this has nothing to do with TOE and the Chimney stuff like the other servers had. It just happens to have the same symptoms.

That being said, I think I will take this up with my FW guys and see what's up.

Unless you guys have any more suggestions...
Thanks for all the help.
Nelson
KarloChacon
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

hi

let me link some MS articles that maybe can help to get a whole idea about these issues


Regarding RSS and disable it - I added here because it refers Firewall

http://support.microsoft.com/kb/927695/

this other one has different scenarios that I think you can try just to check one of them apply for those servers

http://support.microsoft.com/kb/936594

and there is another article that refers something you already did

"""Windows Server 2003 Scalable Networking Pack open issues Third-party firewall vendors cannot connect to the Windows networking stack by using undocumented APIs when TCP Chimney offload is enabled.

TCP/IP statistics that are obtained from offload-capable hardware will be double counted if the following conditions are true:
â ¢The TCP Chimney offload functionality is turned off by using the EnableTCPChimney registry entry.
â ¢The TCP Chimney offload functionality is turned off by using the Netsh int ip set chimney DISABLED command
"""

this is a long one
http://support.microsoft.com/kb/912222/


I think I'm going to print those out

Didn't your momma teach you to say thanks!
Tech-To-Tech
Occasional Visitor

Re: Slow file copy speed in one direction on DL380 G5

Nelson, did you have any issue with the FW prior applying SP2? Also, make sure that you have the server in a Gigabit port on the switch as well as the FW. BTW, what kind of FW do you have?
NJK-Work
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

Karlo - thanks for the suggestions. I had already tried all those steps including installing the hotfix listed in the one MS KB article.

Tech-to-Tech - the servers are built from an image with SP2 already installed. Since we have a volume licensing agreement with MS, we have access to the 2003 CDs with SP2 slipstreamed into the OS - I don't even think I have the option to uninstall SP2 when it is slipstreamed.

I do not know what type of FW we have and I am not sure our security team would appreciate me broadcasting that info anyways - sorry :)

Thanks
Nelson
NJK-Work
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

DING DING DING - we have a solution!

External interface on the FW was misconfigured for 100/Half instead of 100/Full. Not sure why this would only affect traffic in one direction and not the other, but it did.

That's 8 hours of server troubleshooting I will never get back.
NJK-Work
Honored Contributor

Re: Slow file copy speed in one direction on DL380 G5

First problem
Run this on the server:
Netsh int ip set chimney DISABLED


Second problem
Never assume coworkers configured their part of the equation properly AND never believe them the first two times you ASK them if everything is configured properly; ask them a THIRD time :)