topic Re: Slow file copy speed in one direction on DL380 G5 in ProLiant Servers (ML,DL,SL)

Slow file copy speed in one direction on DL380 G5

NJK-Work — Thu, 06 Sep 2007 12:27:13 GMT

Hi,

We have about 15 DL380 G5 servers all acting in the same way. Copying files from it is very slow. However, if I am on the server and copy files to some other device, it runs at normal (fast) speed. In other words, a copy pull is slow, a copy push is fast. So if I am logged in to my Windows XP PC and copy files from one of the servers to my local drive it is slow. If I RDP to the server and copy those same files (from the server) to my PC, it is very quick.

Our environment:
DL380 G5 - all firmware and HP drivers at latest version.
Windows 2003 R2 SP2 (standard and enterprise)
NICs are configured for 1000 GB/s Full duplex hardcoded on both the server and the switches
Switches are Cisco 6509 chassis with WS-X6748-GE-TX blades (I am not a networking guy - this is the info they provided me).

We have older DL380 G4 servers running at GB Ethernet on the same switches with no problems.

Anyone have any ideas?
Thanks
Nelson

Re: Slow file copy speed in one direction on DL380 G5

James ~ Happy Dude — Fri, 07 Sep 2007 06:34:32 GMT

Hello Nelson,

1) Run Firmware CD from
http://h18023.www1.hp.com/support/files/server/us/download/27783.html

2) Is there any NIC team ?
if so, dissolve NIC, update individual NIC firmware, Update NIC Drivers, PAIR NIC using Latest NIC Utility.

3) Apply Latest PSP:
http://h18023.www1.hp.com/support/files/server/us/locate/69_6213.html#0

4) In your PC, go to Device manager, select the network Card & goto Properties; then Advance TAB; Set "Speed & Duplex" to be AUTO;;; If its already AUTO, Force it to 100MBps & check for any difference.

Regards,
James.

Re: Slow file copy speed in one direction on DL380 G5

James Kennedy_5 — Fri, 07 Sep 2007 08:29:53 GMT

Generally for 1000 MB you should be doing Auto on the Gigabit switch and NICs (as apposed to forcing 100/FULL on the lower speed NICs) .. but forcing should be ok too.

Anyways, I've been having lots of Network issues with G5 servers and PSP 7.90. Try disableing TOE on the team and both NICs. See if that helps.

Also, since you're running SP2, you may want to install this patch from microsoft: http://support.microsoft.com/kb/936594

Re: Slow file copy speed in one direction on DL380 G5

NJK-Work — Fri, 07 Sep 2007 10:13:12 GMT

Thanks for the responses.

I am already at the latest firmware.
I am already at Support Pack 7.9
The problem also exists with Support pack 7.8
I already have the correct speed and duplex configure on my pc, my server and all ports involved. But it is just not my PC anyways - it is any device I try to copy files from the problematic server including other (good)servers.

I will try AUTO/AUTO and see if that helps
I will investigate the MS article.

Thanks for the tips.
Nelson

Re: Slow file copy speed in one direction on DL380 G5

Hense, Klaus — Fri, 07 Sep 2007 10:37:21 GMT

Hallo, try connect your server and your PC or other server with a cross Link cabel and then test the performance!
test with break the Teaming.
Good luck..

Re: Slow file copy speed in one direction on DL380 G5

NJK-Work — Fri, 07 Sep 2007 11:05:07 GMT

I had our Network guy switch the port to 100 FULL and I hardcoded the NIC to 100 FULL and now it works fine. So it is something with running those NICs on the DL380 G5 at GB that breaks something.

Nelson

Re: Slow file copy speed in one direction on DL380 G5

Igor Karasik — Fri, 07 Sep 2007 13:28:37 GMT

G5 has newest NIC, therefore
try to disable RSS and TCPChimney:

http://support.microsoft.com/default.aspx?scid=kb;EN-US;927695
http://msexchangeteam.com/archive/2007/07/18/446400.aspx

Re: Slow file copy speed in one direction on DL380 G5

NJK-Work — Fri, 07 Sep 2007 14:36:18 GMT

Disabling TCPChimney fixed the problem...thanks!

Netsh int ip set chimney DISABLED

But what does that break, or should I say, what does it limit by disabling it. Am I loosing some other features by fixing another?

Thanks
Nelson

Re: Slow file copy speed in one direction on DL380 G5

Igor Karasik — Sat, 08 Sep 2007 12:49:55 GMT

>>But what does that break, or should I say, what does it limit by disabling it. Am I loosing some other features by fixing another?

TCP Chimney offloads networking tasks from CPU, leaving more power for applications - at least in theory :-)

AFAIK, it is relatively new feature, exists only on newest NIC's and has many problems (at least in Microsoft implementation)

Also, according this HP article - TCP Chimney will not work if HP virus throttle feature installed (part of PSP):
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=398225&prodTypeId=15351&prodSeriesId=398225&objectID=c00810667

Re: Slow file copy speed in one direction on DL380 G5

NJK-Work — Mon, 10 Sep 2007 08:50:01 GMT

Thanks again!

Nelson

Re: Slow file copy speed in one direction on DL380 G5

Tom E Reynolds — Fri, 14 Dec 2007 18:51:48 GMT

I also had to disable TCPChimney to fix my upload problems through a VPN to my new HP DL380G5 servers with the onboard HP NC373i NIC.

Netsh int ip set chimney DISABLED

in the Biz forum see:

http://forums11.itrc.hp.com/service/forums/bizsupport/questionanswer.do;HP-FORUMS-S-WPA-IDX=Hv8JY1vnT51pNhfT6YS10pHP1QTy02pnJHJpfpgTvJ4V0B9znlJM!-1885766594!-124372549?threadId=1153566&admit=109447626+1197653161555+28353475

Re: Slow file copy speed in one direction on DL380 G5

PinnacleCS — Fri, 14 Dec 2007 22:49:45 GMT

We had this same issue. We found the best way to do this is to go into the HP Network Config utility, properties of the NIC or team, settings tab and disable TCP Offload. This fixed all of our slowness issues.

Re: Slow file copy speed in one direction on DL380 G5

Jimbo! — Mon, 17 Dec 2007 19:25:31 GMT

For the benefit of the search, this can be caused by auto-negotiate at one end and fixed full duplex at the other.

What happens is that since one end is fixed, auto-negotiate fails and the port or NIC immediately drops to auto-sense. Again, since the other end is fixed, this succeeds so one end will be 100/full (or 1000/full) and the other 100/half (or 1000/half).

Say the switch is 100/full and the server 100/half. Client is correctly configured and 100/full and both ends.

Files are transferred with a sliding window, allowing multiple data packets to be acknowledge with a single ACK.

Consider copying to the server:

Client sends a load of packets, server starts discarding them once it's window is full and awaits a gap for transmission. The client now stops because of lack of ACKs, the server can then send the ACK (with some packets lost), and the process can continue. The overall throughput seems reasonable.

Consider copying a file from the server:

Server sends a load of packets, the very first ACK, soon after transmission starts, from the client is seen as a collision and the server stops sending. Client assumes packet is lost and re-sends ACK after maybe one second delay. Process repeats, resulting in only a few packets each second, hence very slow throughput.

Hope that helps!

Re: Slow file copy speed in one direction on DL380 G5

NJK-Work — Fri, 04 Jan 2008 22:13:39 GMT

Well, the problem is back on a new DL380 G5 server I just built. Using the this command does not fix the problem anymore:

Netsh int ip set chimney DISABLED

I tried setting the server to AUTO and the switch port to AUTO, and I also tried setting the server to 1GB FULL and the switch port to 1GB FULL. I get the same results.

Pulling files to the server is fine. Pushing files to another host (from that server) is slow. We discovered the problem because backups (NetBackup) were super slow. And backups are a push - files going from the server to the backup host, which results in the slow copy. But I can simulate the problem by just using a simply file copy (so NetBackup is not the problem).

Arrrg!
Any other suggestions?
Thanks
Nelson

Re: Slow file copy speed in one direction on DL380 G5

Tech-To-Tech — Sat, 05 Jan 2008 01:06:53 GMT

Here's what I've done so far as a temp soultion until HP releas a newer driver for the NIC.
1) On the current NIC card disabled the following:
- Large Send and Checksum Offload
- RSS
2) Registry:
** Disable RSS:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableRSS and setting it to 0
** DisabledTaskOffload: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\DisableTaskOffload and set it to 1

the driver I have is version 3.4.10.0 dated 5/25/07 so I didn't need to update that. Also, you don't want to disable the TCP Chimney so once HP release a new driver then you can take advantage of the new features offered by SP2. Plus, Windows Update might reset those registry keys.

Re: Slow file copy speed in one direction on DL380 G5

KarloChacon — Sat, 05 Jan 2008 02:39:20 GMT

hi

Tech-To-Tech I want to ask something when you say disable:
- Large Send and Checksum Offload
- RSS

you did that in the NIC Cards

and also did these changes in register

** Disable RSS:
HKEY_LOCAL_MACHINE\......

** DisabledTaskOffload: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet....

I mean you did both changes or you disabled both using register

sorry for my silly question but this topic is been a long issue SP2 - RSS- TOE - chimney - ..

regards

Re: Slow file copy speed in one direction on DL380 G5

Tech-To-Tech — Sat, 05 Jan 2008 03:42:44 GMT

Yes I disabled them on both places and I'm glad you asked since I forgot to say that you have to reboot your server(s) after you make these changes on the registry

Re: Slow file copy speed in one direction on DL380 G5

KarloChacon — Sat, 05 Jan 2008 04:12:21 GMT

thanks a lot for clarify that

that is because I've seen that people say I disable RSS but that was only on the NIC cards settings not using register as well as other NIC properties only in NIC cards but not in registry

regards

Re: Slow file copy speed in one direction on DL380 G5

NJK-Work — Sat, 05 Jan 2008 04:57:06 GMT

Thanks for the tips.

I made the changes you suggested verbatum and verified my NIC driver is the same as your; it is.

However, I still get the same problem.

The one difference between this server and the others that I fixed by disabling the chimney stuff is that this server is behind a Firewall. Is there anything special I need to have my FW admins set on thier end to make this work?

As a test, I tried the copy process between others servers also behind the FW. I actually have 4 servers, as part of an overall project, that are behind the same FW and configured exactly. I see the same problem from each when going to a device outside the FW. But between them, the copy is fine. So I guess I just answered my own question...it has to be the FW. But what?

Any suggestions?
Nelson

Re: Slow file copy speed in one direction on DL380 G5

NJK-Work — Sat, 05 Jan 2008 05:09:53 GMT

I think I am making progess :)

I undid all the settings you suggested (so the server looks the same as day one with the normal settings). File copies between devices behind the firewall still work fine. So I think this has nothing to do with TOE and the Chimney stuff like the other servers had. It just happens to have the same symptoms.

That being said, I think I will take this up with my FW guys and see what's up.

Unless you guys have any more suggestions...
Thanks for all the help.
Nelson