Slow network performance with W2K and NOT XP

[doubleH]

Howdy,

I am starting to do a parallel migration from our legacy network to our new HP network consisting of 2 5406's and 1 3500.

I know this sounds weird, but I am having performance issues with only Windows 2000 SP4 pc's and servers and NOT with Windows XP or Windows Server 2003. When I say performance issues I mean latency. On a 2000 workstation I can ping its local gateway and get <10ms response time. If I do the same thing on WinXp I get <1ms which is what I expect. I've proven this by taking a WinXP machine and plugged it into the same network connection that the W2K pc was using and don't have latency. I even rebuilt the W2K pc with XP and am now getting <1ms response time. On the W2K pc I made sure that the nic drivers were up to date and also tried hard coding the speed/duplex settings on the pc and switch with no luck.

Thanks

[Bruce Campbell_3]

This is just a wild guess, but any chance
the time resolution on the older OS's are
10msec, while the new OS is 1msec ?

Do you see actual performance problems
copying data ? Like if you have a
genuine 10 millisecond per packet delay,
your throughput will be pretty awful,
max 150kilobytes/sec or so.

[Matt Hobbs]

Bruce is spot on. It's just the ping application under 2000 won't go any lower than <10ms.

If you run a packet capture you will see the actual response is much quicker.

[doubleH]

I wish it was cosmetic like you are indicating, but in this case it is not. Performance is very slow especially when running on of our main applications. I should note that the application ran fine on my legacy network.

Thanks

[Matt Hobbs · Accepted solution]

If it is Windows 2000 only, then you may benefit from some TCP tuning parameters. This page has some good information and link to a PDF Microsoft Document.

http://rdweb.cns.vt.edu/public/notes/win2k-tcpip.htm

The fact that it was running fine on the old network is hard to explain, but I think you should focus on the OS rather than the new network infrastructure.

This application that is suffering, can you share more detail about it?

What about basic file copy tests?
Windows 2000 Server > 2000 client = ?
Windows 2000 Server > XP client = ?
And the same tests for 2003 Server.

For much better throughput testing I would recommend using iperf: http://dast.nlanr.net/Projects/Iperf/

With Windows 2000 you will never see it display less than <10ms, to truly see the ping response you will need to use a packet capture application like Ethereal.

[doubleH]

awesome thanks. i adjusted the tcpwindowsize and performance appears to be back to normal!

[doubleH]

Looks like I spoke too soon. Things just seem a little pokey on the new network. My backups are now suffering and they are taking an extra 2 hrs longer to complete then when on the legacy network. The legacy network is a 100FX network. I migrated the backup server (W2K SP4) from its 100FX network to 100/1000T and the performance deteriated. About 50% of the servers, and few test pc's and a couple of printers are on the new network.

There are also some XP pc's that again seem pokey when I used their 1000T nic compared to when they were on the 100FX network.

I looked at iperf, but couldn't figure out how to run under windows. Can anyone recommend a windows based tool that will tell me what is going on the new network?

[Sergej Gurenko]

Try to enable qos pass through mode on a switch. ( http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1002810 )
Try to connect all gig NICs to one switch, and all the 100mbps NICs to another (if there is a possibility.

[doubleH]

the 'qos-passthrough-mode' doesn't appear to be available on my 5406's.

[Matt Hobbs]

With iperf you simply set one side as the server and one as the client:

server: iperf -s
client: iperf -c

You can set a lot more options if required, the default test is 10 seconds so I increase that to 60 seconds with this:

iperf -c -t 60 -i 10

Looking back at your network, if you do a 'show span', how long ago was your last topology change? If it is in the seconds or even the minutes that is probably going to be your problem. Otherwise look through the port counters to try and see anything unusual there.

Otherwise if you can post a copy of the show tech from the ProCurve switches, it may help (remove any sensitive info, public IP's or SNMP community names).

Make sure you're running K.11.33 on these switches too.

[doubleH]

thanks for the reply matt. i've attached the results of show tech all for the cores. note that on both cores there is a redundant power supply installed just not plugged in and Core_1 port A9 is the port that connects the new network to the legacy network. the server that we are running the app from (w2k3) was on the legacy network. i took it off and put it on the new network and that seems to have made a big difference although itâ s hard to gage because no one is here now. i've recorded the results and will compare tomorrow during business hours. if you could still look at the logs that would be very much appreciated.

i don't mean to sound like a dumbarse, but i still don't understand how to get iperf to run on windows. i downloaded the source code, but I don't see a file called iperf. what am i missing?

[Matt Hobbs]

No problem, with iperf you just need to download the previous 1.7.0 version which has a Win32 binary.

http://dast.nlanr.net/Projects/Iperf/iperf-1.7.0-win32.exe

[Matt Hobbs]

Also the show tech's don't seem to be attached?

[doubleH]

doh..

[Matt Hobbs]

Spanning-tree looks healthy which is great.

I think you've found the problem which was is port A9. It looks to have a duplex mismatch. The logs indicate you should be setting it back to Auto.

Just a couple of other recommendations, for spanning-tree you have both Core switches set to Priority 0. I'd assume that you want Core1 to be the spanning-tree root. I'd recommend that you set Core2 to Priority 1.

I'm guessing in time that you'll set up VRRP for router redundancy? I would recommend you do.

Finally the timezone setting is in minutes, so if you're -4GMT, the setting should be -240.

[doubleH]

If I remember correctly when I had it set to auto it was giving excessive collisions which is why i put it to 100fx. Maybe it doesn't like the fact that it is going through a media converter.

I've implemented your recommendation for the spanning-tree priority...good catch! I also have a 3500 and have attached the show tech all as well. I've made it spanning-tree priority 3. Is that correct?

Yeah..I need to talk to the fella's that hold the credit card and see if we can get VRRP.

I ran Iperf and have also attached the results. There is 2 sections. The first section shows the results running iperf when the server is connected to the legacy network using a gigabit fiber card and the 2 xp workstations running on the hp copper network. The second section shows the details of running the same tests, but with the server on the HP copper network using a gigabit copper nic. You will notice a huge improvement with the workstation running with the gig nic once the server was put on the hp network. It looks like the link between the 2 networks is causing the problem. What are your thoughts?

[Matt Hobbs]

If A9 is now at 100FX and you're no longer seeing those errors, then definitely leave it at that.

Auto to Auto = OK
100FDx to 100FDx = OK
100FDx to Auto = bad news.

The iperf results are interesting, I think you should be able to get more out of it though. With 100Mbit connections I usually see results between 80-90Mbit. Disable any firewalls you have on the clients for testing. Also check the utilisation on the links through the web interface to make sure that the paths between the 2 hosts have a clear path.

It also depends on the networks cards though and the age of the machines.

For spanning-tree on the 3500, it doesn't really matter what priority it is set to as long as it's lower than 1. The default of 8 would be just as good. Also you probably don't need ip routing enabled on the 3500 or those helper addresses. It should be just acting as a switch.

Don't forget to keep assigning points along the way.

[doubleH]

i've run some more iperf tests from my laptop that is running 100/Full (hard coded on the nic and on the port) to a server running 1000/Full. both devices are on the new HP network. the results seem to be all over the map. is this expected on a new network with only a few servers, pc's, and printers?

laptop (xp) is about 3yrs old using the onboard nic Intel Pro 100/VE. server is a proliant ml330 g3 (w2k3) using the onboard nic NC7760

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1896] local 192.168.77.252 port 5001 connected with 192.168.74.24 port 9969
[ ID] Interval Transfer Bandwidth
[1896] 0.0-10.0 sec 71.8 MBytes 60.2 Mbits/sec
[1864] local 192.168.77.252 port 5001 connected with 192.168.74.24 port 9980
[ ID] Interval Transfer Bandwidth
[1864] 0.0-10.5 sec 90.5 MBytes 72.1 Mbits/sec
[1888] local 192.168.77.252 port 5001 connected with 192.168.74.24 port 9990
[ ID] Interval Transfer Bandwidth
[1888] 0.0-10.0 sec 69.1 MBytes 58.0 Mbits/sec
[1884] local 192.168.77.252 port 5001 connected with 192.168.74.24 port 9998
[ ID] Interval Transfer Bandwidth
[1884] 0.0-10.0 sec 97.6 MBytes 81.9 Mbits/sec
[1876] local 192.168.77.252 port 5001 connected with 192.168.74.24 port 10065
[ ID] Interval Transfer Bandwidth
[1876] 0.0-10.7 sec 68.9 MBytes 53.9 Mbits/sec
[1892] local 192.168.77.252 port 5001 connected with 192.168.74.24 port 10072
[ ID] Interval Transfer Bandwidth
[1892] 0.0-10.0 sec 83.4 MBytes 69.9 Mbits/sec
[1888] local 192.168.77.252 port 5001 connected with 192.168.74.24 port 10234
[ ID] Interval Transfer Bandwidth
[1888] 0.0-10.2 sec 75.7 MBytes 62.1 Mbits/sec
[1860] local 192.168.77.252 port 5001 connected with 192.168.74.24 port 10248
[ ID] Interval Transfer Bandwidth
[1860] 0.0-10.5 sec 96.9 MBytes 77.7 Mbits/sec

[Matt Hobbs]

To make sure you have a good baseline figure, the best option would be to use a cross-over cable between the 2 machines, run the iperf test and see what results you can get that way. Next connect them to the same switch, results should be very similar if other devices aren't using the resources of the server at the same time.

Finally connect them to different switches which will test the switch to switch links.

For results like that it sounds like there is a bottleneck somewhere, possibly due to regular network traffic already on the network. Easiest way to find any bottlenecks and errors on the network would be to use ProCurve Manager Plus, start Traffic Monitor (sampler), and hopefully you will start to see a bigger picture of the network.

PCM+ also has a configuration checking report which will alert you to any misconfigurations on the switches which can be handy.

[doubleH]

well i did some more testing and it looks like something on Core 2 is messed up. Here are the results ...

==================
Testing Devices
==================
Workstation, WinXP SP2(no firewall), On board Broadcom NetExtreme Gig.
Laptop, WinXP SP2(no firewall), On board Broadcom NetExtreme Gig.
Server, W2K3 SP1, HP NC7771 Gig Nic

==============================
Testing Scenerios and Results
==============================

1. Workstation to Laptop with Cross Over Cable
Both nics were set to auto. Using iperf i was seeing 432-450mbit/sec

2. Workstation to Laptop connected to Core 2
Both nics were set to auto. Using iperf i was seeing 424-467mbit/sec

3. Workstation to Server Connected to Core 2.
Both nics were set to auto. Using iperf i was seeing 10-35mbit/sec

4. Workstation (Core 2) to Server (Core 1).
Both nics were set to auto. This time I moved the server to Core 1 and had much better results. Using iperf i was seeing 366-401mbit/sec

I'm really not sure what is causing this. Any ideas?