NC522SFP 10Gbit card only pushes 2-3 Gbit - Win-2003 x64

[rick jones]

Yes, the *card* supports MSI-X. While that is necessary, it is not sufficient - the OS must also support MSI-X. That was the basis for my asking if Windows 2003 SP2 had MSI-X support.

[Udi Karni]

Oh... the HT Hops. Groan. This seems to be the Achilles' heel of the DL785 G5. It seems to get in the way of database reads as well.

We've been unable to read Oracle sequential at more than 3.5 Gbyte/sec on the DL785 G5, even though synthetic "disktest" can read at 8 Gbyte/sec, and the DL580G5 that's only 16-way can read more than 4 Gbyte/sec. It all seems to stem from "the hops" while on the DL580 G5 the setup is more lean and mean.

I didn't read your previous post carefully enough. I thought you had a DL580G6 - the one with the PCI gen2 slots. Now I understand - you have a DL585 G6 which is still the PIC gen1. So - yes - we are both waiting anxiously for the snazzy new Intel model.

So - back to our predicament. It sounds like you are certainly getting > 5 Gbit from an X4. Realistically - that's all we are looking for. If each of our processes can move 30-35 Mbyte/sec, and each process maxes out a CPU - on a 16-way server like the DL580 we would be able to run no more than 16 transfers maximum, and 35MByte/sec * 16 processes = 5 Gbit. So on the DL580 we don't have cycles for more than 5 Gbit anyway and so that's our target. True, on the DL785 there are more cycles - but most of our servers are 16-way DL580s - so we are just looking for the lowest common denominator.

So - having set out sights on the more modest goal of 5 Gbit - which you have proven is doable on Linux - the question is - why are we only getting 1/2 as much on Windows, and is there anything else we can try.

Next - I was going to try registry parms GlobalMaxTcpWindowSize and TcpAckFrequency although I can't say I'm happy I have to go down the registry route.

[Udi Karni]

I will also continue to check whether Win-2003 SP2 supports MSI-X.

Contrary to popular belief Windows 2003 SP2 and above is actually quite agile and performant and is completely not the oft maligned OS it would appear to be in the blogosphere. A lot of people are still thinking about the clunkiness of NT and win-2000 but win-2003 SP2 is actually very nice.

You do, however, need to push a few concealed buttons, or set things up just so to make this come out.

I will expand my 10Gbit PhD to this topic as well. Naive me, I thought we'll just put the card in and everything will just work.

[rick jones]

HT hops can be an issue for *any* Opteron-based system (well above single-socket I guess). It may be close to epsilon for a two socket, but it can increase from there. Now that Intel have gone to "links" instead of busses we will see the extent to which the same thing happens there.

[Udi Karni]

Rick - does this mean that win-2003 does not support msi-x? Only Win-2008?

http://download.microsoft.com/download/D/1/D/D1DD7745-426B-4CC3-A269-ABBBE427C0EF/NET-T717_DDC08.pptx

[rick jones]

I presume you are referring to slide 5? Certainly that would be my interpretation of that slide.

[Günter Weber]

Hello Udi,

can you tell me with what cable do you connect the two cards directls without switch?

Greetings from Germany Günter Weber

[Udi Karni]

Hi Gunter,

Here are the details for the cable from our Quote:

Item # - 0473060
Manufacturer Part $ - 221692-B23
Description - HP - Fibre Channel cable - LC (M) - LC (M) - 49 ft


By the way - we have switched to testing this card with Windows 2008 and we are getting more like 400-500 Mbyte/sec - about double what we were getting with Windows 2003, not too far from the Linux numbers, and about what can be expected when placing a 10 Gbit card on an X4 slot instead of an X8 as it should.

So - Windows 2003 is definitely the problem, although we still haven't gotten a definite answer as to whether it can me made to work - or you simply shouldn't bother with 10Gbit with Windows 2003.

[Michael H. Jenkins]

I had the same issue using Redhat Linux and was able to get more throughput using these options:

1. Need the latest Firmware and Driver from HP. The driver install in linux will load the firmware.

2. Set Cisco Switch and NIC configuration to Jumbo Frames MTU=9000. Cisco switches can be set higher.

2. Tuning the transmission queue length can sometimes help.

3. Also sometimes you get better performance by pinning the process to a single CPU.
Linux command examples when running iperf:
iperf Server: taskset -p 0x00000001 iperf -s -l 256k -w 1024k -p 9099 -r
iperf Client: taskset -p 0x00000001 iperf -c -l 256k -i 2 -w 1024k -p 9099 -r

I don't follow forms very well so if I don't respond to a question as I probably did not see it. :)

Cheers,
Mike