- Community Home
- >
- Networking
- >
- Legacy
- >
- Switches, Hubs, Modems
- >
- NC522SFP 10Gbit card only pushes 2-3 Gbit - Win-20...
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-09-2009 10:19 PM
тАО11-09-2009 10:19 PM
NC522SFP 10Gbit card only pushes 2-3 Gbit - Win-2003 x64
The 2 cards on the 2 servers are connected via crossover cables - so no network, latency, etc.
We can't seem to push more than 2-3 Gbit/sec even though it appears we are using the correct 64-bit drivers, etc.
??????????????
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-10-2009 04:25 PM
тАО11-10-2009 04:25 PM
Re: NC522SFP 10Gbit card only pushes 2-3 Gbit - Win-2003 x64
*) What sort of server hardware are you running?
*) are you using the latest firmware and drivers from www.hp.com? I think the driver is up to 4.0.505-1 - at least in the Linux space.
*) what are you using to measure transfer rate?
*) is any one CPU in your server(s) saturated when you are measuring transfer rates?
*) are there any packet losses reported for the protocol you are using for transfer? (eg tcp)
*) before you switched from 1 Gig to 10G, how saturated was your most heavily saturated CPU while doing transfers over the 1Gig link?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-11-2009 03:49 PM
тАО11-11-2009 03:49 PM
Re: NC522SFP 10Gbit card only pushes 2-3 Gbit - Win-2003 x64
The servers are DL785 running Win-2003 SP2 x64. They are running the latest firmware and drivers - 4.0.505.0 from 8/20/09.
We have Oracle 11g on both machines and we copy tables across an Oracle db-link. This is a CPU intensive process that maxes out one of the 32 CPUs while pushing about 35 MByte/sec. So - 3-4 of these max out a 1 Gbit connection, and since we want to run many of them concurrently - we are testing 10 Gbit. However - no matter how many we pile on - the curve turns horizontal past 2 Gbit and doesn't increase by much beyond that. We check the speed via perfmon - both for network and disk write speed. The disks on the receiving side can write well over 1 Gbyte/sec - so that's not a problem.
Also - every card has 2 ports - and the card in total - won't exceed 2-3 Gbit - with 2 ports - it just gets divided in half.
I don't think there are packets getting lost.
Before we switched we would run 4 concurrent jobs over 1 Gbit and see around 120 Mbyte/sec transfer rates - so that works flawlessly.
One thing to note - the card is on an X4 instead of the recommended X8 - we have no X8 available. However - we were led to believe that with an X4 we should still expect around 5 Gbit, but 2 Gbit is a little low.
Increasing Jumbo frames to 9000 added maybe 10%. Increasing the max number of Jumbo frames, number of receiving and sending buffers - and setting TCP parms in the registry - 1323=1 and Windowsize as high as 1/4GB didn't help at all.
help !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-12-2009 01:27 PM
тАО11-12-2009 01:27 PM
Re: NC522SFP 10Gbit card only pushes 2-3 Gbit - Win-2003 x64
PCIe1.1 x4 is "the same bandwidth" as a PCI-X 133 MHz slot. The very best you can expect from such a slot is ~7 Gbit/s.
Can you tell how many CPUs are taking interrupts from the NIC? Does Windows 2003 SP2 have support for MSI-X functionality to make that happen on more than one core? If not, then all the interrupts will be handled by one core and it won't matter at that point how many more concurrent connections you are running - that core will become the bottleneck.
In broad handwaving terms, within the context of the "technology" and not specifics of implementation, 10 GbE does little to nothing to make it easier to transfer data than it was for 1 GbE, or for that matter than it was for 100 BT or 10 BT. It still takes just as many CPU cycles to transfer a KB of data with each. Now, there certainly have been implementation details, outside the scope of the IEEE specs, that mean the 100 BT cards might have had better programming models than the 10 BT cards, and the 1 Gig cards have added CKO and TSO to make them "easier" than 100BT. The main thing that 10G NICs add beyond 1 Gig NICs is the ability to spread interrupts across more than one core. Hence the interest in 2003 SP2's support for "MSI-X" and whether or not interrupts from the NIC are being spread - especially on the receiver.
There is still latency when connected back to back - it should though be small :) It likley doesn't matter here since you have several connections, but even back to back, a TCP window size that could hit link-rate over 1 GbE may not be sufficient to hit link-rate over 10 GbE... but we aren't there yet with this situation :)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-12-2009 03:08 PM
тАО11-12-2009 03:08 PM
Re: NC522SFP 10Gbit card only pushes 2-3 Gbit - Win-2003 x64
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-12-2009 03:18 PM
тАО11-12-2009 03:18 PM
Re: NC522SFP 10Gbit card only pushes 2-3 Gbit - Win-2003 x64
I am kicking off 14 transfers - and each one saturates a processor - so on a 32-way - each fully used processor equals 3% CPU - so I do in fact see around 42%-43% CPU utilization. However - I am not seeing the 35 MByte/sec per process I would normally see if I kicked off only 1 or 2 or 3 of them.
All that doesn't really answer your question of whether the interrupts are handled by many CPUs or there's one bottleneck CPU that's taking all the hits. I'm still scratching my head how I look that up.
Are you an HP employee? We routinely have HP employees RDP to our servers so they can see for themselves - using the HP virtual room software. If you have the time or inclination - that would be fabulous. If you can't - I still appreciate any and all suggestions you have - and I will try and pursue them.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-12-2009 03:36 PM
тАО11-12-2009 03:36 PM
Re: NC522SFP 10Gbit card only pushes 2-3 Gbit - Win-2003 x64
I don't think I am sufficiently Windows-savvy to make my RDP'ing into your system worthwhile, sorry.
Meanwhile, your mentioning having used an x4 slot rather than an x8 slot got me curious, so I've shifted my NC522SFP from an x8 slot (#5) of the DL585 G6 to an x4 slot (#6) and will see what comes of it. Sending to two separate systems seems to peak at about 5400 Mbit/s, the receive figures aren't measured yet. Since we have different HW and very different OSes YMMV :)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-12-2009 04:17 PM
тАО11-12-2009 04:17 PM
Re: NC522SFP 10Gbit card only pushes 2-3 Gbit - Win-2003 x64
Also - since your server is a G6 - are these X4 gen 2? those are really like X8 gen 1, right? If so - that's even more curious because it would mean that the "4" after the "X" trumps the "gen". Am I getting all confused? I thought that an X8 gen 1 should give you full performance and by extension - X4 gen 2 should as well.
?????????????????????????
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-12-2009 04:58 PM
тАО11-12-2009 04:58 PM
Re: NC522SFP 10Gbit card only pushes 2-3 Gbit - Win-2003 x64
http://forums.techarena.in/web-news-trends/1161416.htm
Then it would appear that it supports MSI-X and spreads the interrupt load around. No?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-12-2009 05:01 PM
тАО11-12-2009 05:01 PM
Re: NC522SFP 10Gbit card only pushes 2-3 Gbit - Win-2003 x64
My PCIe 1.1 x4 ~= PCI-X 133 handwaving arriving at 7 Gbit/s was based on a different PCI-X 10G card getting 7 Gbit/s in an Integrity rx4640 and that maxing-out the PCI-X 133 MHz slot on that system.
The first couple inbound tests on my DL585 G6 RHEL 5.3 setup have completed, and they have hit ~6200 Mbit/s. That is getting closer to the 7 Gbit/s handwaving I did.
Also, depending on where things are running on your DL785, and which slot is in use by the NIC, the NC522SFP may have to reach as far as 4 HT hops away to fetch the data to transmit (one from the HT2PCIe chip to the first CPU and as many as three to reach memory hanging from another CPU). In the DL585 G6, since it is only a 4P configuration, the most it has to reach-out is three HT hops. The farther away the memory, well you get the idea. That Integrity rx4640 was a bus-based system so there was only ever one "hop" essentially.
As for G6 being PCIe 2.0. It is my understanding that the Intel-based G6 systems have PCIe 2.0 slots (the DL380 G6 at least, I have one on internal order for just that very reason) I *do* know that a DL785 G6 is still PCIe 1.1. I ass-u-me but do not know a DL585 G6 also remains PCIe 1.1, the DL585 G6 Quickspecs seem to be mum on the topic.