DL380G5 with HP NC510C losing connection

[vivpau]

Hello,

we have two DL380G5, each with 2 CPU X5450, 12 GB RAM, Windows Server 2003 Std SP 2, 2 x NC510C 10GbE Network Adapter. The 10 GbE adapters are connected to two diffrent HP ProCurve 6400cl-6XG switches (J8433A). Firmware and driver version for the NC510C are 3.4.336.
The problem is that sometimes the adapters loses their connection (telling a network cable is not pluged in). If we go to the Network Connections window and right click on the adapter icon selting "disable" then "enable" again the adapter resets an is working again. The switch schows no errors. All adapters in the servers are affected, but not at the same time. Them time between two failures is about one to tree weeks.
Any suggestions?
Thread http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1244786 seems to be related, but has the prblem of to much RAM. This is not the case here.

[Blazhev_1]

Hi,

you can try to disable "Large send offload" from device manager-> NIC Properties -> Advanced... It causes sometimes problems, you can test.

regards

[rick jones]

Did you actually flash the 336 firmware onto the NICs explicitly? I got burned at one point under Linux when the driver (via ethtool) reported a firmware version of 336 even though I'd not explicitly flashed the 336 firmware. There may have been some implicit override happening, but the kernel dmesg output was suggesting that some features were not enabled because the card had old firmware.

After I explicitly flashed the 336 firmware via the utility, those messages went away.

Of course, that was linux and wasn't a connectivity problem (well, my netperf runs are longish, but not that long that I can say there wasn't connectivity difficulting lurking) so it may not be germane to your problem but perhaps it helps.

[KarloChacon]

hi vivpau

any update about recommendations above?

I got similar issue Dl360 g5 W2003 x64 bits - At least once a day connection is lost

driver up to date, I will reflash firmware and disable Large send offload

any update?

thanks

[vivpau]

Hello,

thanks for the suggestions.
The firmware was explicitly flashed. I will try to disable "Large send offload".
Does any body know how much performance this may cost?

regards

[rick jones]

The effect of disabling large send offload will depend entirely on the nature of the traffic your system is sending. If it is mostly smallish request/response sort of stuff, then the effect will be minimal. If the traffic is mostly inbound the effect will still be minimal.

If the traffic is bulk outbound - say FTP server or some variations on file server - the effect will be to increase the number of trips up and down the protocol stack per quantity of data transferred. This will increase CPU utilization. If you had plenty of idle CPU you may see little to no effect on throughput. If you are close to CPU saturation then you will see a drop in throughput.

One of those wonderful "It depends" answers isn't it ?-)

[vivpau]

Yes Rick, that's a wonderful answer. ;-)
We use the adapters to handle iSCSI traffic. The system design should have no bottlenecks and that's the reason why we selected the 10 GbE cards. At this time the CPU load is quite asymmetric, since we have two CPUs each with 4 cores (total 8 cores). One of the cores has 100% (runnig thread is "system") while all the others have almost nothing to do. It could be a problem of the driver or the iSCSI-Software (DataCore) that doesn't support mulithreading.
Very unsatisfactory :(
Regards

[rick jones]

I have very weak windows fu but how many TCP connections are driving the iSCSI stuff? If there are only a few, even if the NIC has its interrupts spread across several cores, only one core or a small number of cores will get the traffic. Were this linux I'd look at /proc/interrupt output to see which CPUs were getting interrupts from the NIC. Perhaps someone can suggest how one does that under Windows.

From your writup I'm guessing the iSCSI is host based software? Indeed if that cannot scale with cores, even having a NIC and driver that can scale with cores won't get you very much.

IIRC others have reported in this thread they see link issues too. Still, while I cannot in any way shape or form suggest it is involved, you might want to take the opportunity to make sure your 6400 is up-to-date on firmware. If you see link loss on both NICs in each system, you could, just for experimentation (modulo any teaming) try connecting one NIC in each of the two systems directly and not go through the switch, and run some traffic through it (say perhaps netperf :) and see if the back-to-back connection is any more stable than the connection through the switch.

[Russell Kole]

I have the identical issue with 5 Brand new DL580's. (32 GB mem, 16 Core, windows 2k3 x64 edition.

I am running 3.4.336 both in bootcode, and driver, and 3.4.337 on both, on a different server, and have the issue with both.

These servers have 2 NC510C installed in each going to seperate Cisco 6509 switches.
each connection times out on its own, and not together, or at the same time.

I am contacting HP support over this issue. Has anyone had a resolution for this?

[Russell Kole]

I have contact HP support, i was instructed to do the following:

Disable Recive Side Scaling
Disable Transmit TCP/UDP checksum offload
Disable IP Checksum Offload.

I have also update all servers to driver version 3.4.337.

Im curious if I should go to Firmware version 3.0. I dont know what the current version is ATM.

Ill post here if it resloves the issue.

[Antoine Baumann]

Hi, we have a HP AiO1200r with one NC510C, running on windows storage server 2003 Standard Edition, Service Pack 2, R2. It is loaded with 4GB or ram and has the PAE, it also has a smart array P800 controller.

We are experimenting a similar problem, may be not exactly same issue: we are not getting the "network cable is unpluged" icon&message, but the 10gig card is loosing connection every two hours or so. When we disable it and enable it back, it works fine again for a couple of hours.
Now we have switched back to the gigabit inboard card, because we have to work, but we really need this 10gig to work. Any a update of info would be much appreciate. Thanks,
antoine.

PS: we have contacted HP support, but no solution for now.

[Rich Bryant]

Please tell me someone has found a solution to this. I've been pulling my hair(only a little left) out with this problem. Now I found out I'm not the only one? I've replaced NIC cards, cables, servers, rebuilt OS and all the time it could be firmware. I feel like Florida Evans when James died.

[Antoine Baumann]

Hi Rich,

Well first sorry I thought I had posted the "next step to our story".
So after a few try to tweak the gigabit settings without any success, HP sent a new NC510C (note that it was the second time we were going to replace it, as the first card didn't even start), but they said it is the latest build version of this gigbit card. We tried out and since we replaced it, the connectino hasn't been lost. That is nearly one month ago.

antoine baumann

[Rich Bryant]

Problem resolved. I seen this on a sister thread.

HP NC510C losing connection

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1244786

The problem is the NIC doesn't function properly with >32GB RAM in the system, in my case Windows 2003 server x64, Proliant DL380G5. My server had exactly 32GB of ram. I took it down to 24GB of ram and it worked like a charm. Took it back to 32GB of ram and the connection started dropping again. The problem is with the NC510C (NetXen rebranded NIC)working in servers having more than 32GB of ram. Go figure.

[rick jones]

Ah, _that_ problem. I'd internalized that one so long ago id didn't even occur to me :( Sorry.

Not that it helps, but the RAM limit for the NC510 is higher under Linux.

Also, there is another "standup" PCIe 10G NIC on the HP CPL - the AD386A. It is there for HP-UX/Integrity systems but has been known to work under Linux/ProLiant. I cannot say if it has been tried under Windows/Proliant.

Of course that would all fall under:

Supported, known to work -> warm fuzzies all around
Supported, not known to not work -> an HPite may be in trouble
Supported, known to not work -> an HPite is in trouble
Unsupported, known to work -> lucky today, unlucky tomorrow?
Unsupported, not known to not work -> there but for the grace of Turing
Unsupported, known to not work -> no, it was not deliberate ;-)

Under Linux the card is driven by the cxgb3 driver - at its heart the card has a Chelsio T3C chip.

[KarloChacon]

hi guys

I know this kind old thread but..

I got Dl380 g5 - NC510C
When files transmitting files like 10 - 15 GB size it takes long time to transmit the files almost 3 minutes

latest driver - firmware same issue

these are the NIC values

802.1p QoS Packet Tagging = 0
Maximum Frame Size = 8014 Bytes
Flow Control = 1
Tx IP Checksum Offload = 1
Rx IP Checksum Offload = 1
Tx TCP Checksum Offload = 1
Rx TCP Checksum Offload = 1
Large Send Offload = 0
Number of Tx Descriptors = 1024
Number of Rx Descriptors = 8192
Number of Rx Descriptors (Jumbo) = 4096
TaskOffloadLSOMaxOffloadSize = 64000
TaskOffloadLSOMinSegmentCount = 2
TaskOffloadLSOIPOptions = 0
TaskOffloadLSOTCPOptions = 1
TaskOffloadChecksumIPv4TxIPOptions = 0
TaskOffloadChecksumIPv4RxIPOptions = 0
TaskOffloadChecksumIPv4TxTCPOptions = 0
TaskOffloadChecksumIPv4RxTCPOptions = 0
Receive-Side Scaling = 0


any idea what could it be?
is there any table performance comparison to check if my NIC is OK? I mean maybe you guys are getting same time like

thanks a lot

[KarloChacon]

sorry I forgot to add W2003 x64 bits

[Rich Bryant]

Windows 2003 SP2? is the SP2 slipstreamed or did you add it after? The reason I'm asking is becuase SP2 has the Scalable Networking Pack (SNP) that slows the nic down significantly. If it's slipstreamed like mine was I couldn't disable it. I had re-installed Win2k3 SP1 I think if you installed SP2 manual you can disable the SNP. Google it.

[KarloChacon]

Rich interesting I will check that out

[sgadsby_1]

vivpau,

I am looking to deploy a similar solution on same hardware with Datacore SANmelody & wondered whether you could expand on the CPU issue & performance you were seeing.

Is your 10GbE NIC now stable? Are you getting good iSCSI performance? Has your CPU utilisation settled down? Are you running ESX/Xen on it at all?

This may be a bit off topic so if so let's start a new thread.

Thanks,
Simon.

[Peter J. Milanese]

I am having a similar issue. I have a 510F in a DL380 running as a Cell Manager for DP6. It's fine all day, but once a backup job start, the 10G sends out 4 million multicast packets causing my entire core to go nuts.

Of course, this in turn causes the network connection to drop, as well as the other 4000 nodes on the network.

Have there been any concrete resolutions to the 510 issues?

[rick jones]

Peter - that sounds like a "different enough" issue that it should be its own thread. Including some captures of these multicast packets would be a good thing to include in the new base thread.