DL380G5 with HP NC510C losing connection

[vivpau]

Hello,

we have two DL380G5, each with 2 CPU X5450, 12 GB RAM, Windows Server 2003 Std SP 2, 2 x NC510C 10GbE Network Adapter. The 10 GbE adapters are connected to two diffrent HP ProCurve 6400cl-6XG switches (J8433A). Firmware and driver version for the NC510C are 3.4.336.
The problem is that sometimes the adapters loses their connection (telling a network cable is not pluged in). If we go to the Network Connections window and right click on the adapter icon selting "disable" then "enable" again the adapter resets an is working again. The switch schows no errors. All adapters in the servers are affected, but not at the same time. Them time between two failures is about one to tree weeks.
Any suggestions?
Thread http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1244786 seems to be related, but has the prblem of to much RAM. This is not the case here.

[Blazhev_1]

Hi,

you can try to disable "Large send offload" from device manager-> NIC Properties -> Advanced... It causes sometimes problems, you can test.

regards

[rick jones]

Did you actually flash the 336 firmware onto the NICs explicitly? I got burned at one point under Linux when the driver (via ethtool) reported a firmware version of 336 even though I'd not explicitly flashed the 336 firmware. There may have been some implicit override happening, but the kernel dmesg output was suggesting that some features were not enabled because the card had old firmware.

After I explicitly flashed the 336 firmware via the utility, those messages went away.

Of course, that was linux and wasn't a connectivity problem (well, my netperf runs are longish, but not that long that I can say there wasn't connectivity difficulting lurking) so it may not be germane to your problem but perhaps it helps.

[KarloChacon]

hi vivpau

any update about recommendations above?

I got similar issue Dl360 g5 W2003 x64 bits - At least once a day connection is lost

driver up to date, I will reflash firmware and disable Large send offload

any update?

thanks

[vivpau]

Hello,

thanks for the suggestions.
The firmware was explicitly flashed. I will try to disable "Large send offload".
Does any body know how much performance this may cost?

regards

[rick jones]

The effect of disabling large send offload will depend entirely on the nature of the traffic your system is sending. If it is mostly smallish request/response sort of stuff, then the effect will be minimal. If the traffic is mostly inbound the effect will still be minimal.

If the traffic is bulk outbound - say FTP server or some variations on file server - the effect will be to increase the number of trips up and down the protocol stack per quantity of data transferred. This will increase CPU utilization. If you had plenty of idle CPU you may see little to no effect on throughput. If you are close to CPU saturation then you will see a drop in throughput.

One of those wonderful "It depends" answers isn't it ?-)

[vivpau]

Yes Rick, that's a wonderful answer. ;-)
We use the adapters to handle iSCSI traffic. The system design should have no bottlenecks and that's the reason why we selected the 10 GbE cards. At this time the CPU load is quite asymmetric, since we have two CPUs each with 4 cores (total 8 cores). One of the cores has 100% (runnig thread is "system") while all the others have almost nothing to do. It could be a problem of the driver or the iSCSI-Software (DataCore) that doesn't support mulithreading.
Very unsatisfactory :(
Regards

[rick jones]

I have very weak windows fu but how many TCP connections are driving the iSCSI stuff? If there are only a few, even if the NIC has its interrupts spread across several cores, only one core or a small number of cores will get the traffic. Were this linux I'd look at /proc/interrupt output to see which CPUs were getting interrupts from the NIC. Perhaps someone can suggest how one does that under Windows.

From your writup I'm guessing the iSCSI is host based software? Indeed if that cannot scale with cores, even having a NIC and driver that can scale with cores won't get you very much.

IIRC others have reported in this thread they see link issues too. Still, while I cannot in any way shape or form suggest it is involved, you might want to take the opportunity to make sure your 6400 is up-to-date on firmware. If you see link loss on both NICs in each system, you could, just for experimentation (modulo any teaming) try connecting one NIC in each of the two systems directly and not go through the switch, and run some traffic through it (say perhaps netperf :) and see if the back-to-back connection is any more stable than the connection through the switch.

[Russell Kole]

I have the identical issue with 5 Brand new DL580's. (32 GB mem, 16 Core, windows 2k3 x64 edition.

I am running 3.4.336 both in bootcode, and driver, and 3.4.337 on both, on a different server, and have the issue with both.

These servers have 2 NC510C installed in each going to seperate Cisco 6509 switches.
each connection times out on its own, and not together, or at the same time.

I am contacting HP support over this issue. Has anyone had a resolution for this?

[Russell Kole]

I have contact HP support, i was instructed to do the following:

Disable Recive Side Scaling
Disable Transmit TCP/UDP checksum offload
Disable IP Checksum Offload.

I have also update all servers to driver version 3.4.337.

Im curious if I should go to Firmware version 3.0. I dont know what the current version is ATM.

Ill post here if it resloves the issue.