DL360p gen8 2630 - performance issue

[MSage]

I have two new DL360p gen8 2630 server's that I installed W2012 R2 on.  I updated bios to the latest versions and used intelligent provisioning when I installed the OS.  All was good.  Then I shipped both servers to the datacenters and racked them and connected network.  One server is perfect but the other server has very slow network connectivity.  All four NICs are configured as a Team. I changed the switch ports to auto\auto - no improvement.  Changed the switch ports to 1000 full - no improvement.

Proliant Bios - P71 2/10/2014

iLO v1.51

P420i v 5.42 

I am suspecting this server may have been damaged in shipping - is there a tool that I can run to see motherboard or hardware  health\performnace?

[waaronb]

Have you verified the network activity on the switch side?

Also, sometimes for no reason that I can tell, one of my Gen8 boxes will have an issue with one of the teamed network ports where all 4 show a good gigabit connection, but one of the 4 is having some issues.

To solve that, I disable one NIC at a time until I find the one that was causing problems. If your team is setup to balance the load or as a failover, then this shouldn't affect connectivity.

I'm able to find the bad one and when I re-enable it, it's fine again.

I've only had this happen three times, I think, in the past 6 months, but it has happened, and that's with all the latest Server 2012 R2 drivers, firmware for the BIOS and NIC, etc.

So give that a try... disable one NIC at a time until you hopefully find the one that was the problem.

It might also depend on how your team is setup. Mine are Hyper-V hosts connected to diverse switches, so I'm just using the Hyper-V switching mode. If you're doing switch-assisted trunking or something then it still might have the same symptoms.

Also, if you are doing switch-assisted balancing you might make sure your switch ports are setup correctly.

[MSage]

I disabled 2 of the 4 NIC's in the NIC Team and performance is much better. Is there a tool that can be run to determine if the onboard NIC is failing?

[waaronb]

The Broadcom Advanced Control Suite (BACS) might be useful to check out some advanced stats on the network ports. It should be on the download page for the DL360p Gen8. I don't know if it's included with Intelligent Provisioning installations or as part of the HP SUM updates.

I've only used that once and I don't remember what all it has, but hopefully it's helpful.

Did you try re-enabling those 2 NICs and see if performance was still okay? That worked for me when I had the issue, so I'm not sure if it's some memory leak in the driver perhaps, or maybe something on the server was holding ports open and slowing things down, but having it's network setup change around "fixed" it.

Unfortunately it hasn't happened enough that I can tell what causes it, although I guess that's also a good thing.

[MSage]

I still have two of four NICs disabled.  This morning while copying files to the server the network connectivity disconnected completely.  The server does not ping however via iLO I can log onto windows.  I would think drivers initially but I have another DL360p that is set up exactly the same and it has no issues with 4 NIC's running.

I'll download the software to do a check but I'm also opening a ticket with HP to replace the NIC's.

[MSage]

BACS4 was installed as part of Intergent provisioning.  I opened it up and enabled all four NICs to see statistics while I copy files to the server.  It failed to copy an .iso image - every time - just fails 25% into the job.  "Network Error". 

Statistics are all good with a few "Out of Recv. buffer" on one NIC.

I did ping with -l 1000 and that works.

The NIC is a 331FLR and HP is sending me a replacement.