530T NICs not coming online when cable removed and reinserted

[Rob Leadbeater]

We've recently come across an issue where the network link on a 530T (10Gbps CNA) NIC in a DL580 Gen 9, wouldn't come back online after the cable was removed and reinserted.

• Server: DL580 Gen 9
• NIC: 2 x HP 530T (Broadcom NetXtreme II 57810)
• Switch: Cisco C3750X
• OS: RHEL 6.7  Stock Red Hat drivers in use.
• NIC Bonding: Port 1 of each NIC bonded together to create bond1 in Active-Backup mode.

At the start of the test, bond1 is running fine, with the first NIC (eth4) active at 1Gbps as expected.

When the cable connecting the NIC to the switch is removed, the bond fails over correctly to the second NIC (eth6).

When the cable is reinserted, no link is detected on the first NIC, so it doesn't rejoin the bond.

The link could be made to come up by modifying the switchport configuration to be Fixed 1000/Full and then back to Auto/Auto, which suggests some form of speed negotiation issue with the Cisco switch.  The other ports on the two cards are connected to a Juniper switch and seem to be working fine.

Has anyone come across anything similar before ?