NICs flapping after firmware upgrade (3.18 to 3.7)

[RameshSA]

All Experts,

I just notified by my incident team that some of the blades in an enclosure are reporting NIC flapping. Upon checking further i noticed that all the blades in a particular rack are having NIC flapping issue. The current setup is -

     3 enclosures in a Rack stacked as recommended by HP and two uplinks. Here is the stack link info

                      Link  Speed  Connected From  Connected To
===========================================
1     10Gb   enc0:1:X1       enc2:5:X2
2     10Gb   enc0:1:X7       enc0:2:X7
3     10Gb   enc0:2:X1       enc0:4:X1
4     10Gb   enc0:3:X1       enc0:5:X1
5     10Gb   enc0:3:X7       enc0:4:X7
6     10Gb   enc0:5:X2       enc1:1:X2
7     10Gb   enc1:1:X7       enc1:2:X7
8     10Gb   enc1:2:X1       enc1:4:X1
9     10Gb   enc1:3:X1       enc1:5:X1
10    10Gb   enc1:3:X7       enc1:4:X7
11    10Gb   enc1:5:X2       enc2:1:X2
12    10Gb   enc2:2:X1       enc2:4:X1
13    10Gb   enc2:2:X7       enc2:1:X7
14    10Gb   enc2:3:X1       enc2:5:X1
15    10Gb   enc2:3:X7       enc2:4:X7

I checked few blades and here are the nic details.  Could you check below details and let me know what could be the problem?

enc0 - Blade 5 - 2 NICs that are connected to Bay 4 and Bay5 are flapping

enc0 - Balade 10 - 2 NICs that are connected to Bay2 and Ba5 are flapping

enc1 - Blade2 - 2 NICs that are connected to Bay1 and Bay5 are flapping

enc0 - Blade3 - 2 NICs that are connected to Bay1 and Bay5 are flapping

here is the uplink info

========================
Name          Status
========================
Lnk_Backup  OK
Lnk_Pri     OK
Lnk_Alt     Degraded

Lnk_Alt

External Uplink Ports
=================================================
ID         Status                Type     Speed
=================================================
enc1:1:X5  Unpopulated           absent   Auto
-------------------------------------------------
enc1:4:X6  Linked (Active)       SFP-DAC  Auto
           (10Gb)
-------------------------------------------------

Lnk_Pri

External Uplink Ports
=================================================
ID         Status                Type     Speed
=================================================
enc0:1:X5  Linked (Active)       SFP-DAC  Auto
           (10Gb)
-------------------------------------------------

External Uplink Ports
==================================================
ID         Status                Type      Speed
==================================================
enc2:5:X5  Linked (Active)       SFP-RJ45  Auto
           (1Gb)
--------------------------------------------------
enc2:5:X6  Linked (Active)       SFP-RJ45  Auto
           (1Gb)
--------------------------------------------------
enc2:5:X7  Linked (Active)       SFP-RJ45  Auto
           (1Gb)
--------------------------------------------------
enc2:5:X8  Linked (Active)       SFP-RJ45  Auto
           (1Gb)
--------------------------------------------------

[Michael Leu]

Perhaps this advisory applies to your situation?

HP Virtual Connect- Servers Transmitting Excessive Pause Frames May Cause Virtual Connect Uplink Port To Experience Channel Flapping, Stacking Link Removal and Server Communication Loss

[Hongjun Ma]

Usually NIC flapping is related with NIC firmware and driver. What's the version for NIC fw and driver and OS. Could you try to upgrade to latest version s of NIC fw/driver for one blade to see? This shouldn't have anything to do with vc uplink and stacking links.

[Hongjun Ma]

Also root cause of the advisory is also bad NIC driver and firmware.vc is victim

[RameshSA]

VC firmware is upgraded to 3.7 on 22 Feb. All the blades are on latest NIC driver. How to check which blade is making others to flap? Also, there are no pause frame errors.

[RameshSA]

enc2:Bay1 VC module has been powered-off. There is no NIC flapping now. Suspecting either VC or stacking cable.