HPE 10504 IRF HEARTBEAT ERROR

[AjinS]

I have created IRF on two brand new HPE 10504 switches and the two switches formed IRF fabric. But its showing heartbeat timed out error. The IRF port go down and after few seconds it comes back up. It keeps on going on in cycles every 30 seconds. Below are the IRF error messages displaying on the console.

%Feb 8 21:26:23:963 2020 HPE STM/2/STM_LINK_TIMEOUT: -MDC=1; IRF port 2 went down because the heartbeat timed out.
%Feb 8 21:26:23:979 2020 HPE STM/3/STM_LINK_DOWN: -MDC=1; IRF port 2 went down.
%Feb 8 21:26:24:098 2020 HPE STM/2/STM_LINK_TIMEOUT: -MDC=1-Chassis=2-Slot=0; IRF port 1 went down because the heartbeat timed out.
%Feb 8 21:26:24:153 2020 HPE STM/3/STM_LINK_DOWN: -MDC=1-Chassis=2-Slot=0; IRF port 1 went down.
%Feb 8 21:26:27:400 2020 HPE IFNET/3/PHY_UPDOWN: -MDC=1; Physical state on the interface Ten-GigabitEthernet2/2/0/45 changed to down.
%Feb 8 21:26:27:401 2020 HPE IFNET/5/LINK_UPDOWN: -MDC=1; Line protocol state on the interface Ten-GigabitEthernet2/2/0/45 changed to down.
%Feb 8 21:26:27:410 2020 HPE IFNET/3/PHY_UPDOWN: -MDC=1; Physical state on the interface Ten-GigabitEthernet2/2/0/46 changed to down.
%Feb 8 21:26:27:411 2020 HPE IFNET/5/LINK_UPDOWN: -MDC=1; Line protocol state on the interface Ten-GigabitEthernet2/2/0/46 changed to down.
%Feb 8 21:26:27:436 2020 HPE STM/3/STM_LINK_DOWN: -MDC=1-Chassis=2-Slot=0; IRF port 1 went down.
%Feb 8 21:26:27:663 2020 HPE IFNET/3/PHY_UPDOWN: -MDC=1; Physical state on the interface Ten-GigabitEthernet1/2/0/47 changed to down.
%Feb 8 21:26:27:664 2020 HPE IFNET/5/LINK_UPDOWN: -MDC=1; Line protocol state on the interface Ten-GigabitEthernet1/2/0/47 changed to down.
%Feb 8 21:26:27:676 2020 HPE IFNET/3/PHY_UPDOWN: -MDC=1; Physical state on the interface Ten-GigabitEthernet1/2/0/48 changed to down.
%Feb 8 21:26:27:676 2020 HPE IFNET/5/LINK_UPDOWN: -MDC=1; Line protocol state on the interface Ten-GigabitEthernet1/2/0/48 changed to down.
%Feb 8 21:26:27:733 2020 HPE STM/3/STM_LINK_DOWN: -MDC=1; IRF port 2 went down.
%Feb 8 21:26:28:369 2020 HPE IFNET/3/PHY_UPDOWN: -MDC=1; Physical state on the interface Ten-GigabitEthernet2/2/0/45 changed to up.
%Feb 8 21:26:28:370 2020 HPE IFNET/5/LINK_UPDOWN: -MDC=1; Line protocol state on the interface Ten-GigabitEthernet2/2/0/45 changed to up.
%Feb 8 21:26:28:378 2020 HPE IFNET/3/PHY_UPDOWN: -MDC=1; Physical state on the interface Ten-GigabitEthernet2/2/0/46 changed to up.
%Feb 8 21:26:28:379 2020 HPE IFNET/5/LINK_UPDOWN: -MDC=1; Line protocol state on the interface Ten-GigabitEthernet2/2/0/46 changed to up.
%Feb 8 21:26:28:504 2020 HPE IFNET/3/PHY_UPDOWN: -MDC=1; Physical state on the interface Ten-GigabitEthernet1/2/0/47 changed to up.
%Feb 8 21:26:28:504 2020 HPE IFNET/5/LINK_UPDOWN: -MDC=1; Line protocol state on the interface Ten-GigabitEthernet1/2/0/47 changed to up.
%Feb 8 21:26:28:517 2020 HPE IFNET/3/PHY_UPDOWN: -MDC=1; Physical state on the interface Ten-GigabitEthernet1/2/0/48 changed to up.
%Feb 8 21:26:28:518 2020 HPE IFNET/5/LINK_UPDOWN: -MDC=1; Line protocol state on the interface Ten-GigabitEthernet1/2/0/48 changed to up.
%Feb 8 21:26:29:391 2020 HPE STM/6/STM_LINK_UP: -MDC=1-Chassis=2-Slot=0; IRF port 1 came up.
%Feb 8 21:26:29:692 2020 HPE STM/6/STM_LINK_UP: -MDC=1; IRF port 2 came up.

The same error comes every 30-35 seconds.

[HPE]displ irf
MemberID  Slot  Role       Priority   CPU-Mac             Description
*+1               0      Master   32           00e0-fc0a-15e0    ---
    1               1      Standby 32           00e0-fc0f-8c02    ---
    2              0     Standby 30           00e0-fc0f-8c13     ---
    2              1      Standby 30           00e0-fc0f-8c14    ---
--------------------------------------------------
* indicates the device is the master.
+ indicates the device through which the user logs in.

The bridge MAC of the IRF is: 4cae-a381-0600
Auto upgrade : yes
Mac persistent : always
Domain ID : 0
Auto merge : yes
IRF mode : normal

[HPE]disp irf configuration
MemberID    NewID       IRF-Port1                                      IRF-Port2
1                        1             Ten-GigabitEthernet1/2/0/45    Ten-GigabitEthernet1/2/0/47
                                        Ten-GigabitEthernet1/2/0/46    Ten-GigabitEthernet1/2/0/48
2                       2             Ten-GigabitEthernet2/2/0/45    Ten-GigabitEthernet2/2/0/47
                                        Ten-GigabitEthernet2/2/0/46    Ten-GigabitEthernet2/2/0/48

[HPE]dis irf topology
                                     Topology Info
-------------------------------------------------------------------------
                     IRF-Port1                         IRF-Port2
MemberID  Link      neighbor       Link     neighbor       Belong To
1                  UP        2                     UP       2                   00e0-fc0a-15e0
2                 UP         1                     UP       1                   00e0-fc0a-15e0

[HPE]dis irf topology
                                     Topology Info
-------------------------------------------------------------------------
                     IRF-Port1                              IRF-Port2
MemberID   Link             neighbor       Link             neighbor        Belong To
2                  TIMEOUT     ---                 UP               1                      00e0-fc0a-15e0
1                   UP                2                    TIMEOUT    ---                  00e0-fc0a-15e0

What might be the issue and how to resolve it?

Thanks

[parnassus]

Was the IRF port configuration activated on your IRF? in other words did you execute the irf-port-configuration active CLI command to activate IRF Ports?

Please verify you followed step by step all required steps described on HPE FlexNetwork 10500 Switch Series - Configuration Guides (20/10/2017 valid starting from R7557P01) (inside there is the Virtual Technologies Configuration Guide PDF especially for IRF deployment) for correctly setting up an IRF Domain.

Your one is a IRF Ring Topology where IRF Port 1 on IRF Member 1 is connected to IRF Port 2 on IRF Member 2 and IRF Port 1 on IRF Member 2 is connected back to IRF Port 2 on IRF Member 1...thus forming a Ring (1-2-1) instead of the classical Chain (1-2) usually found on a two-only members IRF setup.

Could you also post a sanitized running configuration and the output of display version and display irf configuration CLI commands?

[VoIP-Buddy · Accepted solution]

While you can technically create a ring with 2 10500 switches, or for that matter any two like model Comware switches, we do not recommend doing that.  What is the point of setting up a ring in this setup?  None, really.

What I would recommend is to configure the IRF-Ports such that you have the 1 - 2 setup that Parnassus suggested and with 4 connections, you'll have 40 Gb throughput, if you need it.  So you would have:

IRF-Port 1 - 1/2/0/45, 1/2/0/46, 1/2/0/47, and 1/2/0/48

IRF-Port 2 - 2/2/0/45, 2/2/0/46, 2/2/0/47, and 2/2/0/48

If you have the ports and can afford to do so, I would fan out the 4 ports onto an additional card so that you don't lose the IRF if the 1/2 or 2/2 cards fail for any reason.

Depending on the line cards you have, you have to be careful.  some cards use 4-port banks and all 4 ports would need to be used as IRF ports even if they are not connected to the IRF currently.  Plan wisely.

Regards,

David

[parnassus]

Exactly, this was explained on the SUPPORT COMMUNICATION - CUSTOMER ADVISORY c04706976 "HP Networking - Intelligent Resilient Framework (IRF) Ring Topology for Two Devices in IRF Fabric" last updated during mid 2018.

Before citing the advisory above as a possible suggestion to solve his issue I wanted to be sure about its IRF configuration...well, it's clear that is using an unsupported IRF topology between his two HPE 10504.

[VoIP-Buddy]

Hi Parnassus!

I wouldn't go that far.  There are lots of customers that have configured IRF as a ring with two switches and it works fine.  I'd characterize this as a "Best Practice" to not use a ring with two switches. 

When I talk to customers with this kind of scenario, I always let them know that a ring is not advised for two switches.

Regards,

David

[AjinS]

Reconfiguring IRF with Daisy Chain topology solved the issue. The issue was with ring topology as its not supported with the switch. Thank you.