HPE Community read-only access December 15, 2018
This is a maintenance upgrade. You will be able to read articles and posts, but not post or reply.
Hours:
Dec 15, 4:00 am to 10:00 am UTC
Dec 14, 10:00 pm CST to Dec 15, 4:00 am CST
Dec 14, 8:00 pm PST to Dec 15, 2:00 am PST
Comware Based
cancel
Showing results for 
Search instead for 
Did you mean: 

BFD MAD ports should be excluded from shutdown?

 
Tomas_Kaminskas
Occasional Advisor

BFD MAD ports should be excluded from shutdown?

Hello,

I am trying to run 3 HPE 5510HI switches in IRF with BFD MAD configured. IRF is working fine, when I try to simulate split-brain scenario and shutdown all 10Gb ports on Master device, all other physical 1Gb ports are shut down. Should I MAD exclude 1Gb ports from shutdown, so that devices could establish BFD sessions? It is not clearly stated in any manual, but it seems logical to me.

 

 

13 REPLIES
parnassus
Honored Contributor

Re: BFD MAD ports should be excluded from shutdown?

Hi!

As is explained here it looks so (it's reasonable)...especially reading the statement on the Basic principle of MAD paragraph: "... the other one will shutdown all the local interfaces (except the MAD-excluded interfaces)."

Maybe the command mad exclude interface can help here (reference, as example: HP 5920 & 5900 Switch Series IRF Configuration Guide)

Tomas_Kaminskas
Occasional Advisor

Re: BFD MAD ports should be excluded from shutdown?

Thanks for the reply. Its strange that this is not mentioned in official configuration guides. Will try to test it and will report results here in forum.

parnassus
Honored Contributor

Re: BFD MAD ports should be excluded from shutdown?

Hi Tomas,

It seems mentioned here: HPE FlexNetwork 5510 HI Switch Series IRF Configuration Guide (on Page 30: "Excluding a port from the shutdown action upon detection of multi-active collision").

If I read the reported statement "By default, all ports (except the console and IRF physical interfaces) shut down automatically when the IRF fabric transits to the Recovery state." correctly a question rises: Are BFD-MAD dedicated ports considered not part of the IRF dedicated physical interfaces' group and so they're not automatically excluded from the imposed shutdown?

Tomas_Kaminskas
Occasional Advisor

Re: BFD MAD ports should be excluded from shutdown?

I think after enabling BFD MAD all 10Gb ports automatically were excluded, but not 1Gb ports used for BFD MAD.

Tomas_Kaminskas
Occasional Advisor

Re: BFD MAD ports should be excluded from shutdown?

I cannot stil lconfigure IRF with BFD MAD, I will paste here my setup, and what happens after I disconenct Master from the ring.

Here is my IRF setup:

irf mac-address persistent timer
 irf auto-update enable
 undo irf link-delay
 irf member 1 priority 30
 irf member 2 priority 20
 irf member 3 priority 10

irf-port 1/1
 port group interface Ten-GigabitEthernet1/0/49
 port group interface Ten-GigabitEthernet1/0/50
#
irf-port 1/2
 port group interface Ten-GigabitEthernet1/0/51
 port group interface Ten-GigabitEthernet1/0/52
#
irf-port 2/1
 port group interface Ten-GigabitEthernet2/0/51
 port group interface Ten-GigabitEthernet2/0/52
#
irf-port 2/2
 port group interface Ten-GigabitEthernet2/0/49
 port group interface Ten-GigabitEthernet2/0/50
#
irf-port 3/1
 port group interface Ten-GigabitEthernet3/0/51
 port group interface Ten-GigabitEthernet3/0/52
#
irf-port 3/2
 port group interface Ten-GigabitEthernet3/0/49
 port group interface Ten-GigabitEthernet3/0/50

mad exclude interface GigabitEthernet1/0/1
 mad exclude interface GigabitEthernet1/0/2
 mad exclude interface GigabitEthernet2/0/1
 mad exclude interface GigabitEthernet2/0/2
 mad exclude interface GigabitEthernet3/0/1
 mad exclude interface GigabitEthernet3/0/2

Here is all IRF information when IRF ring is working ok:

<IRF>dis irf
MemberID    Role    Priority  CPU-Mac         Description
 *+1        Master  30        00e0-fc0f-8c02  ---
   2        Standby 20        00e0-fc0f-8c03  ---
   3        Standby 10        00e0-fc0f-8c04  ---
--------------------------------------------------
 * indicates the device is the master.
 + indicates the device through which the user logs in.

 The bridge MAC of the IRF is: 2c23-3aee-60cf
 Auto upgrade                : yes
 Mac persistent              : 6 min
 Domain ID                   : 0

<IRF>dis mad verbose
Multi-active recovery state: No
Excluded ports (user-configured):
  GigabitEthernet1/0/1
  GigabitEthernet1/0/2
  GigabitEthernet2/0/1
  GigabitEthernet2/0/2
  GigabitEthernet3/0/1
  GigabitEthernet3/0/2
Excluded ports (system-configured):
  Ten-GigabitEthernet1/0/49
  Ten-GigabitEthernet1/0/50
  Ten-GigabitEthernet1/0/51
  Ten-GigabitEthernet1/0/52
  Ten-GigabitEthernet2/0/49
  Ten-GigabitEthernet2/0/50
  Ten-GigabitEthernet2/0/51
  Ten-GigabitEthernet2/0/52
  Ten-GigabitEthernet3/0/49
  Ten-GigabitEthernet3/0/50
  Ten-GigabitEthernet3/0/51
  Ten-GigabitEthernet3/0/52
MAD ARP disabled.
MAD ND disabled.
MAD LACP disabled.
MAD BFD enabled interface: Vlan-interface3
  MAD status                 : Normal
  Member ID   MAD IP address       Neighbor   MAD status
  1           192.168.2.1/24       2          Normal
  1           192.168.2.1/24       3          Normal
  2           192.168.2.2/24       1          Normal
  2           192.168.2.2/24       3          Normal
  3           192.168.2.3/24       1          Normal
  3           192.168.2.3/24       2          Normal

<IRF>dis irf topology
                              Topology Info
 -------------------------------------------------------------------------
               IRF-Port1                IRF-Port2
 MemberID    Link       neighbor      Link       neighbor    Belong To
 1           UP         2             UP         3           00e0-fc0f-8c02
 2           UP         3             UP         1           00e0-fc0f-8c02
 3           UP         1             UP         2           00e0-fc0f-8c02

<IRF>dis bfd session
 Total Session Num: 2     Up Session Num: 0     Init Mode: Active

 IPv4 Session Working Under Ctrl Mode:

 LD/RD          SourceAddr      DestAddr        State    Holdtime    Interface
 97/0           192.168.2.1     192.168.2.2     Down        /        Vlan3
 98/0           192.168.2.1     192.168.2.3     Down        /        Vlan3

<IRF>display vlan 3
 VLAN ID: 3
 VLAN type: Static
 Route interface: Configured
 IPv4 address: 192.168.2.1
 IPv4 subnet mask: 255.255.255.0
 Description: VLAN 0003
 Name: VLAN 0003
 Tagged ports:   None
 Untagged ports:
    GigabitEthernet1/0/1          GigabitEthernet1/0/2
    GigabitEthernet2/0/1          GigabitEthernet2/0/2
    GigabitEthernet3/0/1          GigabitEthernet3/0/2

Then I do Split-Brain (by shuting down all Ten gig ports on Master device (ID=1)):

<IRF>dis irf
MemberID    Role    Priority  CPU-Mac         Description
 *+1        Master  30        00e0-fc0f-8c02  ---
--------------------------------------------------
 * indicates the device is the master.
 + indicates the device through which the user logs in.

 The bridge MAC of the IRF is: 2c23-3aee-60cf
 Auto upgrade                : yes
 Mac persistent              : 6 min
 Domain ID                   : 0

<IRF>dis mad verbose
Multi-active recovery state: No
Excluded ports (user-configured):
  GigabitEthernet1/0/1
  GigabitEthernet1/0/2
Excluded ports (system-configured):
  Ten-GigabitEthernet1/0/49
  Ten-GigabitEthernet1/0/50
  Ten-GigabitEthernet1/0/51
  Ten-GigabitEthernet1/0/52
MAD ARP disabled.
MAD ND disabled.
MAD LACP disabled.
MAD BFD enabled interface: Vlan-interface3
  MAD status                 : Faulty
  Member ID   MAD IP address       Neighbor   MAD status
  1           192.168.2.1/24       2          Normal
  1           192.168.2.1/24       3          Faulty

<IRF>dis irf topology
                              Topology Info
 -------------------------------------------------------------------------
               IRF-Port1                IRF-Port2
 MemberID    Link       neighbor      Link       neighbor    Belong To
 1           DOWN       ---           DOWN       ---         00e0-fc0f-8c02

<IRF>dis bfd session
 Total Session Num: 2     Up Session Num: 0     Init Mode: Active

 IPv4 Session Working Under Ctrl Mode:

 LD/RD          SourceAddr      DestAddr        State    Holdtime    Interface
 97/97          192.168.2.1     192.168.2.2     Init     3800ms      Vlan3
 98/0           192.168.2.1     192.168.2.3     Down        /        Vlan3

Session with SRC 192.168.2.1 and DST 192.168.2.2 is constantly changing from Init to Down to Init.

Session with SRC 192.168.2.1 and DST 192.168.2.3 is constantly Down.

What is happening there?

Tomas_Kaminskas
Occasional Advisor

Re: BFD MAD ports should be excluded from shutdown?

In case someone is facing the same issue, I have found this document:

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_sf-EN_US000005491

And in their configuration, they have:

<Rack4sw1>display current-configuration | include exclude
mad exclude interface GigabitEthernet1/4/0/1
mad exclude interface GigabitEthernet2/4/0/4
mad exclude interface Vlan-interface100
 
I havent seen this in any other guides. I have tried not only excluding MAD BFD ports from shutdown but also Vlan-interface which is used for MAD.
Now I have one BFD session up, but second one is still down:
[IRF]dis mad verbose
Multi-active recovery state: No
Excluded ports (user-configured):
GigabitEthernet1/0/1
GigabitEthernet1/0/2
Vlan-interface3
Excluded ports (system-configured):
Ten-GigabitEthernet1/0/49
Ten-GigabitEthernet1/0/50
Ten-GigabitEthernet1/0/51
Ten-GigabitEthernet1/0/52
MAD ARP disabled.
MAD ND disabled.
MAD LACP disabled.
MAD BFD enabled interface: Vlan-interface3
MAD status : Faulty
Member ID MAD IP address Neighbor MAD status
1 192.168.2.1/24 2 Normal
1 192.168.2.1/24 3 Faulty

[IRF]dis bfd session
Total Session Num: 2 Up Session Num: 1 Init Mode: Active

IPv4 Session Working Under Ctrl Mode:

LD/RD SourceAddr DestAddr State Holdtime Interface
97/97 192.168.2.1 192.168.2.2 Up 1384ms Vlan3
98/0 192.168.2.1 192.168.2.3 Down / Vlan3
 
Any ideas why second session might be down?
Tomas_Kaminskas
Occasional Advisor

Re: BFD MAD ports should be excluded from shutdown?

UPDATE:

I have loged in to Devices 2 and 3 via console and both of them are constantly giving these log messages:

%Jan 1 05:39:25:306 2013 IRF BFD/5/BFD_CHANGE_FSM: Sess[192.168.2.2/192.168.2.1, LD/RD:97/97, Interface:Vlan3, SessType:Ctrl, LinkType:INET], Sta: UP->DOWN, Diag: 3
%Jan 1 05:39:25:413 2013 IRF BFD/5/BFD_CHANGE_FSM: Sess[192.168.2.2/192.168.2.1, LD/RD:97/97, Interface:Vlan3, SessType:Ctrl, LinkType:INET], Sta: DOWN->INIT, Diag: 0
%Jan 1 05:39:25:414 2013 IRF BFD/5/BFD_CHANGE_FSM: Sess[192.168.2.2/192.168.2.1, LD/RD:97/97, Interface:Vlan3, SessType:Ctrl, LinkType:INET], Sta: INIT->UP, Diag: 0
%Jan 1 05:39:25:414 2013 IRF DEV/1/MAD_DETECT: Multi-active devices detected, please fix it.

 

parnassus
Honored Contributor

Re: BFD MAD ports should be excluded from shutdown?

Wait.

Initially you wrote "I am trying to run 3 HPE 5510HI switches in IRF with BFD MAD configured": does your IRF scenario is a 3 chassis IRF fabrics (without any Intermediate Device) or a 2 chassis IRF fabrics (with the third Switch acting as the Intermediate Device)?

From the dis irf topology result it is a 3 chassis IRF fabrics (three members).

For a 3 chassis IRF fabrics I mean there is a Master and two Subordinates (connected with a Daisy-chain or a Ring topology).

Isn't that BFD MAD works only in a 2 chassis IRF fabric scenario with or withouth an Intermediate Device and, as soon as you open up to a 3 (or 4) chassis IRF fabric scenario you're forced to use an Intermediate Device?

bakerbas
Advisor

Re: BFD MAD ports should be excluded from shutdown?

Yes - for more than 2 switches in IRF fabric, using BFD MAD, you need to use an intermediate device, and disable STP on the BFD MAD interfaces.

The IRF configuration guide should have a configuiration examples section titled - "BFD MAD-enabled IRF configuration example".

Please try to emulate what is shown there.

 

Tomas_Kaminskas
Occasional Advisor

Re: BFD MAD ports should be excluded from shutdown?

I am a bit confused. Actually there is a configuration example showing more than 2 devices connected in a ring topology with full mesh BFD MAD connections:

http://www.h3c.com.hk/Technical_Support___Documents/Technical_Documents/Switches/H3C_S5800_Series_Switches/Configuration/Typical_Configuration_Example/H3C_S5820X[S5800]_Configuration_Examples-6W100/#_Ref270520212

parnassus
Honored Contributor

Re: BFD MAD ports should be excluded from shutdown?

I must to admit that BFD MAD requirements and restrictions seem to vary reading documentation (e.g. HPE FlexNetwork 10500 Switch Series versus HPE FlexNetwork 5510 HI Switch Series).

For the HPE FlexNetwork 5510 HI Switch Series, in contrast to what is reported for the bigger HPE FlexNetwork 10500 Switch Series in its IRF Configuration Guide, there is not a clear statement about the Intermediate Device requirement when IRF members are more than two (exactly as it happens reading the H3C documentation reported above which doesn't report that requirement).

See this:

MAD_Mechanisms_BFD_MAD_on_HPE_FlexNetwork_5510_HI_Switch_Series_IRF_Configuration_Guide.png

taken from the MAD Mechanisms paragraph of the IRF Configuration Guide for the HPE FlexNetwork 5510 HI Switch Series.

When using the common Ethernet ports...notice the statement: "Each member device has a BFD MAD link to an intermediate device, or all member devices have a BFD MAD link to each other." which admits a Mesh configuration ("all to each other") if the Intermediate Device is not available...at least as I understand it.

Last minute edit:

Among BFD MAD Disadvantages are reported:

  • If no intermediate device is used, any two IRF members must have a BFD MAD link to each other.
  • If an intermediate device is used, every IRF member must have a BFD MAD link to the intermediate device.

So it seems that the magic of two IRF members' threshold associated with the presence/absence of an Intermediate Device is there.

Mike_ES
Valued Contributor

Re: BFD MAD ports should be excluded from shutdown?

Hi,

I think for MAD types we should concern what H3C site presenting (reference screen-shoot attached), but:

1. If you have daisy chain topology (min. 3 IRF nodes) , you cannot use LACP MAD without intermediate switch.

2. Also you cannot use BFD MAD with daisy chain , because not having full mesh (BFD sessions between IRF nodes).

So, for 3 IRF nodes you can use LACP MAD with intermediate switch OR you can setup/change to IRF ring topology and having BFD MAD possible.

Michal

Tomas_Kaminskas
Occasional Advisor

Re: BFD MAD ports should be excluded from shutdown?

My setup is IRF ring topology and I cannot understand why one bfd link is UP and another one is DOWN:

[IRF]dis bfd session
Total Session Num: 2 Up Session Num: 1 Init Mode: Active

IPv4 Session Working Under Ctrl Mode:

LD/RD SourceAddr DestAddr State Holdtime Interface
97/97 192.168.2.1 192.168.2.2 Up 1384ms Vlan3
98/0 192.168.2.1 192.168.2.3 Down / Vlan3