Comware Based
1748165 Members
3888 Online
108758 Solutions
New Discussion юеВ

BFD MAD ports should be excluded from shutdown?

 
Tomas_Kaminskas
Occasional Advisor

BFD MAD ports should be excluded from shutdown?

Hello,

I am trying to run 3 HPE 5510HI switches in IRF with BFD MAD configured. IRF is working fine, when I try to simulate split-brain scenario and shutdown all 10Gb ports on Master device, all other physical 1Gb ports are shut down. Should I MAD exclude 1Gb ports from shutdown, so that devices could establish BFD sessions? It is not clearly stated in any manual, but it seems logical to me.

 

 

13 REPLIES 13
parnassus
Honored Contributor

Re: BFD MAD ports should be excluded from shutdown?

Hi!

As is explained here it looks so (it's reasonable)...especially reading the statement on the Basic principle of MAD paragraph: "... the other one will shutdown all the local interfaces (except the MAD-excluded interfaces)."

Maybe the command mad exclude interface can help here (reference, as example: HP 5920 & 5900 Switch Series IRF Configuration Guide)


I'm not an HPE Employee
Kudos and Accepted Solution banner
Tomas_Kaminskas
Occasional Advisor

Re: BFD MAD ports should be excluded from shutdown?

Thanks for the reply. Its strange that this is not mentioned in official configuration guides. Will try to test it and will report results here in forum.

parnassus
Honored Contributor

Re: BFD MAD ports should be excluded from shutdown?

Hi Tomas,

It seems mentioned here: HPE FlexNetwork 5510 HI Switch Series IRF Configuration Guide (on Page 30: "Excluding a port from the shutdown action upon detection of multi-active collision").

If I read the reported statement "By default, all ports (except the console and IRF physical interfaces) shut down automatically when the IRF fabric transits to the Recovery state." correctly a question rises: Are BFD-MAD dedicated ports considered not part of the IRF dedicated physical interfaces' group and so they're not automatically excluded from the imposed shutdown?


I'm not an HPE Employee
Kudos and Accepted Solution banner
Tomas_Kaminskas
Occasional Advisor

Re: BFD MAD ports should be excluded from shutdown?

I think after enabling BFD MAD all 10Gb ports automatically were excluded, but not 1Gb ports used for BFD MAD.

Tomas_Kaminskas
Occasional Advisor

Re: BFD MAD ports should be excluded from shutdown?

I cannot stil lconfigure IRF with BFD MAD, I will paste here my setup, and what happens after I disconenct Master from the ring.

Here is my IRF setup:

irf mac-address persistent timer
 irf auto-update enable
 undo irf link-delay
 irf member 1 priority 30
 irf member 2 priority 20
 irf member 3 priority 10

irf-port 1/1
 port group interface Ten-GigabitEthernet1/0/49
 port group interface Ten-GigabitEthernet1/0/50
#
irf-port 1/2
 port group interface Ten-GigabitEthernet1/0/51
 port group interface Ten-GigabitEthernet1/0/52
#
irf-port 2/1
 port group interface Ten-GigabitEthernet2/0/51
 port group interface Ten-GigabitEthernet2/0/52
#
irf-port 2/2
 port group interface Ten-GigabitEthernet2/0/49
 port group interface Ten-GigabitEthernet2/0/50
#
irf-port 3/1
 port group interface Ten-GigabitEthernet3/0/51
 port group interface Ten-GigabitEthernet3/0/52
#
irf-port 3/2
 port group interface Ten-GigabitEthernet3/0/49
 port group interface Ten-GigabitEthernet3/0/50

mad exclude interface GigabitEthernet1/0/1
 mad exclude interface GigabitEthernet1/0/2
 mad exclude interface GigabitEthernet2/0/1
 mad exclude interface GigabitEthernet2/0/2
 mad exclude interface GigabitEthernet3/0/1
 mad exclude interface GigabitEthernet3/0/2

Here is all IRF information when IRF ring is working ok:

<IRF>dis irf
MemberID    Role    Priority  CPU-Mac         Description
 *+1        Master  30        00e0-fc0f-8c02  ---
   2        Standby 20        00e0-fc0f-8c03  ---
   3        Standby 10        00e0-fc0f-8c04  ---
--------------------------------------------------
 * indicates the device is the master.
 + indicates the device through which the user logs in.

 The bridge MAC of the IRF is: 2c23-3aee-60cf
 Auto upgrade                : yes
 Mac persistent              : 6 min
 Domain ID                   : 0

<IRF>dis mad verbose
Multi-active recovery state: No
Excluded ports (user-configured):
  GigabitEthernet1/0/1
  GigabitEthernet1/0/2
  GigabitEthernet2/0/1
  GigabitEthernet2/0/2
  GigabitEthernet3/0/1
  GigabitEthernet3/0/2
Excluded ports (system-configured):
  Ten-GigabitEthernet1/0/49
  Ten-GigabitEthernet1/0/50
  Ten-GigabitEthernet1/0/51
  Ten-GigabitEthernet1/0/52
  Ten-GigabitEthernet2/0/49
  Ten-GigabitEthernet2/0/50
  Ten-GigabitEthernet2/0/51
  Ten-GigabitEthernet2/0/52
  Ten-GigabitEthernet3/0/49
  Ten-GigabitEthernet3/0/50
  Ten-GigabitEthernet3/0/51
  Ten-GigabitEthernet3/0/52
MAD ARP disabled.
MAD ND disabled.
MAD LACP disabled.
MAD BFD enabled interface: Vlan-interface3
  MAD status                 : Normal
  Member ID   MAD IP address       Neighbor   MAD status
  1           192.168.2.1/24       2          Normal
  1           192.168.2.1/24       3          Normal
  2           192.168.2.2/24       1          Normal
  2           192.168.2.2/24       3          Normal
  3           192.168.2.3/24       1          Normal
  3           192.168.2.3/24       2          Normal

<IRF>dis irf topology
                              Topology Info
 -------------------------------------------------------------------------
               IRF-Port1                IRF-Port2
 MemberID    Link       neighbor      Link       neighbor    Belong To
 1           UP         2             UP         3           00e0-fc0f-8c02
 2           UP         3             UP         1           00e0-fc0f-8c02
 3           UP         1             UP         2           00e0-fc0f-8c02

<IRF>dis bfd session
 Total Session Num: 2     Up Session Num: 0     Init Mode: Active

 IPv4 Session Working Under Ctrl Mode:

 LD/RD          SourceAddr      DestAddr        State    Holdtime    Interface
 97/0           192.168.2.1     192.168.2.2     Down        /        Vlan3
 98/0           192.168.2.1     192.168.2.3     Down        /        Vlan3

<IRF>display vlan 3
 VLAN ID: 3
 VLAN type: Static
 Route interface: Configured
 IPv4 address: 192.168.2.1
 IPv4 subnet mask: 255.255.255.0
 Description: VLAN 0003
 Name: VLAN 0003
 Tagged ports:   None
 Untagged ports:
    GigabitEthernet1/0/1          GigabitEthernet1/0/2
    GigabitEthernet2/0/1          GigabitEthernet2/0/2
    GigabitEthernet3/0/1          GigabitEthernet3/0/2

Then I do Split-Brain (by shuting down all Ten gig ports on Master device (ID=1)):

<IRF>dis irf
MemberID    Role    Priority  CPU-Mac         Description
 *+1        Master  30        00e0-fc0f-8c02  ---
--------------------------------------------------
 * indicates the device is the master.
 + indicates the device through which the user logs in.

 The bridge MAC of the IRF is: 2c23-3aee-60cf
 Auto upgrade                : yes
 Mac persistent              : 6 min
 Domain ID                   : 0

<IRF>dis mad verbose
Multi-active recovery state: No
Excluded ports (user-configured):
  GigabitEthernet1/0/1
  GigabitEthernet1/0/2
Excluded ports (system-configured):
  Ten-GigabitEthernet1/0/49
  Ten-GigabitEthernet1/0/50
  Ten-GigabitEthernet1/0/51
  Ten-GigabitEthernet1/0/52
MAD ARP disabled.
MAD ND disabled.
MAD LACP disabled.
MAD BFD enabled interface: Vlan-interface3
  MAD status                 : Faulty
  Member ID   MAD IP address       Neighbor   MAD status
  1           192.168.2.1/24       2          Normal
  1           192.168.2.1/24       3          Faulty

<IRF>dis irf topology
                              Topology Info
 -------------------------------------------------------------------------
               IRF-Port1                IRF-Port2
 MemberID    Link       neighbor      Link       neighbor    Belong To
 1           DOWN       ---           DOWN       ---         00e0-fc0f-8c02

<IRF>dis bfd session
 Total Session Num: 2     Up Session Num: 0     Init Mode: Active

 IPv4 Session Working Under Ctrl Mode:

 LD/RD          SourceAddr      DestAddr        State    Holdtime    Interface
 97/97          192.168.2.1     192.168.2.2     Init     3800ms      Vlan3
 98/0           192.168.2.1     192.168.2.3     Down        /        Vlan3

Session with SRC 192.168.2.1 and DST 192.168.2.2 is constantly changing from Init to Down to Init.

Session with SRC 192.168.2.1 and DST 192.168.2.3 is constantly Down.

What is happening there?

Tomas_Kaminskas
Occasional Advisor

Re: BFD MAD ports should be excluded from shutdown?

In case someone is facing the same issue, I have found this document:

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_sf-EN_US000005491

And in their configuration, they have:

<Rack4sw1>display current-configuration | include exclude
mad exclude interface GigabitEthernet1/4/0/1
mad exclude interface GigabitEthernet2/4/0/4
mad exclude interface Vlan-interface100
 
I havent seen this in any other guides. I have tried not only excluding MAD BFD ports from shutdown but also Vlan-interface which is used for MAD.
Now I have one BFD session up, but second one is still down:
[IRF]dis mad verbose
Multi-active recovery state: No
Excluded ports (user-configured):
GigabitEthernet1/0/1
GigabitEthernet1/0/2
Vlan-interface3
Excluded ports (system-configured):
Ten-GigabitEthernet1/0/49
Ten-GigabitEthernet1/0/50
Ten-GigabitEthernet1/0/51
Ten-GigabitEthernet1/0/52
MAD ARP disabled.
MAD ND disabled.
MAD LACP disabled.
MAD BFD enabled interface: Vlan-interface3
MAD status : Faulty
Member ID MAD IP address Neighbor MAD status
1 192.168.2.1/24 2 Normal
1 192.168.2.1/24 3 Faulty

[IRF]dis bfd session
Total Session Num: 2 Up Session Num: 1 Init Mode: Active

IPv4 Session Working Under Ctrl Mode:

LD/RD SourceAddr DestAddr State Holdtime Interface
97/97 192.168.2.1 192.168.2.2 Up 1384ms Vlan3
98/0 192.168.2.1 192.168.2.3 Down / Vlan3
 
Any ideas why second session might be down?
Tomas_Kaminskas
Occasional Advisor

Re: BFD MAD ports should be excluded from shutdown?

UPDATE:

I have loged in to Devices 2 and 3 via console and both of them are constantly giving these log messages:

%Jan 1 05:39:25:306 2013 IRF BFD/5/BFD_CHANGE_FSM: Sess[192.168.2.2/192.168.2.1, LD/RD:97/97, Interface:Vlan3, SessType:Ctrl, LinkType:INET], Sta: UP->DOWN, Diag: 3
%Jan 1 05:39:25:413 2013 IRF BFD/5/BFD_CHANGE_FSM: Sess[192.168.2.2/192.168.2.1, LD/RD:97/97, Interface:Vlan3, SessType:Ctrl, LinkType:INET], Sta: DOWN->INIT, Diag: 0
%Jan 1 05:39:25:414 2013 IRF BFD/5/BFD_CHANGE_FSM: Sess[192.168.2.2/192.168.2.1, LD/RD:97/97, Interface:Vlan3, SessType:Ctrl, LinkType:INET], Sta: INIT->UP, Diag: 0
%Jan 1 05:39:25:414 2013 IRF DEV/1/MAD_DETECT: Multi-active devices detected, please fix it.

 

parnassus
Honored Contributor

Re: BFD MAD ports should be excluded from shutdown?

Wait.

Initially you wrote "I am trying to run 3 HPE 5510HI switches in IRF with BFD MAD configured": does your IRF scenario is a 3 chassis IRF fabrics (without any Intermediate Device) or a 2 chassis IRF fabrics (with the third Switch acting as the Intermediate Device)?

From the dis irf topology result it is a 3 chassis IRF fabrics (three members).

For a 3 chassis IRF fabrics I mean there is a Master and two Subordinates (connected with a Daisy-chain or a Ring topology).

Isn't that BFD MAD works only in a 2 chassis IRF fabric scenario with or withouth an Intermediate Device and, as soon as you open up to a 3 (or 4) chassis IRF fabric scenario you're forced to use an Intermediate Device?


I'm not an HPE Employee
Kudos and Accepted Solution banner
bakerbas
Advisor

Re: BFD MAD ports should be excluded from shutdown?

Yes - for more than 2 switches in IRF fabric, using BFD MAD, you need to use an intermediate device, and disable STP on the BFD MAD interfaces.

The IRF configuration guide should have a configuiration examples section titled - "BFD MAD-enabled IRF configuration example".

Please try to emulate what is shown there.