HPE 7900 CORE SWITCH upgrade

[hemh]

Hi experts,

We are about to upgrade our 7900 flexfabric chassis(2 members) for the first time. The actual firmware don't allow us to use ISSU for the upgrade. So, we planned to disable BFD-MAD and to upgrade one of the 2 chassis, so the one that won't be upgraded will remain master in the old firmaware version. 

Can you tell me waht will happen when the upgraded chassis will comme UP? Fro shure the stack won't be abble to form due to firmware version mismatch, but wil we have two masters in the same time? running 2 different versions?

Thanks for your answer

[jmpk]

Hello, 

Since the switches are in IRF , you need to move the firmware to both chassis and set boot loader on both chassis as well.

Then you need to reboot , which make both switch reboot and once it back online it will join in IRF . I suggest you refer below link for similiar upgrade process.

https://support.hpe.com/hpesc/public/docDisplay?docId=kc0129233en_us&docLocale=en_US

Note: You need to take down time to perform the update, when you upgrade switch individually it will get into split brain scenario . So suggest you to set boot-loader on both chassis and reboot the switch. 

[Ivan_B]

Hello @hemh!

If you can afford 10-15 minutes downtime, I recommend you to proceed as per @jmpk instructions - it is the simplest process, no brainer. However, if you want to minimize the downtime, then you will need to play a little bit with the upgrade process.

The problem with your plan is that if you disable BFD MAD and the second chassis will boot with new s/w version, it will become Master. So you will end up with two Masters. It is so called split-brain scenario, very unpleasant situation that will unpredictably break your network - either some or all traffic will be affected, especially the routed flows. 

In order to minimize the impact you can follow a quite simple procedure:

1. Keep BFD MAD. It will be your protection against the split-brain.
2. Using the 'boot-loader file' command set the NEW version as the main image on BOTH Chassis. It will be used during the next bootup.
3. Save the configuration.
4. Reboot Chassis 2. It should reboot using the new image. After the boot it will try to join the stack, will fail and will try to become Master. At this moment BFD MAD will kick in and will shutdown all non-IRF links on the Chassis 2.
5. While Chassis 2 is fully upgraded and running, but with all its links in shutdown state, reboot the Chassis 1. As soon as it reboots, the BFD MAD on the Chassis 2 should unblock all previously blocked ports on the Chassis 2 and it will take over as Master. Of course there will be a hickup in traffic forwarding, it is not ISSU after all, but the downtime will be minimal. 
6. As soon as Chassis 1 reboots using the new image, it will join the stack as IRF Standy, because images now on both Chassis are equal, so nothing will stop them from forming a stack. At this moment you should have IRF stack of 2 Chassis running new image.

Hope this helps!

[hemh]

Hi Yvan, thanks for your answer.

When you say "At this moment BFD MAD will kick in and will shutdown all non-IRF links on the Chassis 2." , how to be shure the BFD-MAD won't shut chassis 1 ports?

[Ivan_B · Accepted solution]

This behavior is described in the IRF Configuraiton Guide, https://psnow.ext.hpe.com/doc/c04693970 , Page 9, 'Collision Handling' section:

BFD MAD uses the following process to handle a multi-active collision:

1. Allows the IRF fabric that has the lowest numbered master to forward traffic.
2. Sets all other fabrics to the Recovery state.
3. Takes the same action on the network ports in Recovery-state fabrics as LACP MAD.

[hemh]

Thank you very much, veery apreciated.

[hemh]

Hi,

I suppose at the end of the upgrade, chassis 2 will remain the master. Is there a kind of election process taht will turn chassis 1 as master, or is there a command to force chassis 1 to gain back its master role?

Thanks for your answer.

[Ivan_B]

Unfortunately, only reboot of the Chassis 2, current Master at that time will trigger the failover. However, I don't see any issue Chassis 2 to remain Master, at least technical. 

[hemh]

Hi,

I wanted to bring a little feedback on the upgrade we did this sunday, everything went as expected, exept one thing, let me detail it:

On our two members 7910's stack with BFD-MAD enabled, chassis 2 was the master.

We fist upgraded chassis 2, after this, the IRF get broken, chassis 1 became master in the old firmware version.

Then we upgraded chassis 1, IRF get back on, after the reboot, because of BFD-MAD chassis 2 (witch became master during chassis 1 reboot) shutted down all its ports(Excepted the ones configured to not to) so it was a mess, we had to run the command "mad restore" on chassis 2 and then it went fine, and chassis 1 became master.

Excepted this, everything went fine.

[Ivan_B]

Hi @hemh !

Unfortunately we need to clarify what has happened during the upgrade, just to be sure I correctly understood the situation:

1. Chassis 1 - Standby; Chassis 2 - Master.
2. Upgrade Chassis 2, Chassis 2 reboots, Chassis 1 becomes Master.
3. Chassis 2 completes the bootup, couldn't join the stack because of s/w version mismatch and MAD disabled all ports on it. Chassis 1 keeps working as Master with ports UP.
4. Upgrade Chassis 1, reboot. 

Now here is the point when I didn't understand what happened. Do you say that after the reboot of Chassis 1 all ports on Chassis 2 were still DOWN? Or they went UP as soon as Chassis 1 rebooted, but then turned DOWN as soon as it completed the reboot and joined the stack? 

[parnassus]

---

@hemh wrote: On our two members 7910's stack with BFD-MAD enabled, chassis 2 was the master. We fist upgraded chassis 2, after this, the IRF get broken, chassis 1 became master in the old firmware version.

Out of curiosity: Shouldn't the ISSU (incompatible or not) be started exactly by upgrading the IRF Standby member first? ...Chassis 1 in the above scenario.

[Ivan_B]

@parnassus  Exactly! I think that is the point were things started to go in the wrong direction....

[hemh]

Hi Yvan,

As soon as we lauched the chassis 1 reboot, point 4 in your message, chassis 2 shut down all its ports, excepted the ones excluded of course, and we had to run the command mad restore to re-establish connectivity int the datacenter.

[hemh]

---

Hi Parnassus,

So can I assume that the best way to do this kind of upgrade, is to have chassis 1 as master before the operation, to make the operation easier?