IRF upgrades for HPE 5930 switches performed using the In-Service Software Upgrade (ISSU)

[GerryOBrien]

Hi,

Are all upgrades for IRF on HPE 5930 switches performed using the In-Service Software Upgrade (ISSU) method? We are trying to build a resilient network which uses IRF but which can be upgraded without have to reboot all members of the IRF. Is this possible or are there some upgrades which cannot be done using ISSU?

      Regards,

          Gerry

[VoIP-Buddy]

Gerry,

It depends.  You need to consult the release notes on the version that you want to upgrade to.  Some you can and some you cannot. 

All upgrades can be staged on the switch and then reboot them when you are able to if ISSU is not supported.  It is a nice idea to be able to upgrade with no downtime but there are a lot of cases where you don't have that choice.,

Regards,

David

[spgsitsupport]

On my 4 member stack with 5900AF I did not manage to do upgrade with ISSU. It failed miserably splitting IRF into two parts with same IP (just disaster really)

Seb

[VoIP-Buddy]

Seb,

Did you look at the release notes to see the ISSU compatibility matrix to see if what you were trying to do was supported?  The information is there. 

I'm sorry your IRF stack split with the same IP addresses.  I hope you were able to recover it quickly before too many issues occurred.

Regards,

David

[spgsitsupport]

Isn't this the issue:

When using more than 2 units in the IRF system, ISSU assumes the peer devices are connected to all switches in the IRF system. For example: if you have a server IRF system with 4 switches, server are assumed to be connected to each of the 4 switches. This is highly unlikely, and therefore I recommend to stick to 2 units in the IRF system for any deployment which requires ISSU. When a customer can have a maintenance window and accepts downtime, more than 2 switches in the IRF can be used of course."

4 member IRF stack seems to need full reboot

<HPE5900-SR1>display version comp-matrix file ipe flash:/5900_5920-CMW710-R2432P01.ipe
Verifying the file flash:/5900_5920-cmw710-r2432p01.ipe on slot 1..........Done.
Boot image: 5900_5920-cmw710-boot-r2432p01.bin
  Version:
  7.1.045

System image: 5900_5920-cmw710-system-r2432p01.bin
  Version:
  7.1.045-Release 2432P01
  Version compatibility list:
  7.1.045-Feature 2421
  7.1.045-Feature 2424
  7.1.045-Feature 2426
  7.1.045-Feature 2427
  7.1.045-Feature 2428
  7.1.045-Release 2416
  7.1.045-Release 2418P01
  7.1.045-Release 2418P06
  7.1.045-Release 2422
  7.1.045-Release 2422P01
  7.1.045-Release 2422P02
  7.1.045-Release 2423
  7.1.045-Release 2432P01
  Version dependency boot list:
  7.1.045

  Slot                        Upgrade Way
  1                           Reboot
  2                           Reboot
  3                           Reboot
  4                           Reboot

That is just plain awful and confusing, as there is mention of multi member ISSU upgrade here:

http://www.h3c.com.hk/Technical_Support___Documents/Technical_Documents/Switches/H3C_S12500_Series_Switches/Configuration/Operation_Manual/H3C_S12500_CG-Release7128-6W710/01/201301/772597_1285_0.htm#_Toc341179349

[VoIP-Buddy]

Hi Spgsitsupport!

So no, there is no assumption whatsoever on the part of ISSU.  The only thing that it does is check to see if there is a possibility of doing a compatible upgrade in the ISSU sense.  Engineering tests this prior to release to come up with the matrix.

It is up to the customer to ensure that their servers are dual-homed to reduce the possibility of a network outage and to minimize that downtime.  If an inventory of all connections to the stack are done and the links are split among the stack members there is a chance that there will be a reduced outage..  THe members still need to reboot though.

The latest versions of Comware 7 on some platforms allow up to 8 members.  That's a lot of equipment but some sites need that connectivity.

Regards,

David

[parnassus]

---

VoIP-Buddy wrote: It is up to the customer to ensure that their servers are dual-homed to reduce the possibility of a network outage and to minimize that downtime.  If an inventory of all connections to the stack are done and the links are split among the stack members there is a chance that there will be a reduced outage..  THe members still need to reboot though.

---

Just dual-homed or, better, multi-homed [*] to keep the outage window as low as possible (or to let the necessary outage time to not impact IRF Stack's multi-homed connected hosts at all)?

Just curious about such scenario...

[*] As example when the IRF Stack is made of more than, let's say, four or more members.

[spgsitsupport]

As I said, last time I tried it with 4 members it failed in spectacular way.  Followed instructions, upgraded member, rebooted, swapped master & then I had it split, with both parts using the same IP

Total disaster

Seb

Ofcourse each of my servers is dual connected, which did not help in that case

[GerryOBrien]

Hi,  We have a 4 x FlexFabric 5930 IRF stack., slots 1 to 4, connected in ring topology with fully connected BFD MAD. The odd slots, 1 & 3 are in building A and the even slots, 2 & 4 are in building B. The ring is slots 1, 3, 4, 2 and back to 1. Servers and storage in building A are dual homed and connected to slots 1 & 3. Likewise, servers and storage are dually connected to slots 2 & 4.

 Our local HPE support informed us that ISSU updates are not supported for more than a 2 slot IRF stack. This seems strange guiven that a stack can have something like 15 members. Is this HPE official policy?

 In any event, we have sucessfully performed an ISSU compatible upgrade by following the manual. After the "issu run switchover" we performed "issu commit slot N" on each of the remaining slots in turn, waiting until each one had rejoined the stack before continuing on to the next slot. The was no server or storage downtime, as each is dually connected, only one slot was down at any given time and the slots are connected in a ring. This is what one would expect, nothwithstanding HPE's disclaimer above.

  Our real issue is how to prform an imcompatible upgrade. The issue here is that after a stack slot that has been upgraded it cannot rejoin the current IRF stack and MAD will shutdown all its ports. Any idea how a ring stack can be upgraded without downtime?

    Regards,

      Gerry

[parnassus]

---

spgsitsupport wrote: Ofcourse each of my servers is dual connected, which did not help in that case

---

And that's the reason because I asked: if your Server were multi-homed instead of only dual-homed (so each Server host connected to the IRF Stack had four uplinks equally distributed to all four members of the IRF Stack...instead of only two links as I understand your Servers had) things could have been better (in terms of impact against the necessary outage time) or essentially not?

I mean: managing (or suffering) an IRF split brain is a thing, managing a Compatible ISSU with a sub group of 2 IRF members (of the grand total of 4) that reboot while Server hosts still have two links up (and two others down) to remaining still active IRF members is another thing...

Hope to be not Off Topic with my question.