Stacking connection failed and no redundancy

[tristar]

I hope to leverage on HPE community who can advise me on the issue I faced and how to resolve the problem.

Any suggestions are welcome.

I have a c7000 with primary VC FlexFabric-20/40 F8 Module (4.30) in bay 1, and CE replaced  a VC FlexFabric-20/40 F8 Module (4.45) in Bay 2, VC 8Gb 24-Port FC Module (1.11) in bay 3 and VC 8Gb 24-Port FC Module (1.11) in Bay 4. OA is at 4.97

The Domain configuration and Module configurations are in Sync, but Stacking connection and redundancy failed. In the VCM webpage saw Bay 1 is degraded. Bay 2 incompatible.

Run health check via vcsu and the result as follows:

[Mon Dec 11 12:16:27 2023] Stacking Connection : Failed
[Mon Dec 11 12:16:27 2023] Stacking Redundancy : No Redundancy

[Mon Dec 11 12:16:27 2023] Bay 1 : HP VC FlexFabric-20/40 F8 Module
[Mon Dec 11 12:16:27 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:27 2023] Power : On
[Mon Dec 11 12:16:27 2023] Health : Ok
[Mon Dec 11 12:16:27 2023] IP Address : xx.xx.xx.xx
[Mon Dec 11 12:16:27 2023] IP Connectivity : Passed
[Mon Dec 11 12:16:27 2023] Version : 4.30
[Mon Dec 11 12:16:27 2023] Mode : Primary
[Mon Dec 11 12:16:27 2023] Domain Configuration : In Sync
[Mon Dec 11 12:16:27 2023] Module Configuration : In Sync
[Mon Dec 11 12:16:27 2023]
[Mon Dec 11 12:16:27 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:27 2023] Bay 2 : HP VC FlexFabric-20/40 F8 Module
[Mon Dec 11 12:16:27 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:27 2023] Power : On
[Mon Dec 11 12:16:27 2023] Health : Ok
[Mon Dec 11 12:16:27 2023] IP Address : xx.xx.xx.xx
[Mon Dec 11 12:16:27 2023] IP Connectivity : Passed
[Mon Dec 11 12:16:27 2023] Version : 4.45
[Mon Dec 11 12:16:27 2023] Mode : Backup
[Mon Dec 11 12:16:27 2023] Domain Configuration : In Sync
[Mon Dec 11 12:16:27 2023] Module Configuration : In Sync
[Mon Dec 11 12:16:27 2023]
[Mon Dec 11 12:16:27 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:27 2023] Bay 3 : HP VC 8Gb 24-Port FC Module
[Mon Dec 11 12:16:27 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:27 2023] Power : On
[Mon Dec 11 12:16:27 2023] Health : Ok
[Mon Dec 11 12:16:28 2023] IP Address : xx.xx.xx.xx
[Mon Dec 11 12:16:28 2023] IP Connectivity : Passed
[Mon Dec 11 12:16:28 2023] Version : 1.11
[Mon Dec 11 12:16:28 2023]
[Mon Dec 11 12:16:28 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:28 2023] Bay 4 : HP VC 8Gb 24-Port FC Module
[Mon Dec 11 12:16:28 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:28 2023] Power : On
[Mon Dec 11 12:16:28 2023] Health : Ok
[Mon Dec 11 12:16:28 2023] IP Address : xx.xx.xx.xx
[Mon Dec 11 12:16:28 2023] IP Connectivity : Passed
[Mon Dec 11 12:16:28 2023] Version : 1.11
[Mon Dec 11 12:16:28 2023]
[Mon Dec 11 12:16:28 2023]
[Mon Dec 11 12:16:28 2023] Total execution time: 00:01:30
[Mon Dec 11 12:16:28 2023]
[Mon Dec 11 12:16:28 2023] VCSU log file is available at the following location: C:\Program Files
[Mon Dec 11 12:16:28 2023] (x86)\Hewlett Packard Enterprise\Virtual Connect Support Utility\vcsu-5084.log

[support_s]

System recommended content:

1. C7000 Enclosure Degraded - Stacking Connectivity Error: enc0:iobay2 does not have a stacking path to enc0:iobay1

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[MV3]

Hello,

From the above output we see the VC modules firmware is incompatible with the adjacent module. Could be either of these modules were replaced and FW not updates\ downgraded. We see the Primary module is 4.30 and the Standby module is 4.45. Downgrade the FW to 4.30 , take a backup of the VC Domain and update the FW to latest.

We would strongly suggest you to update the FW to the latest version 4.85 as this version has numerous fixes. 

Link to download the FW :  https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_f99f0bc5bfc4414aac021f81af&tab=releaseNotes

Please Note: The Firmware version installed presently is EOL and the upgrade should be done in an incremental way and a backup of the VC Domain is must prior upgrading to next version.

Cheers.....

[tristar]

Hi MV3

Thanks for incurring a little of your time to feedback. Ultimately the goal is to up the VC firmware to 4.85. HPE advisory is to up level 4.4x or 4.5x at intermediate stage before moving further. So my upgrade path is v4.30 --> v4.50 --> v4.75 --> v4.85.

Yes, the only abnormally is the stacking error reported by vcsu, VCM reported Bay 1 degraded, Bay 2 incompatible. Currently need to resolve the 2 VC modules to common ground before moving forward. I search in forums, blogs and HPE community for leads.

I perused your passage, to downgrade the standby module from v4.45 to v4.30. However, HPE download has removed v4.30 replaced with v4.31. 

Question 1: If I downgrade the standby module from v4.45 to v4.31, would the primary module be upgraded from v4.30 to v4.31 using VCSU Force option in the process? Select Version and Health together?

Question 2: For current situation, Bay 1 is v4.30, Bay 2 is v4.45. Can I use v4.50 firmware update to bring both modules to common ground, specify Force option, Health select? Workable?

Your reply very much apprecaited.

I have backup the VC domain v4.30 configuration, is it compatible to restore into v4.31?

[MV3]

Hello Tristar,

Confirmation: The upgrade path is accurate.

Regarding the removal of Firmware version 4.30, please consider the following steps:

Safeguard the VC Domain by taking a comprehensive backup.

Transfer the standby module to Bays 5-6, and initiate a force option to execute the firmware downgrade.

After the downgrade is completed, return the standby module to Bay 2.

Upgrade the VC Module in Bay 1 to firmware version 4.31, and examine for any synchronization and stacking errors.

Once these issues are resolved, proceed with updating the VC firmware to the latest version incrementally.

Should you require assistance during the activity, feel free to raise a support case with HPE.

Cheers.

[tristar]

@MV3 

Hello MV3

Your suggestion is to place the standby module from Bay 2 to  Bay 5 , downgrade the firmware from v4.45 to v4.31, using Force option, I need to select Version instead of Health, correct? The vcsu command -b parameter to target Bay had been removed.

Bay 1 is the primary module v4.30 running.

Do I require to shutdown all blades for the downgrade process?

Wishing you have a joyous Merry Christmas and also to all in HPE community.

[MV3]

Hello

Wishing you a joyful New Year!
Certainly, you have the option to position the standby module in Bay 5 and designate the version. The goal is to prevent any unplanned reboots or disruptions to the primary module.

It's unnecessary to power off the blades since the primary module is currently handling the network.
Cheers....

[tristar]

@MV3 Hello MV3

Can I remove the secondary VC Flexfabric from Bay 2, then upgrade the primary VC Flexfabric alone in Bay 1 from V4.30 to V4.45.

After that put the secondary VC Flexfabric back in Bay 2.

Does it work this way?

Thank you.

[Vinky_99]

---

@tristar wrote:

@MV3 Hello MV3

Can I remove the secondary VC Flexfabric from Bay 2, then upgrade the primary VC Flexfabric alone in Bay 1 from V4.30 to V4.45.

After that put the secondary VC Flexfabric back in Bay 2.

Does it work this way?

Thank you.

---

Yes, you can follow the approach you've outlined, but there are some important considerations to keep in mind:

1. Remove Secondary VC FlexFabric (Bay 2)>> Safely remove the secondary VC FlexFabric module from Bay 2.

2. Upgrade Primary VC FlexFabric (Bay 1) from V4.30 to V4.45>> Upgrade the firmware of the primary VC FlexFabric module in Bay 1 from version 4.30 to version 4.45.

3. Reinsert Secondary VC FlexFabric (Bay 2)>> After successfully upgrading the primary module, you can reinsert the secondary VC FlexFabric module into Bay 2.
4. Verify Stacking and Redundancy>> Once both modules are back in place, verify the stacking connection and redundancy status. Ensure that both modules are in sync and operating as expected.

5. Monitor System Health>> Keep an eye on the system health and check for any error messages or warnings in the Virtual Connect Manager (VCM) interface.

6. Always consult the release notes for the specific firmware versions you are upgrading to. The release notes may contain important information about the upgrade process, compatibility considerations, and any known issues.

7. Before making any changes, it's a good practice to backup the configuration of your Virtual Connect Manager. This ensures that you can restore the configuration in case of any unexpected issues.

8. Depending on your environment, be prepared for a brief downtime during the removal, upgrade, and reinsertion of modules. Plan the upgrade during a maintenance window to minimize impact on operations.

9. If you encounter any difficulties during the upgrade or face unexpected issues, please contact HPE support for assistance.

Remember to follow the HPE Virtual Connect documentation and best practices for firmware upgrades to ensure a smooth process and to mitigate the risk of any potential issues.

Hope this helps!

[tristar]

Hi Vinky_99

Before your comments came, I worked on by transferring VC FlexFabric (Bay 2) to Bay 6.

My intention just wanted to down the firmware from v4.45 to v4.31 for this module only, to preserve primary Bay 1 which holds the VCM configuration.

Using v4.31, I ran update using vcsu v 1.15.0 with force health. The final table showed 2 actions to take place - Bay 1 would up level to v4.31, and Bay 6 down level to v4.31.

I quit the process because I was unsure what VCM final configuration would be at the end of the update, as the original setup of Bay 1 and Bay 2 stacking no longer the same.

I managed to get hold another spare VC FlexFabric with v4.31 to work on. This time round I would use it to replace the secondary in Bay 2.

From there, I just update the primary Bay 1 to v4.31. Hope this works.