Stacking connection failed and no redundancy

[tristar]

I hope to leverage on HPE community who can advise me on the issue I faced and how to resolve the problem.

Any suggestions are welcome.

I have a c7000 with primary VC FlexFabric-20/40 F8 Module (4.30) in bay 1, and CE replaced  a VC FlexFabric-20/40 F8 Module (4.45) in Bay 2, VC 8Gb 24-Port FC Module (1.11) in bay 3 and VC 8Gb 24-Port FC Module (1.11) in Bay 4. OA is at 4.97

The Domain configuration and Module configurations are in Sync, but Stacking connection and redundancy failed. In the VCM webpage saw Bay 1 is degraded. Bay 2 incompatible.

Run health check via vcsu and the result as follows:

[Mon Dec 11 12:16:27 2023] Stacking Connection : Failed
[Mon Dec 11 12:16:27 2023] Stacking Redundancy : No Redundancy

[Mon Dec 11 12:16:27 2023] Bay 1 : HP VC FlexFabric-20/40 F8 Module
[Mon Dec 11 12:16:27 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:27 2023] Power : On
[Mon Dec 11 12:16:27 2023] Health : Ok
[Mon Dec 11 12:16:27 2023] IP Address : xx.xx.xx.xx
[Mon Dec 11 12:16:27 2023] IP Connectivity : Passed
[Mon Dec 11 12:16:27 2023] Version : 4.30
[Mon Dec 11 12:16:27 2023] Mode : Primary
[Mon Dec 11 12:16:27 2023] Domain Configuration : In Sync
[Mon Dec 11 12:16:27 2023] Module Configuration : In Sync
[Mon Dec 11 12:16:27 2023]
[Mon Dec 11 12:16:27 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:27 2023] Bay 2 : HP VC FlexFabric-20/40 F8 Module
[Mon Dec 11 12:16:27 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:27 2023] Power : On
[Mon Dec 11 12:16:27 2023] Health : Ok
[Mon Dec 11 12:16:27 2023] IP Address : xx.xx.xx.xx
[Mon Dec 11 12:16:27 2023] IP Connectivity : Passed
[Mon Dec 11 12:16:27 2023] Version : 4.45
[Mon Dec 11 12:16:27 2023] Mode : Backup
[Mon Dec 11 12:16:27 2023] Domain Configuration : In Sync
[Mon Dec 11 12:16:27 2023] Module Configuration : In Sync
[Mon Dec 11 12:16:27 2023]
[Mon Dec 11 12:16:27 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:27 2023] Bay 3 : HP VC 8Gb 24-Port FC Module
[Mon Dec 11 12:16:27 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:27 2023] Power : On
[Mon Dec 11 12:16:27 2023] Health : Ok
[Mon Dec 11 12:16:28 2023] IP Address : xx.xx.xx.xx
[Mon Dec 11 12:16:28 2023] IP Connectivity : Passed
[Mon Dec 11 12:16:28 2023] Version : 1.11
[Mon Dec 11 12:16:28 2023]
[Mon Dec 11 12:16:28 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:28 2023] Bay 4 : HP VC 8Gb 24-Port FC Module
[Mon Dec 11 12:16:28 2023] -------------------------------------------------------------------------------
[Mon Dec 11 12:16:28 2023] Power : On
[Mon Dec 11 12:16:28 2023] Health : Ok
[Mon Dec 11 12:16:28 2023] IP Address : xx.xx.xx.xx
[Mon Dec 11 12:16:28 2023] IP Connectivity : Passed
[Mon Dec 11 12:16:28 2023] Version : 1.11
[Mon Dec 11 12:16:28 2023]
[Mon Dec 11 12:16:28 2023]
[Mon Dec 11 12:16:28 2023] Total execution time: 00:01:30
[Mon Dec 11 12:16:28 2023]
[Mon Dec 11 12:16:28 2023] VCSU log file is available at the following location: C:\Program Files
[Mon Dec 11 12:16:28 2023] (x86)\Hewlett Packard Enterprise\Virtual Connect Support Utility\vcsu-5084.log

[support_s]

System recommended content:

1. C7000 Enclosure Degraded - Stacking Connectivity Error: enc0:iobay2 does not have a stacking path to enc0:iobay1

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[MV3]

Hello,

From the above output we see the VC modules firmware is incompatible with the adjacent module. Could be either of these modules were replaced and FW not updates\ downgraded. We see the Primary module is 4.30 and the Standby module is 4.45. Downgrade the FW to 4.30 , take a backup of the VC Domain and update the FW to latest.

We would strongly suggest you to update the FW to the latest version 4.85 as this version has numerous fixes. 

Link to download the FW :  https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_f99f0bc5bfc4414aac021f81af&tab=releaseNotes

Please Note: The Firmware version installed presently is EOL and the upgrade should be done in an incremental way and a backup of the VC Domain is must prior upgrading to next version.

Cheers.....

[tristar]

Hi MV3

Thanks for incurring a little of your time to feedback. Ultimately the goal is to up the VC firmware to 4.85. HPE advisory is to up level 4.4x or 4.5x at intermediate stage before moving further. So my upgrade path is v4.30 --> v4.50 --> v4.75 --> v4.85.

Yes, the only abnormally is the stacking error reported by vcsu, VCM reported Bay 1 degraded, Bay 2 incompatible. Currently need to resolve the 2 VC modules to common ground before moving forward. I search in forums, blogs and HPE community for leads.

I perused your passage, to downgrade the standby module from v4.45 to v4.30. However, HPE download has removed v4.30 replaced with v4.31. 

Question 1: If I downgrade the standby module from v4.45 to v4.31, would the primary module be upgraded from v4.30 to v4.31 using VCSU Force option in the process? Select Version and Health together?

Question 2: For current situation, Bay 1 is v4.30, Bay 2 is v4.45. Can I use v4.50 firmware update to bring both modules to common ground, specify Force option, Health select? Workable?

Your reply very much apprecaited.

I have backup the VC domain v4.30 configuration, is it compatible to restore into v4.31?

[MV3]

Hello Tristar,

Confirmation: The upgrade path is accurate.

Regarding the removal of Firmware version 4.30, please consider the following steps:

Safeguard the VC Domain by taking a comprehensive backup.

Transfer the standby module to Bays 5-6, and initiate a force option to execute the firmware downgrade.

After the downgrade is completed, return the standby module to Bay 2.

Upgrade the VC Module in Bay 1 to firmware version 4.31, and examine for any synchronization and stacking errors.

Once these issues are resolved, proceed with updating the VC firmware to the latest version incrementally.

Should you require assistance during the activity, feel free to raise a support case with HPE.

Cheers.

[tristar]

@MV3 

Hello MV3

Your suggestion is to place the standby module from Bay 2 to  Bay 5 , downgrade the firmware from v4.45 to v4.31, using Force option, I need to select Version instead of Health, correct? The vcsu command -b parameter to target Bay had been removed.

Bay 1 is the primary module v4.30 running.

Do I require to shutdown all blades for the downgrade process?

Wishing you have a joyous Merry Christmas and also to all in HPE community.

[MV3]

Hello

Wishing you a joyful New Year!
Certainly, you have the option to position the standby module in Bay 5 and designate the version. The goal is to prevent any unplanned reboots or disruptions to the primary module.

It's unnecessary to power off the blades since the primary module is currently handling the network.
Cheers....

[tristar]

@MV3 Hello MV3

Can I remove the secondary VC Flexfabric from Bay 2, then upgrade the primary VC Flexfabric alone in Bay 1 from V4.30 to V4.45.

After that put the secondary VC Flexfabric back in Bay 2.

Does it work this way?

Thank you.

[Vinky_99]

---

@tristar wrote:

@MV3 Hello MV3

Can I remove the secondary VC Flexfabric from Bay 2, then upgrade the primary VC Flexfabric alone in Bay 1 from V4.30 to V4.45.

After that put the secondary VC Flexfabric back in Bay 2.

Does it work this way?

Thank you.

---

Yes, you can follow the approach you've outlined, but there are some important considerations to keep in mind:

1. Remove Secondary VC FlexFabric (Bay 2)>> Safely remove the secondary VC FlexFabric module from Bay 2.

2. Upgrade Primary VC FlexFabric (Bay 1) from V4.30 to V4.45>> Upgrade the firmware of the primary VC FlexFabric module in Bay 1 from version 4.30 to version 4.45.

3. Reinsert Secondary VC FlexFabric (Bay 2)>> After successfully upgrading the primary module, you can reinsert the secondary VC FlexFabric module into Bay 2.
4. Verify Stacking and Redundancy>> Once both modules are back in place, verify the stacking connection and redundancy status. Ensure that both modules are in sync and operating as expected.

5. Monitor System Health>> Keep an eye on the system health and check for any error messages or warnings in the Virtual Connect Manager (VCM) interface.

6. Always consult the release notes for the specific firmware versions you are upgrading to. The release notes may contain important information about the upgrade process, compatibility considerations, and any known issues.

7. Before making any changes, it's a good practice to backup the configuration of your Virtual Connect Manager. This ensures that you can restore the configuration in case of any unexpected issues.

8. Depending on your environment, be prepared for a brief downtime during the removal, upgrade, and reinsertion of modules. Plan the upgrade during a maintenance window to minimize impact on operations.

9. If you encounter any difficulties during the upgrade or face unexpected issues, please contact HPE support for assistance.

Remember to follow the HPE Virtual Connect documentation and best practices for firmware upgrades to ensure a smooth process and to mitigate the risk of any potential issues.

Hope this helps!

[tristar]

Hi Vinky_99

Before your comments came, I worked on by transferring VC FlexFabric (Bay 2) to Bay 6.

My intention just wanted to down the firmware from v4.45 to v4.31 for this module only, to preserve primary Bay 1 which holds the VCM configuration.

Using v4.31, I ran update using vcsu v 1.15.0 with force health. The final table showed 2 actions to take place - Bay 1 would up level to v4.31, and Bay 6 down level to v4.31.

I quit the process because I was unsure what VCM final configuration would be at the end of the update, as the original setup of Bay 1 and Bay 2 stacking no longer the same.

I managed to get hold another spare VC FlexFabric with v4.31 to work on. This time round I would use it to replace the secondary in Bay 2.

From there, I just update the primary Bay 1 to v4.31. Hope this works.

[tristar]

Today, replaced with the VC Flexfabric v4.31 in Bay 2.

From Checkpoint Pending to VC Domain Checkpointed took about an hour, I supposed synchronisation had taken place.

Followed up with vcsu on healthcheck, other than the stacking connection failed and stacking no redundancy, the rest were good, particulary both the Domain Configuration and Module Configuration were in sync for primary and backup.

With that, ran vcsu update with health option using v4.31 firmware to uplevel Bay 1, ended in fatal error. Reason being the VC Domain Configuration out of sync..

Rerun the steps an hour later, the result ended the same.

We followed the procedure as documented.

Anyone encounter this problem.

[Vinky_99]

The fact that the VC Domain Configuration is reported as out of sync during the firmware update process is concerning.

Here are some steps and considerations you can take to troubleshoot and address the issue:

>> Verify in the Virtual Connect Manager (VCM) interface that both Domain Configuration and Module Configuration are indeed in sync for both the primary and backup modules.

>> Ensure that the stacking cables are properly connected between the primary and backup modules (Bay 1 and Bay 2). Physical issues with stacking connections can lead to synchronization problems.

>> Check the error logs in the Virtual Connect Manager interface for any specific error messages related to the synchronization issue. This can provide more details about what aspect of the configuration is causing the problem.

>> Confirm that the firmware version you are attempting to install (v4.31) is compatible with your existing configuration and that it aligns with any specific requirements outlined in the release notes.

>> Instead of jumping directly from v4.30 to v4.31, consider whether there are any intermediate firmware versions that need to be applied first. Sometimes, certain versions may have dependencies on prior updates.

>> Double-check the HPE Virtual Connect documentation and release notes for any special instructions or considerations when upgrading firmware in a stacked configuration.

>> Before attempting the firmware update, run the health check using VCSU to ensure that everything is in sync. Address any issues reported by the health check before proceeding with the firmware update.

>> Before making any significant changes, ensure that you have a backup of the current configuration. This allows you to roll back to a known good state if needed.

>>Plan for a maintenance window to minimize the impact on operations during the troubleshooting and update process.

>>If the issue persists and you cannot find a resolution, it's advisable to contact HPE support.

[tristar]

Here are some steps and considerations you can take to troubleshoot and address the issue:

>> Verify in the Virtual Connect Manager (VCM) interface that both Domain Configuration and Module Configuration are indeed in sync for both the primary and backup modules.

Yes,  I run the vcsu healthcheck, return results show both Domain Configuration and Module Configuration are in sync. for primary (Bay 1) and backup (Bay 2).

>> Ensure that the stacking cables are properly connected between the primary and backup modules (Bay 1 and Bay 2). Physical issues with stacking connections can lead to synchronization problems.

Stacking use internal port s x9 and x10, there is no stacking cables used.

>> Check the error logs in the Virtual Connect Manager interface for any specific error messages related to the synchronization issue. This can provide more details about what aspect of the configuration is causing the problem.

VCM log indicated Bay 1 is degraded,. The root cause is the VC FLexFabric firmware mismatch. (Bay 1 is v4.30, Bay 2 is v4.31)

>> Confirm that the firmware version you are attempting to install (v4.31) is compatible with your existing configuration and that it aligns with any specific requirements outlined in the release notes.

The firmware version supports VC FlexFabric 20/40  F8 module and  VC 8GB 24-port FC module

>> Instead of jumping directly from v4.30 to v4.31, consider whether there are any intermediate firmware versions that need to be applied first. Sometimes, certain versions may have dependencies on prior updates.

There is no intermediate version, v4.30 was removed and replaced with v4.31.

>> Double-check the HPE Virtual Connect documentation and release notes for any special instructions or considerations when upgrading firmware in a stacked configuration.

There is no special instruction., to update the firmware using force health option in this situation.

>> Before attempting the firmware update, run the health check using VCSU to ensure that everything is in sync. Address any issues reported by the health check before proceeding with the firmware update.

VCSU version 1.15 is used.
Run vcsu healthcheck, both primary and backup Domain configuration and module configuration are in sync

Run vcsu update with force health ended in fatal error, stating that the VCM domain configuration is Out of Sync.

I had repeated the process a day after but still face the same error.

>> Before making any significant changes, ensure that you have a backup of the current configuration. This allows you to roll back to a known good state if needed.

I did the backup of current configuration (v4.30), but there is no v4.30 available for download, in case I need to fall back.

>>Plan for a maintenance window to minimize the impact on operations during the troubleshooting and update process.

>>If the issue persists and you cannot find a resolution, it's advisable to contact HPE support.

I already contacted HPE,  he presented readme of the version history update from v4.30 to v4.31,  with vcsu v1.10.1 at the time of release. He also said that vcsu v 1.15.0 should be able to do the job which currently I'm using.

I'm not sure should I use vcsu v 1.10.1  instead to see if it can solve the issue.

The next thing I try is to update Bay 1  alone  from v 4.30 to v 4.31 (with Bay 2 module removed) , to make it on par with Bay 2 version, before I put back Bay 2 module. In doing so, I'm not sure the VCM domain configuration would remain the same.

[tristar]

Hello HPE community

I'm now in deadlock situation, unable to update VC FlexFabric firmware to higher versions.

I tried to raise the Bay 1 firmware to v 4.31 first before upgrade both to higher versions, but without success.

I run vcsu v 1.10.1 update with health option, the process terminated with error VCM domain configuration Out of Sync. I also used the latest vcsu version 1.15.0 still encounter the same error. Did I miss out any step before running vcsu? Any clue to this situation?

[simoncorner]

Hi,

I had a similar experience a while ago and the solution was to reboot the module with the older firmware. This will make the newer one active and then you should be able to update the older one. to 4.31.

You will probably still need to use the "health" option to upgrade the older one.

Once that's complete, you should be able to get them up to newer versions.

Obviously there is a chance this will cause an outage so take a backup of your VC config and make the change outside of busy hours.

Simon

[tristar]

@simoncorner Thank you, Simon, for your valuable feedback.

A second thought.

Since Bay 1  VC Flexfabric is active with VCM domain configuration.

When reboot Bay 1 Flexfabric module, the Bay 2 Flexfabric would take over active role with the VCM domain configuration?

It is a production system, I don't have the privilege to work on test system.

Cheers.

[simoncorner]

Yes, the idea is that as long as the module in bay 2 has the config, as soon as you reboot bay 1, bay 2 will take over and then you should be able to update bay 1 to 4.45.
When I did this, I was confident that the standby module had the full vc config on.
The other option, as previously stated is to remove bay 2. Then upgrade bay 1 to 4.45 but that will incur an outage as when bay 1 reboots you won't have another module in bay 2 to switchover to.

[tristar]

Hi Simon

We are closer to overcome the issue. I share the vcsu healthcheck snippet. The report says stacking connection failed and no redundancy. The primary module and backup module domain configuration and module configuration are in sync. Can I safely say that the backup module also has the  same full VC configuration as the primary? As you mentioned to reboot Bay 1 and let Bay 2 to takeover as active. From here, I can update with health option Bay 1 firmware to the same as Bay 2, and subsequently, once both are at the same version, I can proceed on the update path to higher versions.

When I observed in the VC manager, I saw Bay 1 as degraded, and Bay 2 is incompatible. This is where my concern is.

Thank you for your sharing, very appreciative.

[simoncorner]

It certainly looks like the config is ok on both. I'm not sure what impact the stacking issues could have though.
I think it's worth trying but obviously it's not my system. Also, if you're running an air traffic control system or similar, more care may be required!

[tristar]

@simoncorner Totally agreed, different systems have their own behaviour, depending on the setup.

Your sharing method of reboot the active module of lower firmware, to make the passive one of higher firmware to takeover the active role. It worked in your case. I give it a shot.

I came across a forum being shared, the active module was of higher firmware version, and could update the passive module to the same version as active one.

So to say of encountering my issue,  it seems like if the active module is of lower firmware, when vcsu update with health option, the process ended with fatal error stating VCM domain out of sync.

Actually, my intention is either place the Bay 2 module in unused Bay or remove from the Bay as you also mentioned as another option. In doing the firmware update process takes about 40 minutes, and the systems are inaccessible during this period.

My only primary concern is to protect the VCM domain configuration, which I had backed up for VC v 4.30, but not sure is compatible with v 4.31 in case need to roll back, because v 4.30 no longer available. I also make a copy of vcshowall.txt.

Nevertheless, if I don't try, I never know the answer.

Cheers!