Failed VC Flex-10 module. What's the best way to replace without disruption?

[ay_jay]

We have a few errors on our standby VC Flex 10 module.

In the HPOA UI, there is a red alert symbol alongside the second Flex-10 module. When you click the link to expand the configuration display in the UI for the module, it displays "Critical Error" under and as its 'Status', and also under 'Diagnostics' for Management Processor and Device Operational.

HPE sent us a replacement and I've read somewhere that these modules are hot-swappable, but I need instructions on how to get the replacement in service with as little disruption as possible.  Are there instructions or is there documentation on how to properly do this?

[Suman_1978]

Hi,

You may refer to this guide, page# 257
Virtual Connect for c-Class BladeSystem User Guide Version 4.50

Thank You!
I work with HPE but opinions expressed here are mine.
Recent Support Video Releases

[Mark-S]

Regardless if the module is managed by Virtual Connect Manager, or by OneView, the newely inserted module will receive the configuration once it has been inserted and booted up, that is when the configuration will be pushed to that new module. So as long as you have redundant paths setup and operational, there will be no downtime and only the module replacement uplinks/downlinks will be effected.

Thanks- 

[ay_jay]

HPE sent us a replacement but we don't know what firmware version it's on.  Would that have any impact on just swapping the module out?  I have not been getting much info from our engineer and we meant to swap it out last week but rescheduled because when I finally spoke with him, he mentioned we should really do this during a maintenance window since we might need to take the chassis down to perform the swap.  That's all we know so far, per the enginer, and that seems to conflict with what you're saying here.

This is a "standby" module so the failure didn't disrupt any production traffic, but only because the bare-metal blade servers and ESXi hosts in the chassis are configured to use secondary interfaces (bonds in RHEL and redundant interfaces in ESXi) that showed as connected before the failure, so it was somewhat active before it went down.  We would have to take down a number of production systems on those blades which may add time to the maintenance window, so I'm trying to find out clearly if that much is necessary or not because hot-swapping might take a few minutes with no interruption as opposed to hours of downtime.  The documentation does not show much about replacement of a failed VC Flex-10 module or replacement with one that might have an older (or newer) firmware version.  Please advice.  Thanks

[Mark-S]

To be on the safe side, a maintenance window would be approrpriate if something went wrong and I can only assume that is why that was mentioned. There are not any issues I am aware of that you would encounter when replacing the Interconnect Module if FW does not match.

Is this managed by OneView or is this the native management (Virtual Connect Manager) where you use VCSU to update the firmware? What verison of firmware is on the module currently?

thanks-

[ay_jay]

It is managed by the native system HP Onboard Administrator (Bays 1 and 2).  The current firmware versions are as follows:

BladeSystem c7000 DDR2 Onboard Administrator with KVM: 4.95
BladeSystem c7000 Onboard Administrator Tray: 1.7
HP VC Flex-10 Enet Module: 4.63

Keep in mind the VC Manager for this system is Flash-based (and/or Java-based?) and we cannot use Flash to manage the VCs since it is no longer supported and we don't have it installed on our workstations.  If you have any recommendations on how best to monitor and manage this swap without Flash, that would help as well.  We do still have access to the HPOA UI, but it only gives high-level information about the Virtual Connect modules, not direct configuration.  We do have SSH access, however.

[Torsten.]

If you insert the new module it will just sit and do nothing as long as the firmware is different.

Use vcsu to update to the running version and use "health" to force, NOT version!

This will update the new module only while the other is running.

[Mark-S]

If the module is down rev firmware which is highly likely you can install in an empty bay and update that module before removing and replacing the standby module, but that is not a requirement.

I would suggest just replacing and updating in the bay the module is currently installed.

From CLI, when you remove and install with a module down rev, the CLI will show something like this (Example below)

->show systemlog

Enet Module state INCOMPATIBLE

Module in bay enc0:iobay2 firmware version 4.30 does not match the
supported firmware version 4.85

->show domain

Domain Name : ENG8_vc_domain
Checkpoint Status : Not Valid
FIPS Mode : false
CNSA Mode : false

You can then run VCSU command line and run report against the FW package you are installing

Here is an example from 4.85 report

Then run the actual update, you can choose to manual activate if you feel that is necessary to control the process but you have to use the health force option since the stacking connection is failed.

When you run the update, it will prompt you to continue but will show you what is going to take place.

Once the update is complete, the configuration will be pushed to the replacement module and the checkpoint will become valid. Up until then, show domain will report Not Valid for checkpoint. Systemlog will not show you anything until maybe the module is updated and the checkpoint is valid.