MSA2040 firmware upgrade failed - Warning The system is currently unavailable

[R-Sommer]

Hi, I've tried to upgrade a msa 2040 fc from 210 to 225 but it failed. No I cannot longer login via gui due to the message "Warning The system is currently unavailable" but can login via ssh. Is there any chance to make it working again? Thanks!

[Dardan]

Have you tried to restart Management Controller(s)?

restart mc <a> or <b>

# restart mc b
During the restart process you will briefly lose communication with the specified Management Controller(s).
Do you want to continue? (y/n) y

For your info, this won't affect the IO activity.

[R-Sommer]

I've tried to restart mc as well as completely switched off the storage physically. check firmware-upgrade-health result in: Management Controller is getting data from the Storage Controller. Please wait... Management Controller is getting data from the Storage Controller. Please wait... Management Controller is getting data from the Storage Controller. Please wait... Management Controller is getting data from the Storage Controller. Please wait... Management Controller is getting data from the Storage Controller. Please wait... Management Controller is getting data from the Storage Controller. Please wait... Error: The MC is not ready. Wait a few seconds then retry the request. (2020-02-06 14:05:40)

[Shawn_K]

Hello,

If youi shutdown the system be sure to halt host IO first.

Are both controllers reporting the same thing? Trying logging in directly (CLI) to both controllers and see if controller A or B is reporting correctly. If they are both reporting the MC is waiting to refresh message you are likely in a weird state with the firmware upgrade.

If both controllers are not reporting correctly try to unseat one controller. This should allow the other controller to boot fully. Once it comes online check and confirm the firmware version. If the firmware is not correct use the GL225 Smart Component to upgrade the controller. Once complete plug the other controller in and wait at least 15 minutes for the firmware upgrade process to run.

For future reference - if you are upgrading from a version of firmware that is more than two major firmware versions back, it will require a multiple step process to get to the latest firmware version. Firmware should only be upgraded from the last two major firmware releases.

You can also log a support case to get further asistance from HPE Support. 

To open a support case with HPE Support using the following web link, if the unit is under warranty: https://support.hpe.com/hpesc/public/home

If the unit is out of warranty, you can open a chat support case with HPE using the following web link to check for options: https://pg-receiver-pro.glb.itcs.hpe.com/WCLWeb/WCLEntry.aspx

Cheers,
Shawn

I work for Hewlett Packard Enterprise. The comments in this post are my own and do not represent an official reply from HPE. No warranty or guarantees of any kind are expressed in my reply.

[R-Sommer]

Hello Shawn, Many thanks for your detailed information. Unfortunately, it looks like Controller B died with the firmware upgrade. If I remove it Controller A works fine, but I'm not able to upgrade its firmware. Using the Windows upgrade utility: ##################################################### The software is not supported for installation on this system. Firmware upgrade health check failed. Failure reason(s): At least one controller is not up. Please resolve the issue(s) and rerun the Smart Component. Validate the state of the Storage Array. Refer to Release Notes or Press 'Close' to exit Setup. ##################################################### Using the gui: ##################################################### Uploaded 65MiB. Starting /usr/bin/docodeload: Fri Feb 7 13:44:06 UTC 2020 Installed release: GLM210, storage model: HYBRID New release: GLM225, storage model: HYBRID Loading bundle [dbg] debug flags: Activity Checking bundle integrity. Bundle integrity checks passed. Loading bundle at Fri Feb 7 13:44:26 UTC 2020 SC ping: 1: Success Connection with the SC is up. Checking for SC version Check for cache size and CF System has 8 GB Cache and CF. Codeload health check complete. Health state: Fail ************************************************************************************ ERROR: System health insufficient to support firmware update. Reason: Fail At least one controller is not up. Action: You must resolve the this condition before the firmware update will proceed. ************************************************************************************ System health check failed. ========================================== Software Component Load Summary: MC Software: NOT ATTEMPTED SC Software: NOT ATTEMPTED EC Software: NOT ATTEMPTED CPLD Software: NOT ATTEMPTED ========================================== Some number of subcomponents failed. Code load completed successfully. ##################################################### Unfortunately, the service pack is out of time. So what can I do? Do I have to replace Controller B? Or could this be repaired using any kind of software? Thanks & Cheers Roland

[Shawn_K]

Hello,

For controller A check and verify if FTP is enabled.. Also, if you are upgrading a single controller that is not set to single controller mode the controller will report a degraded health status.

ERROR: System health insufficient to support firmware update. Reason: Fail At least one controller is not up.

You can set the controller to single controller mode and then do the upgrade on the A controller.

CLI # set advanced-settings single-controller enabled

Once the controller is in single controller mode you will be able to complete the upgrade.

Once you get controller A fully upgraded, try inserting controller B. Again you will have to wait at least 15 minutes for the controller to settle. You may be able to complete the firmware upgrade on both controllers in this manner.

Cheers,
Shawn

I work for Hewlett Packard Enterprise. The comments in this post are my own and do not represent an official reply from HPE. No warranty or guarantees of any kind are expressed in my reply.

[R-Sommer]

Hello Shawn, Thank you so much for your help. I was able to upgrade controller A to GL225P002-02. Plugging in controller B this morning (more than 4 hours ago) stops MC working properly. GUI: "WarningThe system is currently unavailable", CLI: "Error: The MC is not ready. Wait a few seconds then retry the request. (2020-02-10 12:19:34)". Is controller B dead? Is there something like hard reset? Do I have to replace this controller? Many Thanks! Roland

[SahanaR]

Hi Ronald,

If there is only issue in accessing the management controller(mc),we can restart mc B from cli of controller A.

#restart mc B

If you would like to reboot the storage controller (SC)B.This will cause temporary loss of data availability

#restart sc B

For more info please check page 147 https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c03791989

Thanks

Sahana R

[R-Sommer]

# restart mc b
During the restart process you will briefly lose communication with the specified Management Controller(s).
Do you want to continue? (y/n) y
Error: There was a failure to communicate with the remote MC ; MC restart failed. (2020-02-10 14:55:02)

[R-Sommer]

# restart sc b
Error: The Storage Controller is not ready for the operation. (2020-02-10 14:56:33)

[SahanaR]

Hi,

If you are able to login to controller A GUI or CLI,could you please check what is the status of controller B from there.

#show controllers

You could also try to reseat the controller B.If no LEDs on controller B after the reseat,may be its gone bad and needs replacement.

Thanks

[Shawn_K]

Hello,

Likely the two controllers are not in sync. Unfortunately without access to the system it is hard to tell what is occurring. I woudl suggest a period of downtime to try the following.

Shutdown both controllers. Controller B may not accept the command but Controller A will and will be shutdown correctly.

CLI# shutdown both

Once the controllers have shutdown remove power. Unseat controller A and all the hard drives. You don't have to remove them put disengage them from the backplane. If there are additional drive enclosures, label the cables and unplug the connecting SAS cables. You should be left witth just the raid head enclosure and controller B. Reconnect the power and bring controller B back online.

Once the controller has powered up fully, check if there is any unwritten cache.

# show unwritable-cache

Clearing unwritable cache can cause data loss, so be careful here on next steps. If there is cache you need to check and confirm whether that cache has already been written to a volume or not.  Once you confirm the cache status as 0 set the controller to single controller mode as you did previously.

CLI # set advanced-settings single-controller enabled

You will then be able to upgrade controller B. Once the controller is upgraded power down and reconnect everything you previously disconnected. Powering back on will resync the controllers.

If you power down and still cannot access controller B it will likely need a factory reset which is not something a customer can perform. You will need to log a support ticket for a new controller.

Good Luck!

Cheers,
Shawn

I work for Hewlett Packard Enterprise. The comments in this post are my own and do not represent an official reply from HPE. No warranty or guarantees of any kind are expressed in my reply.

[R-Sommer]

Hello Shawn,

I've removed power, controller A and all disks. But controller does not come up even no ping reply.

The system is out of service, which makes more sense: Let your colleagues repair it or just replace the controller with a new one?

Many thanks!

Roland

[Shawn_K]

Hello,

This sounds like your controller needs a factory reset. If you want you can buy a limited support contract to get a replacement controller. Or buy a new one.

Sorry sounds painful but I believe you need to replace controller B.

Cheers,
Shawn

I work for Hewlett Packard Enterprise. The comments in this post are my own and do not represent an official reply from HPE. No warranty or guarantees of any kind are expressed in my reply.