Big problem after firmware upgrade of MSA 2060

[dmontalan]

Hello everyone,

We recently upgraded our MSA 2060 SAN array to the IN110P002 firmware.

Since the firmware upgrade, we have encountered numerous error messages on the interface, as well as an inability to upgrade the firmware again:

https://ibb.co/4tyY1fH 

https://ibb.co/qFdg2zS 

When trying to reinstall the firmware : https://ibb.co/xSpgjLC 

Of course, we called HPE support, but the answer from L2 was: you have to change the chassis...

Changing the chassis means a huge production stoppage, for a simple firmware upgrade...

Before considering this solution, I'd like to know if any of you have already encountered this problem? And if, by chance, some of you have been able to solve the problem without replacing everything...

Many thanks to those who will read this...

[support_s]

System recommended content:

1. HPE MSA 2060 / 1060 - Supercapacitor has reached end of life event message in firmware version: INS100R02-01

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[dmontalan]

Thank you very much for your reply, but what does this web page have to do with my problem?

The SAN bay was already in version IN110P001 before the upgrade to the problematic IN110P002 firmware.

[JonPaul]

@dmontalan 
Replacing hardware is not what I would do as a first step.
Looking at the versions you show on your controller, it does not appear that the controller is the problem.

Do you have expansion enclosures added?
The firmware versions, 'EC', is one area which may be causing the problem.  The EC version should match what is on the main controller, 5352.
My suggestion would be to push back against the support team about replacing hardware, it may be the correct thing but updating to the latest firmware is the correct thing to do and if the issue is NOT a hardware problem then you will be in the same situation when you upgrade again.
Please look at these CLI command outputs:
CLI>  show redundancy-mode
CLI>  show versions detail   (already have that)
CLI>  show enclosures

[Sunitha_Mod]

Hello @dmontalan,

Let us know if you were able to resolve the issue.

If you have no further query and you are satisfied with the answer then kindly mark the topic as Solved so that it is helpful for all community members.

[dmontalan]

Sorry for the late return,

No the problem is not solved but we have an idea of the problem:

We actually have an extension bay and as mentioned before it seems that it's not the same version as the main bay.

Main bay : Model MSA2060-ENCL-SFF and revision 5352
Expansion bay: MSA-12G-ENCL-SFF and revision 533D

I guess the problem comes from this 5352/533D difference.

The problem is that I don't know how to correct it...

[JonPaul]

@dmontalan 
Yes, you have diagnosed the issues correctly.  The system is showing 'Invalid Bundle' for the version, right?
The system components must all match a version manifest and the EBOD EC firmware is not matching.
Changing the chassis IS NOT the fix.
A. can you send the case ID?
B. can you send the output of the commands above and also the output of   
CLI>  show frus
CLI>  show advanced-settings

[dmontalan]

Thank you for your feedback,

The case ID is 5375262258.

I have sent you the results of the orders by private message.

[jwalker3121]

We had the same issue happen with us, did you ever figure out what the fix was?

The firmware updated successfully on Controller A, but is failing on controller B. The firmware (IN210R004) is showing up on Controller B, but isn't able to be activated.

[JonPaul]

@jwalker3121 
I believe the original problem was resolved but I'm not sure that the problem is the same as what you are experiencing.  That was an update to IN110P002 and yours is IN210R004.  While they may look the same they could be very different.
A support call would be a good place to start.
Can you also respond with the output of the following commands:
CLI>  show redundancy-mode
CLI>  show versions detail
CLI>  show enclosures
CLI>  show firmware-bundles      {from each controller, each one maintains an independent list which is usually sync'd in good behaving systems}

FYI - the system CLI may ask for a confirmation on a command.  In IN110.. and before the confirmation may have appeared to be localized for language. IT IS NOT.  So if your locale is French and the confirmation asks for Oui/Non (o/n)  you must respond in English Yes/No (y/n) or you will be cancelling the command.
IN210 and on will show the non-localized '(y/n)' a and the CLI guide now states that all responses are required in English.

[jwalker3121]

Hi JonPaul,

Funny enough, about 5 minutes after I posted this, the controller firmware updates finally went through after two days of troubleshooting. We still have Drive firmware updates to do, and won't be able to go through with those until a scheduled downtime on Monday. I'm not quite sure what the fix was, but we found it! If we can pinpoint what it was, I'll respond to this post.

I appreciate the quick response, enjoy the rest of your week!

[JonPaul]

@jwalker3121 
Great to hear.  What version of firmware were you updating from?
Do you have expansion enclosures? 
The system has a feature called  "Partner-Firmware-update" (PFU) that is enabled by default.  This will keep trying to get the firmware on the system consistent across the 2 controllers and whole to a bundle.  Likely a restart triggered PFU refresh and finished the update.