P2000 G3 MSA Firmware mismatch issue

[AdamDB]

We had an issue with an aging P2000 MSA G3 with dual combo ICSCI/FC controllers in it. One of the controllers failed. The array has continued to operate normally on one controller (this controller b was running old firmware T201P02 from 2011). A replacement controller was sent. We were told it was downgraded to be close to the existing controller. Partner firmware update was disabled. The replacement controller was placed into slot A. It turns out its running T251R04-01 which is from 2014.

My understanding is that is could be a problem and that no controller with a f/w newer than TS250 should not be placed into the chassis. Currently since the controller running T251R04-01 was inserted it is flagging all the SFF drives in its associated chassis as having an unknown state. It also appears to be constantly rebooting the management controller in b (running T201P02).  Its showing controller B (old firmware) as unavailble on the network. However i was able to ssh into it. It appears that it cannot see anything other then itself. SHOW FRUS just returns nothing and SHOW SYSTEM only returns basic chassis information, most of which is not populated. However the ISCSI ports appear to be online and working so it is operating, just not communucating with anything else.

Can anyone tell me if its safe to enable partner firmware update and allow the new controller running T251R04-01  and allow it to update controller B running T201P02?

The alternative it to take everything down (All the connected hosts) and then power down the storage and remove the controller running T251R04-1 and get it downgraded to a closer firmware before re-inserting.

The main shelf also has 5 x D2700 SAS AJ941A enclosures on it,

I am seeing the following messages....

Error logs…(repeatedly)

 Enclosure event: fault. (enclosure: 1, WWN: 500c0ff0125XXXXX) I/O module A, I/O module status: The firmware version is not compatible, HP SPS-CHASSIS 2028

Warning logs… (repeatedly)

 2016-05-18 06:35:30 B6585 156

The Management Controller was restarted automatically by the Storage Controller for the purpose of error recovery.

2016-05-18 06:35:30 B6584 152

The Storage Controller is not receiving data from the Management Controller. (This is normal during firmware update.)

  Is this what you would expect to see if partner firmware update is off?

#
#
# show system (run from CONTROLLER-A - new firmware)
System Information
------------------
System Name: SXXXXX
System Contact: XX
System Location: XXXXX
System Information: HP P2000 G3 FC iSCSI Combo
Midplane Serial Number: 00XXXXXX
Vendor Name: HP
Product ID: P2000G3 FC/iSCSI
Product Brand: MSA Storage
SCSI Vendor ID: HP
SCSI Product ID: P2000G3 FC/iSCSI
Enclosure Count: 6
Health: Degraded
Health Reason: A subcomponent of this component is unhealthy.
Supported Locales: English (English), Spanish (español), French (français), German (Deutsch), Italian (italiano), Japanese (æ¥æ¬èª), Dutch (Nederlands), Chinese-Simplified (ç®ä½ä¸­æ), Chinese-Traditional (ç¹é«ä¸­æ), Korean (íêµ­ì´)

Unhealthy Component
-------------------
Component ID: Enclosure 1, Current Health: Degraded
Health Reason: The current sensor's status is unknown or its value is unexpected. It may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.16
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.6
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended act
Unhealthy Component
-------------------
Component ID: Disk 1.4
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.19
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.24
Health: Degraded Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.11
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.13
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.
Unhealthy Component
-------------------
Component ID: Disk 1.20
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.8
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.3
Health: Degraded
Health Reason: The disk in this sloane may be unhealthy. Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.18
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.12
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component -------------------
Component ID: Disk 1.1
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.7
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.21
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy. Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.9
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.17
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
------------------- Component ID: Disk 1.15
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.2
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.5
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in tponent and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.23
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.22
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.10 Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Disk 1.14
Health: Degraded
Health Reason: The disk in this slot has unknown health. The disk or the midplane may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Enclosure 1, Controller B, Management Port
Health: Degraded
Health Reason: The network port Ethernet cable is unplugged, or the network is inoperable.
Health Recommendation: - Check that the controller network port is properly connected to the network. - If it is, then check for network problems.

Unhealthy Component
-------------------
Component ID: Enclosure 1, Temperature Loc: lower-IOM B
Health: Degraded
Health Reason: The temperature sensor's status is unknown or its value is unexpected. It may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Enclosure 1, Voltage 5V Loc: lower-IOM B
Health: Degraded
Health Reason: The voltage sensor's status is unknown or its value is unexpected. It may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Enclosure 1, Voltage Health: Degraded
Health Reason: The voltage sensor's status is unknown or its value is unexpected. It may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Success: Command completed successfully. (2016-05-18 07:53:08)

Any advice appreciated. thanks

Adam.

[Mike Texter]

If I'm not mistaken, when you enable partner firmware update, the existing controller (the one with 2011 firmware) will be the "boss" of the system, and the replacement controller will sync down to it's firmware.  (i'd remove the 2014 controller, then enable this feature, then re-insert).  I'd be awfully surprised if the existing controller upgraded itself to the 2014 version.

[Torsten.]

>> the existing controller (the one with 2011 firmware) will be the "boss" of the system, and the replacement controller will sync down to it's firmware

No. The newer firmware will be installed on the controller with older firmware.

[AdamDB]

Yes, i believe Torsten is correct. It now appears we can source a controller running the same firmware as the one installed on earlier this week. (T251R04). As A appears to be functional (can see the other controller and can report stats on itself) it seems to fine. It's B (running the old firmware) which is now in a pickle.  It can't talk to via the backplane to the other controller anymore is is effectively isolated. (although its working and serving data over its ports at the moment).I think this is because the firmware was changed significantly from TS230 to streamline the code over all the MSA devices. So the plan at the moment is to put another controller into B running T251R04 with the whole estate offline to minimise risk. Hopefully the config will copy over from A and everything will be healthly. From here we can then update in the normal way. My only concern is if the disk shevles don't like this but sadly I can't find a definitive answer as to if this will work from anywhere. If all else fails we still have our old controller B to fall back on. Unknown territory here but I will hope for the best. The primary driver here is to avoid losing the data obviously.

If only our supplier had done as we asked and shipped the original replacement with older firmware...sigh.

If anyone thinks this approach is flawed or has previous experience. please let me know.

thanks

Ad