HP P2000 G3 iSCSI MSA Update Firmware Fault

[AlexMn]

I tried to upgrade the firmware on my HP P2000 G3 10GbE iSCSI MSA Dual Controller SFF Array System from firmware TS251P006-02 to TS252R007.

The first controller has been updated successfully.
The second controller returned an error.

Now in Storage Management Utility controller B status - Fault.

Restart the firmware update does not help.
The Smart Component was unable to retrieve all the hardware information from the Storage Array.
 
This may be a result of the Storage Array being in a degraded state or The Smart Component may not be applicable to this hardware.

Installation log: 

Spoiler

Starting discovery...

Controller A Found - IP_Address FW Ver: TS251P006-02
Controller A Status - Operational
Controller B Found - IP_Address FW Ver: TS251P006-02
Controller B Status - Operational
Restarting MC.. Please Wait...
Please Wait..
Please Wait..
Checking if controller is UP after restart ...
Please Wait ..
Controller is Reachable ..
Controller A Found - IP_Address FW Ver: TS251P006-02
Controller A Status - Operational
Controller B Found - IP_Address FW Ver: TS251P006-02
Controller B Status - Operational
Flashing Array to firmware version : TS252R007
Flashing controller: B
Connecting to controller...
Sending Firmware files to controller.
Connected to the controller.
Logged into the controller.
File uploaded, bytes written: 38328320
File uploaded, bytes written:
File Transfer Complete. Starting Operation:
Checking component list
mc bundle component check passed.
Checking bundle integrity...
Initial mc file integrity checks passed.
Checking system health.
System health check complete. Health state: OK
Stopping Management Controller applications.
Starting message server
Initial connection to SC successful.
STATUS: Updating Expander Controller firmware.
Firmware for Expander Controller was not updated because the new version is the same as the current version. (enclosure number = 1, module ID = B, version = 2028)
Expander Controller firmware was not updated.
Finished updating EC firmware.
STATUS: Updating expansion-module firmware.
Performed rescan at the beginning of the expansion-module firmware update. Waiting a few seconds...
No expansion enclosures were found.
Finished updating expansion-module firmware.
STATUS: Updating Storage Controller firmware.
Controller current bundle: TS251P006-02,loading bundle TS252R007
Instructing SC to shut down and reboot when finished updating
Waiting 5 seconds for SC to shutdown.
Shutdown of SC successful.
Sending new firmware to SC.
Updating SC Image:Remaining size 6826493
Updating SC Image:Remaining size 6498813
Updating SC Image:Remaining size 6171133
Updating SC Image:Remaining size 5843453
Updating SC Image:Remaining size 5515773
Updating SC Image:Remaining size 5188093
Updating SC Image:Remaining size 4860413
Updating SC Image:Remaining size 4532733
Updating SC Image:Remaining size 4205053
Updating SC Image:Remaining size 3877373
Updating SC Image:Remaining size 3549693
Updating SC Image:Remaining size 3222013
Updating SC Image:Remaining size 2894333
Updating SC Image:Remaining size 2566653
Updating SC Image:Remaining size 2238973
Updating SC Image:Remaining size 1911293
Updating SC Image:Remaining size 1583613
Updating SC Image:Remaining size 1255933
Updating SC Image:Remaining size 928253
Updating SC Image:Remaining size 600573
Updating SC Image:Remaining size 272893
Waiting for Storage Controller to complete programming.
Please wait...
Storage Controller has completed programming.
Updating SC Image:Remaining size 0
Waiting for SC reboot.
Please wait...
.....................................SC has not rebooted after 240 seconds. Quitting.Storage Controller code failed to load. Stopping firmware update process.
MC will restart. Please reapply the firmware update once the MC has restarted.
==========================================
Software Component Load Summary:
MC Software: Unknown
SC Software: FAILED
EC Software: NOT ATTEMPTED
Expansion Software: NOT ATTEMPTED
==========================================
STATUS: *** Code Load Fail. ***
Finished applying firmware..

[AnkitM]

Add below outputs

#show configuration

#show events (pipe it to 200 event)

[AlexMn]

# show configuration

Spoiler

System Information
------------------
System Name: Uninitialized Name
System Contact: Uninitialized Contact
System Location: Uninitialized Location
System Information: Uninitialized Info
Midplane Serial Number: 00C0FF159755
Vendor Name: HP
Product ID: P2000 G3 iSCSI
Product Brand: MSA Storage
SCSI Vendor ID: HP
SCSI Product ID: P2000 G3 iSCSI
Enclosure Count: 1
Health: Degraded
Health Reason: A subcomponent of this component is unhealthy.
Supported Locales: English (English), Spanish (español), French (français), German (Deutsch), Italian (italiano), Japanese (日本語), Dutch (Nederlands), Chinese-Simplified (简体中文), Chinese-Traditional (繁體中文), Korean (한국어)

Unhealthy Component
-------------------
Component ID: Enclosure 1, Controller B
Health: Fault
Health Reason: The controller is not operational.
Health Recommendation: - Restart the Storage Controller in this controller module, unless it is performing an operation where it is normal for it to be shutdown, such as firmware update.

Unhealthy Component
-------------------
Component ID: Enclosure 1, Current 12V Loc: lower-IOM B
Health: Degraded
Health Reason: The current sensor's status is unknown or its value is unexpected. It may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Enclosure 1, Temperature Loc: lower-IOM B
Health: Degraded
Health Reason: The temperature sensor's status is unknown or its value is unexpected. It may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Enclosure 1, Voltage 12V Loc: lower-IOM B
Health: Degraded
Health Reason: The voltage sensor's status is unknown or its value is unexpected. It may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Unhealthy Component
-------------------
Component ID: Enclosure 1, Voltage 5V Loc: lower-IOM B
Health: Degraded
Health Reason: The voltage sensor's status is unknown or its value is unexpected. It may be unhealthy.
Health Recommendation: - Check in the event log for events related to this component and follow the recommended actions for those events.

Controllers
-----------
Controller ID: A
Serial Number: SN
Hardware Version: 53
CPLD Version: 22
MAC Address: 00:C0:FF:15:B8:20
WWNN: 208000c0ff159755
IP Address: IP_Address
IP Subnet Mask: Mask
IP Gateway: IP_Address
Disks: 21
Vdisks: 0
Cache Memory Size (MB): 2048
Host Ports: 2
Disk Channels: 2
Disk Bus Type: SAS
Status: Operational
Failed Over to This Controller: Yes
Fail Over Reason: Boot handshake timeout
Health: OK
Health Reason:
Health Recommendation:
Position: Top
Phy Isolation: Enabled
Controller Redundancy Mode: Fail Over
Controller Redundancy Status: Operational but not redundant

Controllers
-----------
Controller ID: B
Serial Number:
Hardware Version:
CPLD Version:
MAC Address: 00:00:00:00:00:00
WWNN: 208000c0ff159755
IP Address: IP_Address
IP Subnet Mask: Mask
IP Gateway: IP_Address
Disks: 21
Vdisks: 0
Cache Memory Size (MB): 2048
Host Ports: 2
Disk Channels: 2
Disk Bus Type: SAS
Status: Down
Failed Over to This Controller: No
Fail Over Reason: Not applicable
Health: Fault
Health Reason: The controller is not operational.
Health Recommendation: - Restart the Storage Controller in this controller module, unless it is performing an operation where it is normal for it to be shutdown, such as firmware update.
Position: Bottom
Phy Isolation: Enabled
Controller Redundancy Mode: Fail Over
Controller Redundancy Status: Operational but not redundant

Controller A Versions
---------------------
Storage Controller CPU Type: Intel Tolapai 1200MHz
Bundle Version: TS252R007
Build Date: Fri May 8 15:40:57 MDT 2015
Storage Controller Code Version: T252R06-01
Storage Controller Code Baselevel: T252R06-01
Memory Controller FPGA Code Version: F400R02
Storage Controller Loader Code Version: 23.008
CAPI Version: 3.17
Management Controller Code Version: L252R008-01
Management Controller Loader Code Version: 2.5
Expander Controller Code Version: 2028
CPLD Code Version: 22
Hardware Version: 53
Host Interface Module Version: 52
Host Interface Module Model: 2
Backplane Type: 7
Host Interface Hardware (Chip) Version: 0
Disk Interface Hardware (Chip) Version: 3

Controller B Versions
---------------------
Storage Controller CPU Type: 0MHz
Bundle Version: TS251P006-02
Build Date: Thu Jul 24 10:44:54 MDT 2014
Storage Controller Code Version: Unknown
Storage Controller Code Baselevel:
Memory Controller FPGA Code Version:
Storage Controller Loader Code Version: Unknown
CAPI Version: 0.0
Management Controller Code Version:
Management Controller Loader Code Version:
Expander Controller Code Version:
CPLD Code Version:
Hardware Version:
Host Interface Module Version:
Host Interface Module Model:
Backplane Type: 0
Host Interface Hardware (Chip) Version: 0
Disk Interface Hardware (Chip) Version: 0

Ports Media Target ID Status Speed(A) Health Health Reason Health Recommendation
------------------------------------------------------------------------------------------------------------------------------------
A1 iSCSI iqn.1986-03.com.hp:storage.p2000g3.1235159755 Up 10Gb OK

IP-Ver PIP PIP-Gateway PIP-Netmask MAC
-------------------------------------------------------------------
IPv4 - 0.0.0.0 255.255.255.0 -

A2 iSCSI iqn.1986-03.com.hp:storage.p2000g3.1235159755 Up 10Gb OK

IP-Ver PIP PIP-Gateway PIP-Netmask MAC
-------------------------------------------------------------------
IPv4 - 0.0.0.0 255.255.255.0 00:C0:FF:18:56:11

------------------------------------------------------------------------------------------------------------------------------------
Ports Media Target ID Status Speed(A) Speed(C) Health Health Reason Health Recommendation
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
B1 iSCSI iqn.1986-03.com.hp:storage.p2000g3.1235159755 Disconnected 1Gb Auto N/A There is no host connection to this host port. - No action is required.

IP-Ver PIP PIP-Gateway PIP-Netmask MAC
-------------------------------------------------------------------
IPv4 - 0.0.0.0 255.255.255.0 00:00:00:00:00:00

B2 iSCSI iqn.1986-03.com.hp:storage.p2000g3.1235159755 Disconnected 1Gb Auto N/A There is no host connection to this host port. - No action is required.

IP-Ver PIP PIP-Gateway PIP-Netmask MAC
-------------------------------------------------------------------
IPv4 - 0.0.0.0 255.255.255.0 00:00:00:00:00:00

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Location Serial Number Vendor Rev How Used Type Size Rate*(Gb/s) SP Health Health Reason Health Recommendation
-----------------------------------------------------------------------------------------------------------------------------------------------
1.1 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.2 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.3 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.4 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.5 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.6 - HP HPDF AVAIL SAS 146.8GB 6.0 A OK
1.7 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.8 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.9 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.10 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.11 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.12 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.13 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.14 - HP HPD5 AVAIL SAS 146.8GB 6.0 A OK
1.18 - HP HPD5 AVAIL SAS 300.0GB 6.0 A OK
1.19 - HP HPD5 AVAIL SAS 300.0GB 6.0 A OK
1.20 - HP HPD0 AVAIL SAS 300.0GB 6.0 A OK
1.21 - HP HPD5 AVAIL SAS 300.0GB 6.0 A OK
1.22 - HP HPD5 AVAIL SAS 300.0GB 6.0 A OK
1.23 - HP HPD0 AVAIL SAS 300.0GB 6.0 A OK
1.24 - HP HPD5 AVAIL SAS 300.0GB 6.0 A OK
-----------------------------------------------------------------------------------------------------------------------------------------------
Status Encl Slot Vendor Model Serial Number Size
------------------------------------------------------------------------------------
Up 1 1 HP EG0146FARTR - 146.8GB
Up 1 2 HP EG0146FARTR - 146.8GB
Up 1 3 HP EG0146FARTR - 146.8GB
Up 1 4 HP EG0146FARTR - 146.8GB
Up 1 5 HP EG0146FARTR - 146.8GB
Up 1 6 HP EG0146FAWHU - 146.8GB
Up 1 7 HP EG0146FARTR - 146.8GB
Up 1 8 HP EG0146FARTR - 146.8GB
Up 1 9 HP EG0146FARTR - 146.8GB
Up 1 10 HP EG0146FARTR - 146.8GB
Up 1 11 HP EG0146FARTR - 146.8GB
Up 1 12 HP EG0146FARTR - 146.8GB
Up 1 13 HP EG0146FARTR - 146.8GB
Up 1 14 HP EG0146FARTR - 146.8GB
Not Present 1 15 N/A N/A N/A N/A
Not Present 1 16 N/A N/A N/A N/A
Not Present 1 17 N/A N/A N/A N/A
Up 1 18 HP EG0300FCVBF - 300.0GB
Up 1 19 HP EG0300FCVBF - 300.0GB
Up 1 20 HP EG0300FCSPH - 300.0GB
Up 1 21 HP EG0300FCVBF - 300.0GB
Up 1 22 HP EG0300FCVBF - 300.0GB
Up 1 23 HP EG0300FCSPH - 300.0GB
Up 1 24 HP EG0300FCVBF - 300.0GB
------------------------------------------------------------------------------------
Name Size Free Own Pref RAID Disks Spr Chk Status Jobs Job% Serial Number Drive Spin Down Spin Down Delay Health Health Reason Health Recommendation
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Encl Encl WWN Name Location Rack Pos Vendor Model EMP A CH:ID Rev EMP B CH:ID Rev Midplane Type Health Health Reason
Health Recommendation
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 500c0ff01597553c 0 0 HP SPS-CHASSIS 00:031 2028 unknown 2U24-6G Degraded A subcomponent of this component is unhealthy.
- No action is required.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SKU
---
Part Number: PN
Serial Number: SN
Revision: B

FRU
---
Name: CHASSIS_MIDPLANE
Description: SPS-CHASSIS 2U24 6G w/MIDPLANE
Part Number: PN
Serial Number: SN
Revision: A
Dash Level:
FRU Shortname: Midplane/Chassis
Manufacturing Date: 2012-09-04 21:26:46
Manufacturing Location: Longhua,Shenzhen,CN
Manufacturing Vendor ID: 0x017C
FRU Location: MID-PLANE SLOT
Configuration SN: SN
FRU Status: OK

FRU
---
Name: RAID_IOM
Description: HP P2000 G3 10GbE iSCSI MSA Controller
Part Number: PN
Serial Number: SN
Revision: B
Dash Level:
FRU Shortname: RAID IOM
Manufacturing Date: 2012-09-21 22:39:09
Manufacturing Location: Longhua,Shenzhen,CN
Manufacturing Vendor ID: 0x017C
FRU Location: UPPER IOM SLOT
Configuration SN: SN
FRU Status: OK

FRU
---
Name: IOM
Description:
Part Number:
Serial Number:
Revision:
Dash Level:
FRU Shortname:
Manufacturing Date: N/A
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: LOWER IOM SLOT
Configuration SN: SN
FRU Status: Absent

FRU
---
Name: POWER_SUPPLY
Description: FRU,Pwr Sply,595W,AC,2U,LC,HP
Part Number: PN
Serial Number: SN
Revision: A
Dash Level:
FRU Shortname: AC Power Supply
Manufacturing Date: 2012-09-19 05:52:55
Manufacturing Location: Zhongshan,Guangdong,CN
Manufacturing Vendor ID: 0x
FRU Location: LEFT PSU SLOT
Configuration SN: SN
FRU Status: OK
Original SN:SN
Original PN: SN
Original Rev: R

FRU
---
Name: POWER_SUPPLY
Description: FRU,Pwr Sply,595W,AC,2U,LC,HP
Part Number: PN
Serial Number: SN
Revision: A
Dash Level:
FRU Shortname: AC Power Supply
Manufacturing Date: 2012-09-19 06:17:38
Manufacturing Location: Zhongshan,Guangdong,CN
Manufacturing Vendor ID: 0x
FRU Location: RIGHT PSU SLOT
Configuration SN: SN
FRU Status: OK
Original SN: SN
Original PN: PN
Original Rev: R

FRU
---
Name: MEMORY CARD
Description: SPS Memory Card
Part Number: 768079-001
Serial Number:
Revision:
Dash Level:
FRU Shortname: Memory Card
Manufacturing Date: N/A
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: UPPER IOM MEMORY CARD SLOT
Configuration SN: SN
FRU Status: OK

FRU
---
Name: MEMORY CARD
Description:
Part Number:
Serial Number:
Revision:
Dash Level:
FRU Shortname:
Manufacturing Date: N/A
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: LOWER IOM MEMORY CARD SLOT
Configuration SN: SN
FRU Status: Absent

Info: * Rates may vary. This is normal behavior. (2015-06-26 23:29:35)

Success: Command completed successfully. (2015-06-26 23:29:35) 

[AnkitM]

Reseat Controller B.

Let m know if it works or not.

Also, It seems there is also a disk failure on enclosure: 1, slot: 8.

[AlexMn]

Do you mean restart (SMU - Tools -Shut Down or Restart Controller)?

If it, it doesn't help.

[AnkitM]

No, I meant Remove the Controller and Push it back in after 2mins (Reseat)

If it doesn't work: (maintenance Window Required)

1. Power Down MSA

2. Remove Controller A 

3. Power ON MSA on Controller B...Check if the Controller has Fault LED ON.

4. If yes, swap it with Slot A and Check if the Controller has Fault LED ON.

5. If yes, then mosntly your controller has faulted but if you still want to give it a go.

- a Flash the Controller Firmware via FTP

You can refer to Installation Instructions for FTP in ReleaseNotes.

 ftp://ftp.hp.com/pub/softlib2/software1/pubsw-linux/p1485086127/v109841/740672-010.html

[AlexMn]

It doesn't help.

I made a request to replace the controller.

Thanks for the advice.
This topic can be closed.