Query: FRU error won't clear on HPE MSA2062

[Mike Texter]

Hi, I have an MSA2062 with an alert that won't clear.

"Unable to obtain FRU ID data for this FRU" for iom_2.b  (we're running 1 expansion shelf off of the head).  We've tried everything short of replacing it.  Anyone run across this before?  Have I stumbled upon a bug?  Array appears to be fully functional

After a re-seat of the IOM in question, we got the following alert email:
"

Alert:

Unable to obtain FRU ID data for this FRU.

 - The FRU ID content was not read. This could be temporary or it may not programmed correctly.

- If this persists, contact technical support.

CONTROLLER ID:# A

COMPONENT BASETYPE: controller

COMPONENT NAME: Enclosure 1, Controller B

DETECTED TIME: 2022-03-21 02:22:27

"

^ That doesn't seem to suggest a problem with IOM_2B  (expansion shelf, bottom IOM)  How does an alert show Controller ID: A, but Component Name: Enclosure 1 Controller B?  Make it make sense!

Thanks for any input.

P.S. the severity is an amber square with a minus sign inside, which does not appear in the manual as a valid severity level for any event, so that's cool. 

[support_s]

System recommended content:

1. HPE ProLiant ML / DL Gen9 Server - How to Clear the NVRAM from the System Maintenance Switch

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[Mike Texter]

I don't think that switch exists on an MSA...

Edit:  bad bot.

[feigenL]

Any recent hardware changes on the MSA?

If you inspect both controllers, do you see any ambar or error LED?

Have you recently upgraded the firmware on it? 

Does it show this alert since the beginning of time?

Have you restarted the controllers, one at a time, to see if the error clears?

Have you compared what is shown on the SMU is also shown on CLI?

can you execute this commands:

show alerts
show controllers
show frus

If the output shows the same errors and messages, I'd open a support case, as it might be a bug, or that the wrong controller was installed on the MSA.

Regards.

[Mike Texter]

Luis, thanks for that detailed reply, that's all great stuff to try.  Replies next to your questions I copied and pasted below (in bold

Any recent hardware changes on the MSA?        It's a brand-new unit, freshly installed with an LFF expansion

If you inspect both controllers, do you see any ambar or error LED?    Neither

Have you recently upgraded the firmware on it?        Yep, right when we installed it about 12 days ago

Does it show this alert since the beginning of time?      nope, it appeared on 3/11, suspiciously around the same time as the firmware update, i'll try to get more precise detail on the timing there.

Have you restarted the controllers, one at a time, to see if the error clears?   We haven't, that's a good idea.

Have you compared what is shown on the SMU is also shown on CLI?      We'll want to try that too.

can you execute this commands:

show alerts
show controllers
show frus

^Having our customer pull these today hopefully^

If the output shows the same errors and messages, I'd open a support case (already open!), as it might be a bug (my thoughts too, but if the CLI and SMU are showing the same error, wouldn't that suggest something's actually wrong?  I'd be more inclined to think it's a bug if the CLI did not show the same error.), or that the wrong controller was installed on the MSA (doubtful but possible I guess).

Regards.

Thank you again, that's a great place to start.

[ArunKKR]

Hi, 

Try to restart storage controller B and if issue persists most probably IO module will require replacement.

[Mike Texter]

Here's the output from these commands:  (prepare for copypasta)

================show alerts==============

#show alerts

ID Component Severity Resolved Acknowledged Reason
Recommended Action
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0 iom_2.b WARNING No Yes Unable to obtain FRU ID data for this FRU.
- The FRU ID content was not read. This could be temporary or it may not programmed correctly.

- If this persists, contact technical support.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully. (2022-03-22 21:40:35)

================show controllers=============

# show controllers
Controller
----------
Controller ID: A
Serial Number: Confidential info erased
Hardware Version: 5.0
CPLD Version: 2.8
MAC Address: 00:C0:FF:64:07:AC
WWNN: 500C0FF642090000
IP Address: 172.20.50.26
IP Subnet Mask: 255.255.255.0
IP Gateway: 172.20.50.1
IP6 Link Local: fe80::2c0:ffff:fe64:7ac
IP6 Link Local GW: ::
Autoconfig: Enabled
DHCPv6: ::
SLAAC IP Address: ::
IP6 Auto Gateway: ::
Disks: 17
Virtual Pools: 1
Disk Groups: 3
System Cache Memory (MB): 12288
Host Ports: 4
Disk Channels: 2
Disk Bus Type: SAS
Status: Operational
Failed Over to This Controller: No
Fail Over Reason: Not applicable
Multi-core: Enabled
Health: OK
Health Reason:
Health Recommendation:
Position: Top
Phy Isolation: Enabled
Controller Redundancy Mode: Active-Active ULP
Controller Redundancy Status: Redundant

Controller
----------
Controller ID: B
Serial Number: Confidential info erased
Hardware Version: 5.0
CPLD Version: 2.8
MAC Address: 00:C0:FF:64:07:CE
WWNN: 500C0FF642090000
IP Address: 172.20.50.27
IP Subnet Mask: 255.255.255.0
IP Gateway: 172.20.50.1
IP6 Link Local: fe80::2c0:ffff:fe64:7IP6 Link Local GW: ::
Autoconfig: Enabled
DHCPv6: ::
SLAAC IP Address: ::
IP6 Auto Gateway: ::
Disks: 17
Virtual Pools: 1
Disk Groups: 3
System Cache Memory (MB): 12288
Host Ports: 4
Disk Channels: 2
Disk Bus Type: SAS
Status: Operational
Failed Over to This Controller: No
Fail Over Reason: Not applicable
Multi-core: Enabled
Health: OK
Health Reason:
Health Recommendation:
Position: Bottom
Phy Isolation: Enabled
Controller Redundancy Mode: Active-Active ULP
Controller Redundancy Status: Redundant

Success: Command completed successfully. (2022-03-22 21:19:32)
#

=============show frus================

# show frus
SKU
---
Part Number: R0Q84A
Serial Number: Confidential info erased
Revision:
Enclosure ID: 1

SKU
---
Part Number: R0Q39A
Serial Number: Confidential info erased
Revision:
Enclosure ID: 2

FRU
---
Name: CHASSIS_MIDPLANE
Description: SPS-MSA 2060 SFF Chassis
Part Number: P12942-001
Serial Number: Confidential info erased
Revision: B
Dash Level:
FRU Shortname:
Manufacturing Date: 2021-05-11 04:13:00
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: MID-PLANE SLOT
Configuration SN: Confidential info erased
FRU Status: OK
Enclosure ID: 1

FRU
---
Name: RAID_IOM
Description: SPS-MSA 2060 SAS Controller
Part Number: P12950-001
Serial Number: Confidential info erased
Revision: C
Dash Level:
FRU Shortname:
Manufacturing Date: 2021-05-08 19:35:00
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: UPPER IOM SLOT
Configuration SN: Confidential info erased
FRU Status: OK
Enclosure ID: 1

FRU ---
Name: RAID_IOM
Description: SPS-MSA 2060 SAS Controller
Part Number: P12950-001
Serial Number:Confidential info erased
Revision: C
Dash Level:
FRU Shortname:
Manufacturing Date: 2021-05-08 18:39:00
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: LOWER IOM SLOT
Configuration SN: Confidential info erased
FRU Status: OK
Enclosure ID: 1

FRU
---
Name: POWER_SUPPLY
Description: SPS-MSA PWR supply OneStor 580W AC
Part Number: P12954-001
Serial Number: Confidential info erased
Revision: B
Dash Level:
FRU Shortname:
Manufacturing Date: N/A
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: LEFT PSU SLOT
Configuration SN: Confidential info erased
FRU Status: OK
Original SN: Confidential info erased
Original PN: 0992886-09
Original Rev:
Enclosure ID: 1

FRU
---
Name: POWER_SUPPLY
Description: SPS-MSA PWR supply OneStor 580W AC
Part Number: P12954-001
Serial Number: Confidential info erased
Revision: B
Dash Level:
FRU Shortname:
Manufacturing Date: N/A
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: RIGHT PSU SLOT Configuration SN: Confidential info erased
FRU Status: OK
Original SN: Confidential info erased
Original PN: 0992886-09
Original Rev:
Enclosure ID: 1

FRU
---
Name: CHASSIS_MIDPLANE
Description: SPS-MSA 2060 LFF Chassis
Part Number: P12941-001
Serial Number: Confidential info erased
Revision: B
Dash Level:
FRU Shortname:
Manufacturing Date: 2020-08-08 18:43:00
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: MID-PLANE SLOT
Configuration SN: Confidential info erased
FRU Status: OK
Enclosure ID: 2

FRU
---
Name: BOD_IOM
Description: SPS-MSA 12G IO Module
Part Number: P12951-001
Serial Number: Confidential info erased
Revision: A
Dash Level:
FRU Shortname:
Manufacturing Date: 2020-07-19 03:45:00
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: UPPER IOM SLOT
Configuration SN: Confidential info erased
FRU Status: OK
Enclosure ID: 2

FRU
---
Name: BOD_IOM
Description: SPS-MSA 12G IO Module
Part Number: P12951-001
Serial Number: Confidential info erased
Revision: A
Dash Level: FRU Shortname:
Manufacturing Date: 2020-07-19 14:20:00
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: LOWER IOM SLOT
Configuration SN: Confidential info erased
FRU Status: OK
Enclosure ID: 2

FRU
---
Name: POWER_SUPPLY
Description: SPS-MSA PWR supply OneStor 580W AC
Part Number: P12954-001
Serial Number: Confidential info erased
Revision: B
Dash Level:
FRU Shortname:
Manufacturing Date: N/A
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: LEFT PSU SLOT
Configuration SN: Confidential info erased
FRU Status: OK
Original SN: Confidential info erased
Original PN: 0992886-09
Original Rev:
Enclosure ID: 2

FRU
---
Name: POWER_SUPPLY
Description: SPS-MSA PWR supply OneStor 580W AC
Part Number: P12954-001
Serial Number: Confidential info erased
Revision: B
Dash Level:
FRU Shortname:
Manufacturing Date: N/A
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: RIGHT PSU SLOT
Configuration SN: Confidential info erased
FRU Status: OK
Original SN: Confidential info erased
Original PN: 0992886-09
Original Rev:
Enclosure ID: 2
Success: Command completed successfully. (2022-03-22 21:20:54)

[Mike Texter]

restarting one controller at a time did not work   Still running it up the flagpole with HPE support.

[Mike Texter · Accepted solution]

So, this is fixed now.  Turns out when we re-seated IOM-B, what actually was re-seated was Controller B.  after IOM-B was re-seated, the alert went away.  Thanks all.

[feigenL]

Good to know that it is fixed now!