StoreOnce D2D4106fc pool offline

[DBlago]

Hello, 
D2D has showed an HDD fault indication in bay10. The Hotspare disk began to be used instead. In the DP Manager interface, the second drive of the virtual tape library D2D-IAS-D2 is in the Disable state. It has been turned on. In the DP Manager/Device&Media interface, this virtual tape library appears to be working properly, but backup sessions to it fail with an error. The library's web interface stopped responding and the library was hard rebooted by power button. During a long reboot of the library, the Raid was automatically reconfigured. The HDD fault indication is gone - all disks have the OK status. But now the status of the Fiber Chanel ports is Not used. Storage pool is in OffLine status.

Can anyone help me with the issue? 
Thanks in advance!

[Moderator edit: Removed the confidential info.]

[DBlago]

Have found this in Support Ticket. May be it is the reason?

[Mohammed_I]

Hi DBlago,

As per the screen shoots i see storage pool status if offline and system status is offline.

VTL is offline, cache on this controller has a problem.

In this my recommendation : 

• Check the SmartArraySOB.log (Gen2) to see if the logical drives have been disabled and reports F1/F2 option to re-enable the logical drive(s). 

Perform a reboot. During boot up the message “Press any key to view Option ROM messages” will appear on the Pro-Liant screen. 

Press any key. As the boot up process continues the Controller information will be displayed.

For more information you can refere the below link.

https://support.hpe.com/hpesc/public/docDisplay?docId=c03127437&docLocale=en_US

Even after the above steps if the issue is same capture the support ticket and update .

Thank you. HPE Community team.

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you have no further query, and you are satisfied with the answer then kindly mark the topic as Solved so that it is helpful for all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[DBlago]

@Mohammed_I 
Hello Mohammed,
Thank you for reply!
'Check the SmartArraySOB.log (Gen2) to see if the logical drives have been disabled and reports F1/F2 option to re-enable the logical drive(s). ' - have not actually got how to check it, found this in SmartArraySOB.log:

I can send you fresh Support Ticket if required. Also we have replaced FBWC (578882-001) and Smart Array P212 controller board (462594-001) and updated FW to 5.80, but stoage pool is still offline

[Mohammed_I]

Hello ,

• Press F2. This will re-enable the logical drive(s). The comment about accepting data loss refers to the data being held within the controller cache memory, it does not refer to the data already on the disks.

• If the logical drives were recovered then you will now need to wait for the D2D to rebuild the raid (complete parity initialization). Depending on the amount of stored data this may take approximately 48.00hrs to complete. Do not perform any reboots or try to perform any backups until the rebuild has completed.

• For smart array issues check the log “SmartArraySOB.log” (Gen2)
• For disk issues check the log ‘ticket.HPadu.xml’.
• Check the ‘SmartArraySOB.log’ for excessive HDD media errors.
• Check the ‘SmartArraySOB.log’ for controller resets, search for 'PALResetDevice'.
• Check the ‘SmartArraySOB.log’ for ECC errors.

[DBlago]

@Mohammed_I 

hi Mohammed, 
Thank you for AP and here is the result.

When booting the library, there are no messages about choosing F1 or F2, there is only F8 for configuring the Raid controller board.
In order to reproduce the malfunction in the Raid controller and the message about choosing F1 or F2 appeared on the switched off
library, we removed the disk in bay10 (pulling it out a little so as not to disrupt the cooling system).
When turned on, the Raid controller saw the absence of a disk and the F2 action was selected.
For choosing proper action there are only a few seconds to select this boot mode (F1 or F2), you do not have time to read the message in full.
Rebuilding Raid System was performed, the HotSpare disk was connected instead of the faulty disk.
As a result, the logical disk was rebuilt, but the Storage Pool remained offline (in the screenshot, these are the events from 17:03 2024/07/31 to 8:16 2024/08/01).
Then we turned off the library again, connected the disk back to bay10. When turned on, the Raid controller sees the configuration change again.
As a result, Rebuilding Raid System was executed again, the disk in bay10 returned to work, the HotSpare disk was freed.
On the screenshot, these are events from 9:20 to 11:43 2024/08/01.
As a result, the state of all components (Raid controller, hard drives, logical drives) is OK, but Storage Pool is offline.
Attempts to revive logical drives did not lead to a positive result.
Maybe export the current configuration, reset to factory settings and try to import the settings back?
Thank you in advance!

[DBlago]

@Mohammed_I 
hi Mohammed, 
please advice how to resolve the issue.
thank you in advance!

[Mohammed_I]

Hi DBlago,

In this case frist thing i need support ticket to check the status. Make sure production date is set correctly

(Run the commend 'set production wwnseed <Appliance Node WWN>') check the status.

Once i have logs i can update you the next action plan .

Thank you ..

[DBlago]

@Mohammed_I 
hi Mohammed,

have sent you the support ticket.

thank you!

[Mohammed_I]

Hi DBlago,

I dont see support ticket could you share me the details  again .

Awaiting for support ticket .

Thank you ..

[DBlago]

@Mohammed_I 
hi Mohammed,
i have sent you ST via private message. please check. 
thank you!

[Mohammed_I]

Hi DBlago

As per logs:
RAID Mounted Yes
FileSystem Mounted No
Possible corrupted file system, review file ticket.SystemLogs2.txt
Number of Libraries Configured 1
All Configured Libraries/Shares Online No
All Process Running OK Yes
Disk Status GOOD
Disk Usage 0.00 Bytes / 0.0%
Any Physical Tape/Library seen? No
Replication Job for Libraries Configured? No
Deduplication Ratio 1.0 : 1
Low overall dedupe ratio
Dedupe Manager Dedupe is not running
Mount points Mount point(s) [a][h] missing. Contact higher resource

I see controller fimrware 5.80 -> P212 it should be 5.12.
For now i would request you to down grade the P212 firmware to 5.12.

01-08-2024 09:32:16 Embeded image System Status: Warning
01-08-2024 09:32:14 Embeded image Rebuilding RAID System
31-07-2024 17:13:24 Embeded image System Status: Warning
31-07-2024 17:13:21 Embeded image Disk failure has occurred but the RAID System is still operational with full capacity
31-07-2024 17:13:21 Embeded image Rebuilding RAID System
12-07-2024 15:01:40 Embeded image System Status: Warning

01-08-2024	12:27:46		Storage Online
01-08-2024	12:27:45		System Status: Good
01-08-2024	12:27:29		Daylight Savings or Timezone changed
01-08-2024	12:27:29		System Starting
01-08-2024	12:27:29		********************************************
01-08-2024	12:22:46		All processes stopped
01-08-2024	12:22:18		Stopping System
01-08-2024	12:22:18		System restarting
01-08-2024	11:43:52		System Status: Good
01-08-2024	11:43:48		The Storage RAID Logical Drive 600508B1001CF5A89EFE59ADFF8F2A3E has been rebuilt

Failed to recover EFIs on filesystem: cciss/c0d1p1
XFS: log mount finish failed
Filesystem "cciss/c0d1p5": Disabling barriers, trial barrier write failed
Filesystem "cciss/c0d1p1": Disabling barriers, trial barrier write failed
XFS mounting filesystem cciss/c0d1p5
XFS mounting filesystem cciss/c0d1p1
Starting XFS recovery on filesystem: cciss/c0d1p1 (logdev: internal)
Starting XFS recovery on filesystem: cciss/c0d1p5 (logdev: internal)
XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1599 of file fs/xfs/xfs_alloc.c.  Caller 0xffffffff881b4961

Action plan down grade controller firmware to 5.12 and check the status. 

Hope you have data back up . 

[DBlago]

@Mohammed_I 
hi Mohammed,
the box have been upgraded by using D2D G2 Component Firmware ver. 20130806, which included controller firmware ver. 5.80: https://support.hpe.com/connect/s/softwaredetails?language=en_US&collectionId=MTX-ee8ba2e129e74c03

And also web gui states it should be FW 5.80 for controller:
 

[Mohammed_I]

Hi DBlago 

I have enclosed the detail as per the logs.

How ever if you want to restore to factory you need have USB flash stick that has OS .

Hope you have data backed up, 

SPS-D2D 4106FC G2 USB Flashstick

625740-010