Query: MSA 2040 - The virtual pool is missing disks

[OS12429]

Hello, need help with HPE MSA 2040.

We had an unexpected situation with the HP MSA 2040 after a power failure. 

Currently, the cluster disk is offline and MSA shows an error - Virtual pool is missing disks. Ensure all disks are plugged and all enclosures are connected and powered on.

Although there is nothing wrong in the logs, and the system shows that all disks are UP (except for one, which was successfully overwritten on GLobal Spare a few months ago).
Is there a way to restore MSA to normal state without losing data?

[support_s]

System recommended content:

1. HPE MSA 2040 SAN Storage - Configuration

2. HPE MSA 2040 User Guide

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[OS12429]

Thanks for your help, but the error is still not resolved.
There is no description of this problem in the manuals.

Disk group in FTOL status, Health - OK. But the virtual pool is in Fault status - The virtual pool is missing disks.

[ArunKKR]

Hi,

Could you launch an SSH session to controller IP and share the output of below commands after removing serial number information from command output?

show disks

show versions

show pools

show disk-groups

show disks encl

[ArunKKR]

Hi,

We are awaiting your response. Please confirm status of your query.

[OS12429]

HI,

Comand output:

Version: GL210R004
# show disks
Location Serial Number Vendor Rev Description Usage Jobs
Speed (kr/min) Size Sec Fmt Disk Group Pool Tier Health
-------------------------------------------------------------------------------
1.1 HP HPDC SAS VIRTUAL POOL
10 600.1GB 512n dg1 A Standard OK
1.2 HP HPD5 SAS VIRTUAL POOL
10 600.1GB 512n dg1 A Standard OK
1.3 HP HPD5 SAS VIRTUAL POOL
10 600.1GB 512n dg1 A Standard OK
1.4 HP HPD5 SAS VIRTUAL POOL
10 600.1GB 512n dg1 A Standard OK
1.5 HP HPD5 SAS VIRTUAL POOL
10 600.1GB 512n dg1 A Standard OK
1.6 HP HPD5 SAS VIRTUAL POOL
10 600.1GB 512n dg1 A Standard OK
1.7 HP HPD5 SAS VIRTUAL POOL
10 600.1GB 512n dg1 A Standard OK
1.8 HP HPD5 SAS VIRTUAL POOL
10 600.1GB 512n dg1 A Standard OK
1.9 HP HPD2 SAS VIRTUAL POOL
10 600.1GB 512n dg1 A Standard OK
1.10 HP HPDE SAS GLOBAL SP
10 600.1GB 512n N/A OK
1.11 HP HPD5 SAS VIRTUAL POOL
10 600.1GB 512n dg1 A Standard OK
1.12 HP HPD5 SAS VIRTUAL POOL
10 600.1GB 512n dg1 A Standard OK
1.13 HP HPDE SAS VIRTUAL POOL
10 600.1GB 512n dg1 A Standard OK
-------------------------------------------------------------------------------

Success: Command completed successfully

# show pools
Name Serial Number Class Total Size Avail Snap Size OverCommit Disk Groups Volumes Low Thresh Mid Thresh High Thresh Sec Fmt Health
Reason
Action
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
A Virtual 3594.9GB 1298.4GB 0B Enabled 1 1 25.00 % 50.00 % 94.02 % 512n Fault
The virtual pool is missing disks.
- Ensure all disks are plugged in and all enclosures are connected and powered on.
- If this does not resolve the problem, look for events in the event log related to quarantine (172 or 485) and follow the recommended actions for those events.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully

# show disk-groups
Name Size Free Class Pool Tier % of Pool Own Pref RAID Disks Spr Chk Status Jobs Job% Serial Number Spin Down
SD Delay Sec Fmt Health Reason Action
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dg1 3594.9GB 1298.4GB Virtual A Standard 100 A A RAID10 12 0 512k FTOL Disabled
0 512n OK
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully.

# show disks encl
Status Encl Slot Vendor Model Size
------------------------------------------------------------------------------------
Up 1 1 HP EG0600FBVFP 600.1GB
Up 1 2 HP EG0600FCVBK 600.1GB
Up 1 3 HP EG0600FCVBK 600.1GB
Up 1 4 HP EG0600FCVBK 600.1GB
Up 1 5 HP EG0600FCVBK 600.1GB
Up 1 6 HP EG0600FCVBK 600.1GB
Up 1 7 HP EG0600FCVBK 600.1GB
Up 1 8 HP EG0600FCVBK 600.1GB
Up 1 9 HP EG000600JWJNH 600.1GB
Up 1 10 HP EG0600FBVFP 600.1GB
Up 1 11 HP EG0600FCVBK 600.1GB
Up 1 12 HP EG0600FCVBK 600.1GB
Up 1 13 HP EG0600FBVFP 600.1GB
Not Present 1 14 N/A N/A N/A N/A
Not Present 1 15 N/A N/A N/A N/A
Not Present 1 16 N/A N/A N/A N/A
Not Present 1 17 N/A N/A N/A N/A
Not Present 1 18 N/A N/A N/A N/A
Not Present 1 19 N/A N/A N/A N/A
Not Present 1 20 N/A N/A N/A N/A
Not Present 1 21 N/A N/A N/A N/A
Not Present 1 22 N/A N/A N/A N/A
Not Present 1 23 N/A N/A N/A N/A
Not Present 1 24 N/A N/A N/A N/A
------------------------------------------------------------------------------------
Success: Command completed successfully.

[ArunKKR]

Hi,

Please a try a graceful power cycle after shutting down controllers to check whether it resolves the issue.   

1. Shut down both the controllers using the below command: 

 shutdown both

2. Allow 5-7 minutes for controllers to shut down and OK to remove LED to light up on controller  3. Remove the power to both power supply units.

4. Reconnect the power after 2-3 minutes

5. Allow 5-7 minutes for controllers to initialize and verify the status 

Please log an HPE support case if the issue persists as debug log analysis would be needed to suggest next actions

[OS12429]

@ArunKKR Thanks for reply.

But gracefull shutdown doesn't work. The command runs for about 5 minutes and ends with an error, the controllers don't shut down completely...

[ArunKKR]

Hi,

In that case, I would suggest to log an HPE support case first to get the logs reviewed before trying a power cycle.