MSA 2040 - Virtual Pool offline

[MP_KV]

Dear Team,

We have MSA 2040 storage system with capaicity around 30TB. Recently observed one of the disk is having an issue and we replaced with similar and compatible disk. Post replacement, the disk went into RCON mode and stayed there for more than 4 days with 0% progress. we have two pools and the issue is with pool B with 29TB of data storage. Replaced disk is from Pool B. I observed that this pool went to fault status and the disk drives are in degraded state. All the virtuals disk went offline and my critical applications are not running because of this. 

THe error message says the virtual pool is offline and the all the data is lost.  I did not perform controller restart as it was recommended to take backup before performing this task. 

My questions are:

1.  There are no global spares defined in our storage. So, once the failed disk replaced with new one, do i need to configure the new disk as a spare to start Reconstruction?

2. As per the message, the data is lost. so is there any chance that we can retrieve the data/ take the data backup with the existing condition? if so how can i do it?

3. Since the data is already lost, (as per the error description) shall i perform a controller reboot? individually one after the other.

Request you to kidly look it this as a priority as there are some critical application running in this.

Appreciate your quick response and let me know ifyou need any additional data.Error MessagesPool Fault MessageDisk Degraded Message

[ArunKKR]

Hi,

Once a failed disk is replaced, if dynamic sparing option is enabled it will automatically join disk group and reconstruction will start.

If dynamic sparing option is disabled we will need to configure new disk as global spare for it to join the disk group.

In the current state there is no option to take data back up.

If the disk groups in Pool B are not reporting offline or quarantine state you could try performing a graceful power cycle after shutting down storage controllers through SMU/CLI.

Majority of the Pool offline cases if it doesn't get resolved after power cycle, would require detailed logs review and MSA engineering team engagement by logging an HPE support case.

I do understand that MSA 2040 is an end of support life product.

However, you could still try logging an HPE support case to check one time support options.

[MP_KV]

Dear Arun,

Thank you for your reply.

As mentioned in my post virtual pool B is in offline status. We attempted to restart controllers. We see option to restart management and storage. Could you please help me in choosing the right option to restart? If the pool status is offline can i attempt to restart?

[ArunKKR]

Hi,

Please try a graceful power cycle if the disk groups are not reporting quarantine or offline status.

Shut down both the storage controllers through SMU

Allow 5-7 minutes for the process to get completed.

Remove power to controller enclosure and attached disk enclosures if any.

Provide power to attached disk enclosures

Allow 3-5 minutes for disks to initialize

Provide power to controller enclosure 1 and check status.

If issue persists, please log an HPE support case for detailed investigation.

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you are satisfied with the answers then kindly click the "Accept As Solution" button for the most helpful response so that it is beneficial to all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".