MSA 2050 vDrain takes months!

[Paul41]

Hi Guys,

I have a weird issue on one MSA. Any vDrain takes months(!). It goes to 99% normally within a few hours (depending on the size of course and then it stops for weeks and months!).

So currently I've started removal of 2x 1.9TB SSD, it took few hours to go to 99%, but it stuck there. 

VDRAIN started on 28th of JUL(!) today is 2nd of OCT(!). MSA is on the latest firmware.

 You'd think this is a case for support and why didn't I contact them? Well, I did! But they're saying that that's NORMAL. Is it really normal??? I never had any issues like that on any other 2050 msa with same drives.

Here is the reply from support, basically they're suggesting to delete whole pool to speed it up:

Kindly the check the below link shared by our Level 2 engineer:

It usually takes time and it depends upon the amount of data we have.

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00018177en_us

RESOLUTION 

For a faster alternative to the VDRAIN utility, the current pool data should be carefully backed up and verified as valid. Delete the pool using the following CLI command:

CLI# delete pools A|B

Once the old pool has been deleted, re-create the new pools with the disk group layout and capacity required. Data can then be restored to the new disk groups immediately, and accessed.

[SUBHAJIT KHANBARMAN_1]

@Paul41 

As per my understnding any Virtual Disk Group (VDG) information stored in drive metadata. So when we delete any VDG then that particular VDG information supposed to get removed from underlying drives metadata who are all holding that information.

In your case VDG removal process not completing and that may be due to data structure issue for which HPE support may have suggested to take backup and delete the Pool followed by re-create of VDGs. When you delete the pool then all drives part of that Pool will become new. Then you can re-create VDGs and your issue should get fixed.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

**************************************************************************

[Paul41]

Thank you for your reply,

Unfortunately, it happed 3 times already. With 3 different VDGs. It also finished but finished after months of vDrain, which points more towards firmware bug than anything else. 

I must point out that 3rd time I've updated MSA firmware while it was stuck on 99% (for few months) and after the firmware update, it miraculously completed. However, I'm not sure if FW really fixed it.

I'll keep this post updated it issue repeats with the latest firmware.

[SUBHAJIT KHANBARMAN_1]

@Paul41 

My suggestion would be as below in this situation whenever you face VDRAIN stuck issue,

1. Take the backup of the entire data if possible or at least the back up of all DGs in Pool which has the issue .
2. We need to have a down time planned. Stop all IO access to hosts.
3. Disable the scrub on the Diskgroup that has the issue .
Ex : # end scrub disk-group <DGname>
4. Shut down the MSA gracefully .
5. Leave for 5 minutes, and then turn on the MSA gracefully .

You can check with HPE support and they can check with Engineering team to work on this further to understand it in better way and find the fix.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*********************************************************************

[SUBHAJIT KHANBARMAN_1]

@Paul41 

You can also check this Customer Advisory as alternative,

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00018177en_us

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*********************************************************************

[Paul41]

Thanks, this is actually one of the few things we tried before contacting support. We also tried to shutdown and remove controllers for some period of time. When it boots up, vdrain starts from scratch and then sits again on the same percentage.

[Paul41]

Well, deleting a pool is more of a workaround, it may work if you have a small pool, but if it is 50-100TB, it is clearly not a solution I'm afraid...