topic Re: MSA2324i vDisk Rebuild Problem in MSA Storage

MSA2324i vDisk Rebuild Problem

InTech2000 — Thu, 27 Feb 2020 10:59:35 GMT

Hi,

We have an issue with our MSA2324iand need your help.

Initial Problem

It looks like we had a couple of disk that had SMART errors, and this made it go into a degraded state.
The disks were marked as LEFTOVR
I cleared the metadata on the disks and assigned them as dedicated spares.
The vdisk started rebuilding - Great.

The rebuild finished however and now the vdisk was showing "Fault" with a "red cross" and the message "The virtual disk is offline. Disks may be down or missing. Data may be lost."

I tried clearing the metadata again, and added them as spares and disks are showing as :

2 x AVAIL

2 x VDISKSP

The vDisk is a RAID 10 with 10 Disks allocated.

I have all green ticks showing on all the disks in the enclosure - but the rebuild will not start.

I tried reallocatting again, but still it didnt start.

What I am doing wrong? How do I set off the rebuild process manually?

Many Thanks.

Re: MSA2324i vDisk Rebuild Problem

SUBHAJIT KHANBARMAN_1 — Thu, 27 Feb 2020 12:24:18 GMT

You already done the damage and vdisk recovery not possible it seems.

You shouldn't clear metadata in multiple drives when you have QTOF or Offline Vdisks. You should always take help from storage experts before you try anything.

It's possible that in RAID10 vdisk two drives part of same sub-vdisk you have cleared metadata so it lost that mirror set and that's why data recovery not possible.

You can still log a case with HPE support and try your luck.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************

Re: MSA2324i vDisk Rebuild Problem

InTech2000 — Thu, 27 Feb 2020 13:22:48 GMT

Hi,

Thanks for replying.

If I add more disks in - would it start rebuilding then do you think?

Also , how do I bring the volume back online manually?

Thanks..

Re: MSA2324i vDisk Rebuild Problem

SUBHAJIT KHANBARMAN_1 — Thu, 27 Feb 2020 13:23:57 GMT

No it will not help.

Try to understand the logic, every drive have one section called metadata which contains vdisk configuration details. If you clear that metadata then that drive lost all details so it will not participate for RAID recovery.

For example, suppose there is one vdisk created with 5 drives in RAID 5 this means 4 data and 1 parity

In this condition if 1 data drive failed then you can get back data because other 3 data drives and 1 parity drive will rebuild failed drive data. But in case back to back two drive failed then we can't recover back data. That's called RAID failure.

In your case also same happened. Only difference is your vdisk was in RAID10 where sub-vdisk RAID set got failed.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

************************************************************************

Re: MSA2324i vDisk Rebuild Problem

InTech2000 — Thu, 27 Feb 2020 13:55:43 GMT

Hi,

The drive was in Leftover state. The only way to rebuild was to clear the metadata. This did work and it started rebuilding.

After the rebuild, it showed the vdisk as offline.

The config at the moment shows :

Raid 1-1 - Disks 1,2

Raid 1-2 - Disks 1,2

Raid 1-3 - Disks 1,2

Raid 1-4 - Disks 1,2

Spare - 1,2,3

There are no errros on the disks - so I am not sure why its not rebuilding as there are spares available..

Is there a manual way to start the rebuild?

Re: MSA2324i vDisk Rebuild Problem

SUBHAJIT KHANBARMAN_1 — Thu, 27 Feb 2020 14:23:24 GMT

Thanks for sharing the details but you still not able to catch the RAID logic

Your vdisk created with 10 drives and in RAID10 which means you have 5 RAID1 sub-vdisk which are in RAID0 to build the complete RAID10 vdisk.

You see you have 4 sub-vdisk available out of 5 sub-vdisk. Your 5th sub-vdisk is completely missing because those two drives lost their metadata which means one RAID1 sub-vdisk is missing.

That's why complete vdisk shows as offline.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

************************************************************************

Re: MSA2324i vDisk Rebuild Problem

InTech2000 — Thu, 27 Feb 2020 14:55:05 GMT

Hi,

Apologies I gave you the wrong information.

The original config had 8 disks and had :

Raid 1-1 - Disks 1,2

Raid 1-2 - Disks 1,2

Raid 1-3 - Disks 1,2

Raid 1-4 - Disks 1,2

---------------

After the failure its now showing :

9 Disks (not sure why!)

Raid 1-1 - Disks 1,2

Raid 1-2 - Disks 1,2

Raid 1-3 - Disks 1,2

Raid 1-4 - Disks 1

Spare - Disk 1

I was hoping it would rebuild from the dedicated spare. There are no quarantined disks.

Is there manual way to do this?

Re: MSA2324i vDisk Rebuild Problem

SUBHAJIT KHANBARMAN_1 — Thu, 27 Feb 2020 15:07:11 GMT

Can you please share the below output,

show vdisks <Vdisk_name>

Where Vdisk_name will be the vdisk which you are trying to recover

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

************************************************************************

Re: MSA2324i vDisk Rebuild Problem

InTech2000 — Thu, 27 Feb 2020 15:14:50 GMT

#show vdisks vDisk1_R10
Name Size Free Own Pref RAID Disks Spr Chk Status Jobs
Serial Number
---------------------------------------------------------------------------
vDisk1_R10 2398.0GB 0B A A RAID10 8 1 64k OFFL
00c0ff1162050000c9371b4e00000000
---------------------------------------------------------------------------

Re: MSA2324i vDisk Rebuild Problem

SUBHAJIT KHANBARMAN_1 — Thu, 27 Feb 2020 15:33:19 GMT

I would suggest to get a call log with HPE support because without checking MSA log it will be difficult to work on this further.

As vdisk in offline state so we may have to excute below command but that's not safe to run without checking logs otherwise you will face data loss,

trust enable

trust vdisk vDisk1_R10

Until vdisk back to degraded state Vdisk rebuilt wouldn't start.. Any vdisk in QTOF or Offline state wouldn't start rebuild eventhough you have spare configured.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*********************************************************************

Re: MSA2324i vDisk Rebuild Problem

InTech2000 — Thu, 27 Feb 2020 15:40:28 GMT

Thanks for you assistance.

I think I have at least 3 faulty disks as it shows the same error with different serial numbers :

There is a problem with a FRU. (FRU type: disk, enclosure: 1, device ID: 3, vendor: HP , product ID: EG0600FBVFP , SN: KWxxxxx, version: HPDC, related event serial number: A66399, related event code: 55)

This error gets generated when the MSA does the disk scrub i think. However it still shows a green tick on the disk in the enclousure view.

I have ordered 5 brand new disks to be added tomorrow to the enclosure.

Would it be best to add the new disks, configure as dedicated spares, then use the trust command do you think?

Can I send the logs to you in private message to have a look?

Thanks

Re: MSA2324i vDisk Rebuild Problem

SUBHAJIT KHANBARMAN_1 — Fri, 28 Feb 2020 01:14:05 GMT

You need to clearly tell me below details,

1> How many total drives present in your system initially before you tried metadata clear?

2> How many total drives configured as spare before you tried metadata clear?

3> Ofcourse multiple drives having issue otherwise Vdisk can't go into offline state

4> Location of the drive which you have cleared metadata?

5> Do you have sub-vdisk details means which drive paired with which drive to make one RAID set, like that I need details of the 4 RAID set

You can send me the logs in private as well. I can give it a try.

If you have valid support with HPE then it's always good to log a support case officially.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

**************************************************************************

Re: MSA2324i vDisk Rebuild Problem

InTech2000 — Fri, 28 Feb 2020 12:26:35 GMT

Hi,

Answers below :

1) 15 drives present in enclosure

2) Zero drives Configured as spare

3) .

4) Slots 4, 7, 9 and 13

Disk1-1 Raid1
Disk1-2 Raid1
Disk1-3 Raid2
Disk1-5 Raid2
Disk1-8 Raid3
Disk1-10 Raid3
Disk1-11 Raid4
Disk1-13 Spare

I dont know how to attach logs, I cant see any upload button in the DM.

Re: MSA2324i vDisk Rebuild Problem

SUBHAJIT KHANBARMAN_1 — Fri, 28 Feb 2020 12:45:58 GMT

So I understand RAID sets as below right now,

RAID set 1-> 1,2

RAID set 2-> 3,5

RAID set 3-> 8,10

RAID set 4 -> 11, ?

So you say 4, 7 and 9 were not part of any of the above RAID sets, only 13 was part of them. This can't be possible because drives can only go to LEFTOVER state when they were part of some vdisk and for some reason they marked as LEFTOVER by controllers.

Do you remember how was the RAID sets before drives went to LEFTOVER state?

4 drives metadata clear always risky for data recovery.

You can be extremely lucky if none of 2 drives out of 4,7,9 and 13 part of same RAID set. Then we have good chance to recover this vdisk.

Without logs it's difficult but if you can tell me condition of RAID sets when everything was fine, then which drive join which RAID set after you cleared metadata that may help us. For example, lets say RAID 2 set earlier drive 3 and 4 was in pair but later drive at location went to LEFTOVER state. Then drive at location 5 you have configured as spare and it joins drive 3 as partner. Sameway other sub-vdisks also.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

**************************************************************************

Re: MSA2324i vDisk Rebuild Problem

InTech2000 — Fri, 28 Feb 2020 13:16:52 GMT

Hi,

THanks, for taking time to look at this - really appreciate it.

Luckily I took a screen shot when everything was ok as follows :

vDisk1

RAID set 1-> 1,2

RAID set 2-> 3,5

RAID set 3-> 8,10

RAID set 4 -> 11, 7

vDisk2

RAID set 1-> 6,14

RAID set 2-> 12,15

Disk 13 - must have been a global spare at that particular time.

Not sure about Disk 4,9 though.

Sorry I should have mentioned I have a 2nd vDisk Volume.

Does that help?

Re: MSA2324i vDisk Rebuild Problem

SUBHAJIT KHANBARMAN_1 — Fri, 28 Feb 2020 13:15:58 GMT

So it's possible that 4, 9 and 13 went to LEFTOVER state long back.

Recently 7 went to LEFTOVER state

Going with the fact and information that you have provided you can try below and 13 should participate in rebuilding the vdisk1

trust enable

trust vdisk vDisk1_R10

show vdisks vDisk1_R10

Note: You can still log a HPE Support call and take official help before you try the above.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

**************************************************************************

Re: MSA2324i vDisk Rebuild Problem

InTech2000 — Fri, 28 Feb 2020 13:33:50 GMT

Hi,

Unfortunately we only have hardware support only this MSA so I am unable to log a call with HP.

I am waiting for the 5 brand new disks to be installed first, before I use the trust command.

I think it is best to make disk 13 back to "avail" and configure the new disks as dedicated spares, so the volume would rebuild?

What do you think?

Re: MSA2324i vDisk Rebuild Problem

SUBHAJIT KHANBARMAN_1 — Fri, 28 Feb 2020 15:32:04 GMT

I don't know if drive 13 having medium errors or any kind of hardware errors or not. If so then you should wait for new drives. Otherwise you can use any of 4,7,9 and 13 for this Vdisk1 data recovery if none of them having any hardware errors.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************

Re: MSA2324i vDisk Rebuild Problem

InTech2000 — Fri, 28 Feb 2020 16:41:18 GMT

Thanks,

Will let you know how it goes.

Re: MSA2324i vDisk Rebuild Problem

InTech2000 — Sun, 01 Mar 2020 19:14:00 GMT

We added the 5 disks on friday bays 16-20, and we had some issues with the controllers going offline.

Manage to restart the controller, and configure the 4 out of the 5 as dedicated spares.

We did the "trust" command and the volume came back online and it did a reconstruct momentarily. and it went offline again.. It now showed RAID set 4 -> 16 only - previously is was disk 11.

We did the trust" command and the volume came back online and it did a media scrub.

It all shows online now and configured as follows :

vDisk1

RAID set 1-> 1,2

RAID set 2-> 3,5

RAID set 3-> 8,10

RAID set 4 -> 11, 16

vDisk2

RAID set 1-> 6,14

RAID set 2-> 12,15

According to the previous logs, they have been issues wth Disk 8 and 11, but at moment, its showing all green, ticks vdisk is online and health status "ok".

Whats the best next steps, I am worried disk 8 and 11 may fail together? Is they any commands I can run to do a check on the disks?

Also, is it better to configure as dedicated or dynamic spare?

Thanks.