Background parity initialization

[Salemmahara]

Hello eveybody

It's my first time with Raid 50 

I've created a Raid 50 using 10 X 300G SAS HDD on DL 380 G10 with p408 controller. I found there is a message which says: 

Background parity initialization is in progress : 10% and ... . 

- Can I turn server off during this procedure and start it in another time? Or it should be complete before eveyting?

- Is it a ONE TIME process or it will repeat with any drive replacement ?

- If the answer of second question is YES, does it take a long while or cause any performance issue? I read that DELL servers don't show us new drive space before background parity initialization done . What about HPE servers? 

I thought a parity initialization with new arraived disks without any data will be complete in a few seconds! How long does it take ?

[IvBuh]

Hello Salemmahara,

Here is what I can tell you:

1.Can I turn server off during this procedure and start it in another time? Or it should be complete before eveyting?

         If the server is powered off during the process it will pause and continue when the server is back on.

2. Is it a ONE TIME process or it will repeat with any drive replacement ?

        When a drive is replaced (or even plugged out for some reason) a rebuild proccess will be initated and the parity will be calculated/initializaed once again. In short, yes, it will go thorugh the process with every drive replacement, however it might not be as lengty as the first time.

3.If the answer of second question is YES, does it take a long while or cause any performance issue? I read that DELL servers don't show us new drive space before background parity initialization done . What about HPE servers?

       With a RAID 50 of this size the process can take a long time (days, even weeks) as by default the process is set to priority "Delay" which means that the process will run only when the controller is inactive for more than 15 seconds and pause when any activity is detected.
During the innitial parity initialization process the RAID is protected from single drive failure so I would advise you to let the process finish before activly using the server/controller.
To speed up the process you can try and leave the controller innactive until the process is finished as well as set the "Surface Scan Analysis Priority" to high temporarily, until the process is complete, then switch it back to medium where it should have innitially been.
The process does affect performance somewhat and I would advise that you wait it out before using the controller/server heavily.

So to recap:
1. If you turn the server off the process is paused.
2.The process is initated with every drive replacement as it rebuilds.
3.If the controller is used while the process is running it can take a long time. The process does somewhat affect performance. You should let the process finish before healivy using the controller as one one drive failure is allowed. To speed up the process you can leave the controller inactive and set the "Surface Scan Analysis Priority" to high until the process is finished and set it back to the default value after (medium).

I hope this answers all of your question and you find it helpful.

Best regards,
Ivan

[BOYAN BIANDOV]

Hi @IvBuh ,

Thank you for the reply and for answering right on-point!

That was a very clear answer on an issue that I feel there's so much confusion about out there. If I may follow up with a question in a context of a less sophisticated smart array configuration - good old RAID5. (4 disks, 3 usable total capacity).

Once a drive is replaced it takes an hour or so to rebuild the array. Then HPE indicates that one is "out of the woods" so to speak by the red alerts clearing automatically. However after the array rebuild is complete the next step in the process is background parity initialization, which as you explained can take quite a long time to finish.

My question is exactly during Background parity initialization - is RAID5 protected from a single drive failure or is it not while Background parity initialization is running?

Thank you

~B

[IvBuh]

Hi Boyan,

I am happy my answer was helpful

As for the new questions, the answer is yes and no.

• Yes, the drive is safe as the main data is written to the new drive and the drive is now in use.
• No, because any new data that has been written on the drive will be lost if it fails, but that is only the small amount of extra data on the array that has been added post readding the new drive to the RAID and is currently being calculated (indicated by the background parity initialization). This process runs frequently on the controller as new data is written to the array as it need to calculate new parity data for the new "real" data that's being added. The process runs in the background as not to hinder the controller performance especially during intensive read/write cycles. 

So in term the new drive is safe as the old data is backed up, however any new data (small amounts) is in danger while the new parity is being innitilized, however this goes for any new data and not only in case a drive is replaced.

Let me know if you have any further questions.

Best Regards,

Ivan

[WafflesMcDuff]

I have a related question:
I setup a new RAID in the last few days on my P410i on my ML350 Gen6. The RAID was built and in OK status.
Background parity initialization was 65% done or so. Then we had a major lightning storm that was so intense that the surge got through my surge protected UPS and cooked the motherboards. The drives seem to be in tact.
I’ve got an ML350 Gen8 on the way. Can I expect that as long as I insert the drives in the right order, as long as the drives are in damaged, that my RAID and data should still be there, even though it’s a new raid controller in a new server?

[TurskRO]

---

@WafflesMcDuff wrote:
I have a related question:
I setup a new RAID in the last few days on my P410i on my ML350 Gen6. The RAID was built and in OK status.
Background parity initialization was 65% done or so. Then we had a major lightning storm that was so intense that the surge got through my surge protected UPS and cooked the motherboards. The drives seem to be in tact.
I’ve got an ML350 Gen8 on the way. Can I expect that as long as I insert the drives in the right order, as long as the drives are in damaged, that my RAID and data should still be there, even though it’s a new raid controller in a new server?

---

So this should have worked fine for you. Can you update the thread with the results?

[TurskRO]

I have a followup question.

I have a RAID 5 array with 2 logical drive on it. (DL380 G8)

Here are the specifications:

8 1TB SSD hard drives

7 used for array and 1 hot spare

The SSD drives all need a firmware upgrade. They have been "failing" at about 300 days of in service. I have replaced one failed drive with another with the old firmware then discovered that the failed drive was OK and upgraded it's firmware. I have since replaced another drive with the used upgraded firmware drive and it seems to be running fine.

I want to upgrade the firmware on the rest of the drives by pulling a drive, upgrading the firmware, and replacing it back in the RAID.

My question is this - of course I'm going to wait for the rebuid to happen, but do I need to wait for the background parity to finish before replaceing the drive back in the array?

After putting it back in, I would then wait for the rebuild and also wait for the parity before removing another.

Thoughts?

[pchops]

---

First things first -- make sure you have a good backup. Also, I'm assuming that we are talking about a RAID5 array behind a SmartArray controller, and not software RAID at the OS level.

I want to upgrade the firmware on the rest of the drives by pulling a drive, upgrading the firmware, and replacing it back in the RAID.

If these are HPE drives, you shouldn't need to physically remove them, or "break" the array, to do a firmware upgrade (although I would recommend doing the firmware upgrade offline). If for some reason you must remove them to do a firmware upgrade, I would shutdown the system first, and then update one drive at a time while the system is off (or if you can update multiple drives at a time, be sure to label them so that they go back into their original drive bay slots). Using either of these methods will avoid having to do the rebuild on the array. Also, each time you pull a drive on a healthy RAID5 array that is currently in operation (that is, writes are being done), you run the risk of a second drive failing, which will render the array inoperative (this can also occur after you have replaced the drive, but before the array has completed its recovery).

My question is this - of course I'm going to wait for the rebuid to happen, but do I need to wait for the background parity to finish before replaceing the drive back in the array?

After putting it back in, I would then wait for the rebuild and also wait for the parity before removing another.

You will need to wait for the logical drive to both complete the recovery (rebuild) and for the parity initialization status to complete before physically removing another drive. Also, if performing the physical drive operation with the array online, I would "offline" the drive to be removed before actually physically yanking it out of its drive bay (you can use the SSA GUI or CLI to offline a physical drive).

One other thing to consider if you need to perform these operations online -- you said one of the drives is designated as a hot spare. The spares policy you have set, will determine its behaviour when you remove a drive -- see this post for the details:

https://community.hpe.com/t5/proliant-servers-ml-dl-sl/hpe-ssa-user-guide-spare-management-mode-clarification/td-p/6955840

To avoid a lot of extra recovery/parity initialization operations, you may want to temporarily remove the hot spare from the array until you've completed all of the online firmware updates.

---

[TurskRO]

Just an update.... I did end up moving all the VMs to another VM host and taking the server down to do the firmware upgrades. To answer a couple of questions, yes they were HP SSDs but not enterprise SSDs designed for a server. This was an expirimental server setup with non-server SSDs. The VMs that are on this server are re-creatable and house no data that can't be recreated. (print server, MFA server, remote workstations, etc.)

The firmware upgrade was performed very simular to your advice, shut down server, pulled one drive at a time to do the firmware upgrade, then replaced.

System came back up fine and I "replaced" the failed drive after the firmware was upgraded. Server has rebuillt and pairaty is done. Everything seems fine. I will be putting the VMs back today.

BTW: I hate that the smart drive trays do not show drive activity with non server SSD drives. All I get are the green lights and yellow if a drive fails. The circle of lights showing activity do not work. Thanks HP.

I'll let you know in a while if I have any drives fail again.