How many disks can fail in 3PAR 7000 series?

[sc0tt]

I need help with understanding how many disk failures 3par can survive before losing data. I get that a raidlet (raid set of chunklets) has the same protection as a traditional raid set in that you can lose one chunklet in a r5 set and you can lose two chunklets in a r6 set. That's straightforward.

What I don't get is that there is another layer of striping in the logical drives across the raidlets. This is also where the "rows" come into play.

So, while a RAID5 raidlet might lose data if it loses two chunklets, do you also lose data if there is RAID50 striping across all the raidlets in the Logical Drive? I'm thinking the RAID50 striping at the LD level over its raidlets provides another set of protection, no?

Thanks,

P.S. This thread has been moved from Disk-Based Systems > Disk Array > to Storage Area Networks (SAN) (Enterprise). -HP Forum Moderator

[Dennis Handly]

>you can lose one chunklet in a r5 set and you can lose two chunklets in a r6 set.

That's right.  Except your question above about how many disks can fail is really how many chunklets can fail.

>do you also lose data if there is RAID50 striping across all the raidlets in the Logical Drive?

Yes.  Chunklets are for performance not availability.

>I'm thinking the RAID50 striping at the LD level over its raidlets provides another set of protection, no?

No, that just scrambles where the data is.  The number of copies of your data only depends on the raid level.

Have you looked at the Concepts manual?

[sc0tt]

Hi, thank you for you reply.

"No, that just scrambles where the data is. The number of copies of your data only depends on the raid level."

RAID50 is a raid level. That's what's confusing to me. Calling it RAID50 (Logical Drive RAID50) implies
protection; why do LDs have RAID10, 50 and 60? It also doesn't help that HP uses RAID10/1, 50/6 and 60/6 interchangeably in documentation.

Here's an example of what's contributing to my confusion, Page 35 of Concepts Guide:

RAID 5 and 50
"On a RAID 50 logical disk, data is striped across rows of RAID 5 sets."  <<< If it's not RAID, then why is it referred to as RAID?  Further, if there's no protection and only striping then why not call it RAID0?  

"Have you looked at the Concepts manual?"
Yes, and the Architecture, class manuals, etc... I guess I'm making it more difficult than it needs to be. We
have a EVA customer asking how many drives he can lose in a 3par system (he's familiar with RSS). I was wondering if 3PAR had something similiar and if the RAID50 at LD level was its approach to another protection against failure. I did read "An Introduction to HP 3PAR StoreServ for the EVA Administrator" and it does state that "Drive Magazine level
availability means that a logical disk can tolerate a drive magazine failure..."

Would this be an accurate description of 3PAR protection as it pertains to raid/raidlets/chunklets/drive,etc?

With regard to drive failures and RAID. It’s the same, for example, in that a RAID6 set of 10+2 will utilize 12 drives for that raidlet…each chunklet on its own drive. In this case, this particular raidlet can survive 2 drive failures. The same you'd expect from RAID protection at the drive level.  However, due to the use of chunklets, and the global, distributed nature of spare chunklets, the rebuilds occur much faster b/c you’re relocating the lost data across all the spare chunklets across all available drives – you’re not rebuilding a drive, you’re rebuilding chunklets.

[Dennis Handly]

>It also doesn't help that HP uses RAID10/1, 50/6 and 60/6 interchangeably in documentation.

That's because everything is striped at chunklet layer.

>"On a RAID 50 logical disk, data is striped across rows of RAID 5 sets."  <<< If it's not RAID, then why is it referred to as RAID?

These rows of RAID 5 sets are on different disks.  That's where you get the protection.

>We have a EVA customer asking how many drives he can lose in a 3PAR system

Again, the simple answer is 1 for RAID 1&5 and 2 for RAID6.

But since the chunklets are scattered all over, you could be lucky and lose more disks.  But there would be some VVs where you aren't lucky.  :-(

>if the RAID50 at LD level was its approach to another protection against failure.

No extra protection, just performance.

> it does state that "Drive Magazine level availability means that a logical disk can tolerate a drive magazine failure..."

And there is also Cage and Port availability.

And for a 7000, there is only one disk per magazine, not much help.

>Would this be an accurate description of 3PAR protection as it pertains to raid/raidlets/chunklets/drive,etc?

Yes but it is at a higher level than disk and your initial question.

>in that a RAID6 set of 10+2 will utilize 12 drives for that raidlet…each chunklet on its own drive. In this case, this particular raidlet can survive 2 drive failures. The same you'd expect from RAID protection at the drive level.

Yes.

>the rebuilds occur much faster b/c you’re relocating the lost data across all the spare chunklets across all available drives – you’re not rebuilding a drive, you’re rebuilding chunklets.

Yes, there is that.

[3Par_Eng]

The R10/R50/R60 comes from the fact that we do wide striping.  The striping is done by joining multiple RAID sets in a row to form a Logical Disk (LD). And chunklets are selected such that the same disk is not repeated in the row again.

Another property of an LD is that it will always use disks that are owned by a single node. So in case of a 4 node system it will use chunklets from only 1/4th drives. But the volume would be created across multiple LDs each created using drives from different nodes.

When 1 disk fails RAIDs sets that are using chunklets from this disk will have reduced redundancy, but we will continue to serve data. In the background we would start reconstructing the data from the failed disk onto spare chunklets.

You also need to take into consideration the "HA" (high availability) level of your CPG.  HP recommends to use HA cage for all CPGs.  With HA cage you can lose access to 1 entire cage and still all your data will be accessible.

[sc0tt]

Excellent.  This information has now been processed.  I finally get it.  :) 

Thanks!

[huymeng]

Hello 3Par_Eng

Kindly clarify if I have 16 disks in the same drive enclosure configure raid 5 (3+1)

how many disk can fail at the same time ?

appreciated your answer

[3Par_Eng]

If you want to know the maxmum disks that can fail at random, then the answer is still 1, as you are using R5 CPG.

However if you can pick and choose which disks can fail, then any 2 disks placed next to each other in the enclosure can fail  and you can still have all your data.

[huymeng]

Hi 3Par_Eng

 

Thank for reply

that mean raid 5 CPG with 16 disks still can fail 1 disk

but if I use raid 6 (6+2) with 16 disks it can fail 2 disks at the same time

raid 5 (3+1) and raid 6 (6+2) useable space the same but what's difference is raid 6 slower than raid 5 right ?

[Dennis Handly]

>raid 5 with 16 disks still can fail 1 disk but if I use raid 6 (6+2) with 16 disks it can fail 2 disks at the same time

Yes.

>raid 5 (3+1) and raid 6 (6+2) useable space the same but what's different is raid 6 slower than raid 5 right?

It sure seems like it.

>if you can pick and choose which disks can fail, then any 2 disks placed next to each other in the enclosure

If you have HA cage, the whole cage can fail.  :-)

But that's to protect against links and power failures, not that many disk failures.

[Sheldon Smith]

When writing, RAID-6 has a higher backend IO requirement (6 2/3 IOs) than RAID-5 (4 IOs).

RAID-5 has a 4 IO back end penalty.  As a write comes in the old parity and old data must be read (two read IOs) then parity calculated and new parity and new data written out (two write IOs).  So each random write to R5 has 4 back end IOs (2 read + 2 write).

RAID-6 calculation uses the XOR engine in the ASIC but must calculate two distinct parities (R5 only needs one parity) and need to compute parities over more data since RAID sets are larger.
Most writes require only two parities.  However, a fraction (1/3 for step size 8), of the data blocks are used to compute 3 parities, so updating those blocks requires reading/updating 3 parities, hence the odd number 6 2/3 back end IOs for RAID-6.

Nevertheless, when using the large nearline (NL) drives, HP strongly recommends using RAID-6 and avoid RAID-5 due to the long rebuild time when and if a NL drive fails.

Note: While I work for HP, all of my comments (whether noted or not), are my own and are not any official representation of the company.
----------
If my post was useful, click on my KUDOS! "White Star" !

[Prakash Singh_1]

Hi,

For HP 3PAR storage system querries you can also visit the HP Guided troubleshooting tree.

Below is the link for 3PAR HPGT:

http://h20180.www2.hp.com/apps/Nav?h_pagetype=s-905&h_lang=en&h_cc=us&h_product=304617&h_audience=smb&h_client=s-h-e010-1&h_audiencerestrict=true&lang=en&cc=us

HP Guided Troubleshooting Solving issues

[huymeng]

Hi HP Pro

if raid 5 (3+1) CPG with 16 disks in same drive enclosure still can fail 1 disk

and if I choose raid 5 (7+1) CPG with 16 disks in same enclosure still can fail 1 disk

my better choice is raid 5 (7+1) because it can be usable capacity more than raid 5 (3+1)

I'm still not fully understand about How many disks can fail in 3PAR 7000 series?

appreciated for clarify

[Dennis Handly]

>If raid 5 (3+1) CPG can fail 1 disk

>if raid 5 (7+1) CPG can fail 1 disk

Yes.

>my better choice is raid 5 (7+1) because it can be usable capacity more than raid 5 (3+1)

Yes.  But if one disk fails, you now have to read 7 disks to recompute the data.

>I'm still not fully understand about How many disks can fail in 3PAR 7000 series?

Nothing has changed.  For RAID 1 and 5, just one disk, at the same time.  For RAID 6, just two, at the same time.

If there is a delay, the data can be copied to spare chunklets.

What are you trying to do/learn with this info?