topic Re: EVA Releveling in HPE EVA Storage

EVA Releveling

Bill Mace — Wed, 24 Aug 2005 20:07:09 GMT

When a disk fails, the EVA relevels and distributes the data to the remaining disk. In a frame with 168 disks in one group, configured for maximun performance and 4 disks for double sparing (protection level) the rebuild time is taking multiple days. The EVA's are running at 95% + occupancy. The latest Best Practise document states that only 5GB of free space are required for maintenance. Free space has nothing to do with the rebuild process other that having less to rebuild. Host i/o's accessing the EVA during rebuilding of the array imppacts the process. Is there anything other than limiting host i/o's that will speed up this process. I can not find any documentation on how the rebuild process works. I am concerned that when an EVA is at 98% and is bebuilding that another failure will cause it to corupt data.

Re: EVA Releveling

Uwe Zessin — Thu, 25 Aug 2005 01:44:52 GMT

A disk failure is followed by two tasks:

- the recovery
it does the minimum work to restore redundancy

- the leveling
it makes sure that user data is equally distributed over the disks

If another disk fails during recovery, it depends on the 'location' of the disk if data loss is the result or not (same as a traditional RAID system). The EVA divides its disk groups into different failure domains called RSS (Redundant Storage Set). It can tolerate the loss of multiple disks as long as each disk belonged to a different RSS.

Re: EVA Releveling

Bill Mace — Thu, 25 Aug 2005 03:44:59 GMT

Uwe,
I totally agree, but the EVA is at risk during the reconstruct of the RSS's. If the RSS reconstruct is coompllete then the EVA can handle another failure and resart a new releveling process. I understand that it reconstructs RSS, thern relevels Vraid5, then relevels VRaid1. The only time that it is critical is during the reconstruct of the RSS's. Outside of suspending I/O's (not possible in production), is there any other way to shorten the releveing time, still using 168 disk group, The EVA's are all running with 98%+ occupancy.
Thanks Bill

Re: EVA Releveling

Uwe Zessin — Thu, 25 Aug 2005 04:38:48 GMT

Bill,

this is nothing different than any other storage array that is not running some kind of ADG/RAID-6/RAID-5DP or whatever the vendor calls his implementation of 'double protection'. If too many disks fail, the data is gone :-(

I am not aware that there is any way to 'tune' the leveling process, make it faster, slower, suspend it...

Re: EVA Releveling

Mario_66 — Thu, 25 Aug 2005 04:59:51 GMT

Hi,

can you clarify your first statement: Rebuild takes multiple days?

What do you mean by that? Reconstruction or migration with or without leveling?

Can you point me to the best practise document that you mentioned here?

If I understand you correctly then the most problematical situation could be (Vraid0 is problematical by design ;o):

Vraid5, one failed disk. During reconstruction phase another disk in the same RSS has failed. But it does not neccessary mean that you lose your data. It depends about failure type.

I would say that it is not EVA specific issue. It is RAID5 limitation by design if we are talking about RAID5 by definition.

BTW, I would not say that more free space will not speed up rebuilding process.

Regards,
M.

Re: EVA Releveling

David Ell — Thu, 25 Aug 2005 05:17:56 GMT

I am having the same problem... The question here is not redundancy but one of the time it takes to relevel an entire disk group.
I have pinged HP on this and was told there is no way to "prioritize" the releveling process. What makes it worse is the more host IO the longer leveling takes....

Re: EVA Releveling

Mario_66 — Thu, 25 Aug 2005 06:33:09 GMT

Hi,

that's true, but what is a problem if the leveling is running longer then expected?

I am not aware of any impact on data availability or data integrity. Leveling is optimization process and the most of the storages have some kind of it.

Maybe I missed something?

Regards,
M.

Re: EVA Releveling

David Ell — Thu, 25 Aug 2005 07:05:29 GMT

It the the management overhead... Best practices states that you should wait for leveling to complete before replacing a drive. In addition, if you are experiencing drive failures often, you would never "catch up"

Re: EVA Releveling

Mario_66 — Thu, 25 Aug 2005 07:21:31 GMT

Hi,

"The Best Practise" says that you should wait for the reconstruction to complete, not leveling.

http://h200005.www2.hp.com/bc/docs/support/SupportManual/lpg29448/lpg29448.pdf, page 8

Regards,
Mario.

Re: EVA Releveling

David Ell — Thu, 25 Aug 2005 07:23:59 GMT

Our Gold TAM recommends waiting for leveling to complete. This is interesting though

Re: EVA Releveling

Ivan Ferreira — Thu, 25 Aug 2005 08:19:42 GMT

I agree that you should wait for the "reconstruction" to complete, not leveling.

Also, the disk state should be ungroupped and not migrating.

But what i want to know is where in the best practices do you see that only 5GB is needed for maintenance. Our recomendation was to leave 10% of total space unused.

Can you post the link?

Thanks

Re: EVA Releveling

Ivan Ferreira — Thu, 25 Aug 2005 08:29:41 GMT

Ignore my request, is in the document above.

Thank you!

Re: EVA Releveling

Bill Mace_1 — Thu, 25 Aug 2005 08:42:51 GMT

Ivan, The Best practices document link is posted above by Mario. The refence I made to 5GB is on page 18.

Mario, The EVA is not like tradictional Raid 5. The sparing level 0,1,2 determines the number of drives to use for releveling.
The more free space you have means less data to move so it will be faster. But at 98%+ utilization it takes a long time (Mutliple days). As David pointed out, som,etimes a failure occurs during this process, then that drive has to be migrated out, then recovery of the RSS happens again, then the rebuild processs starts.
Host I/O's must continue and this is what additionally slows the rebuild.

The data is in jepordy during the recovery of the RSS and no one seems to know how long this takes.

Running at 99%+ occupancy, I have seen the sparing level drop From Double requested to Single available. Going one step further I have seen Double requested and NONE Available. I believe that this is an error and attribute this to 3.014 VCS. We have array's running at 3.014, 3.020 and 3.025.
With None (sparring level) available I fell that this is a time were another failure will lead to data construction and can not get a definite answer to how long the Recovery phase takes.

David, As you pointed out I have had failures during the releveling and it has started all over, so far no data lost.

Bill

Re: EVA Releveling

Bill Mace_1 — Thu, 25 Aug 2005 08:48:31 GMT

Ivan,

Consider the new 300 gb drives comming out. An EVA with 168 drives at 10% means that 5TB raw is used as free space. HP stated in the best practice that only 5 GB is required for maintenance. That tells me that I could run at 99.99 peresent occupnacy as long as I have 5GB available for VCS code loads.

Bill

Re: EVA Releveling

Ivan Ferreira — Thu, 25 Aug 2005 08:56:37 GMT

Thanks Bill. We had 1 TB free space to follow the HP recomendations.

I will raise the occupancy level to 95%. I will keep enough free space to ensure leveling and data reconstruction, as free space is used as sparing space.

Re: EVA Releveling

Uwe Zessin — Thu, 25 Aug 2005 09:32:02 GMT

> The sparing level 0,1,2 determines the number of drives to use for releveling.

I've never heard that claim before and I've been working for 3.5+ years with EVA...

It sounds like you're talking about the 'protections space' which is set aside in place of any dedicated spare disks. It has nothing to do with leveling.

0 = no reservation
1 = 2x size of largest disk drive in group
2 = 4x size of largest disk drive in group

The 'leveling' will always go over all disk drives in the group.

> The data is in jepordy during the recovery of the RSS and
> no one seems to know how long this takes.

Of course not. It depends on the size of the RSS (number of disk drives), the speed of the disks, the amount of data to recover, the VRAID level of the data, concurrency with host I/O.

> Running at 99%+ occupancy, I have seen the sparing level
> drop From Double requested to Single available.

After a disk failure I'd say. Add back a disk, do not create additional Vdisks and it should go back to double.

> Going one step further I have seen Double requested
> and NONE Available.

Lost two unmarried disks?

> With None (sparring level) available I fell that this is
> a time were another failure will lead to data construction

ANY disk failure will trigger a recovery attempt. Whether it succeeds depends on whether there is enough free space. It can come from the protection space OR the free space for Vdisks, preference is given to the second.

> Consider the new 300 gb drives comming out.

I've already installed an EVA with those disks, so they are there.

-----

Bill, you post from two different accounts:

http://forums1.itrc.hp.com/service/forums/publicProfile.do?userId=CA1315413&forumId=1
http://forums1.itrc.hp.com/service/forums/publicProfile.do?userId=CA1317758&forumId=1

-----

Ivan,
> I will raise the occupancy level to 95%.
> I will keep enough free space to ensure leveling and
> data reconstruction, as free space is used as sparing space.

Define a protection level > 0 and you have set aside space for reconstruction - that's its purpose. The occupancy level is just a warning highwater mark.

Re: EVA Releveling

Ivan Ferreira — Thu, 25 Aug 2005 09:44:05 GMT

My protection level is double, but leaving "extra" free space, if you loose a disk, the free space will be used first to reconstruct the data. If no enough free space available, then "spare space" (configured in the protection level) will be used.

Re: EVA Releveling

Mario_66 — Thu, 25 Aug 2005 09:58:29 GMT

Hello,

Uwe, you should be faster next time. I have almost finished my reply and now it is worthless :)).

BTW, I think that in a case that EVA does not have enough unallocated space, it will temporary use space dedicated for protection as a space for leveling process. I am not sure that it can be visible as decreasing protection level. I have never tried.

Regards,
M.

Re: EVA Releveling

Uwe Zessin — Thu, 25 Aug 2005 10:24:09 GMT

I apologize for being such an underperformer ;-)

Re: EVA Releveling

Bill Mace_1 — Thu, 25 Aug 2005 12:03:05 GMT

Uve,

"Disk failure protection" is the correct term and I refered to it as sparing level.

The nuber of drives is 168 in the default disk group and they are 146GB on most arrary's. The 99% ocupancy is true for about 85% of the EVA's currently installed.

I have two user ID's because when I treid to respond this morning it would not let me in and said I was not registered, I did that stupid thing and automatically log in on my home machine and they could not find me, hence another ID to access this forum and hope that I could get and answer to my posts.

You are also correct that the Disk failure protection level determines the amount of disk space X 2, X largest drive size x protection level selected.

I still haven't found out how much time elapses from the start of reconstruction/recovery of the RSS to the time it starts the releveling. That is the time I am concerned with as that is when I believe the EVA is susceptable to coruption. I do not know weither it is 20 micro seconds or 20 minutes. another factor I know effects the releveling is the size of the vdisks. Smaller vdisk level quicker but I did not want to go there. The disk group is also only Vraid5 so it does not have to do leveling of Vraid1.

Now to make it interesting say that over the period of 2 years the group has had mulipte disk failures. Say 20 disks have been replaced, (please don't quote 3% industry standard failure rate) then it is very possible that the RSS's have been compromised. Is that not correct?

Is the RSS Disk state, None, Parity, Mirrored an indication of the health state of the RSS's.

Bill