LVM stripe and mirror

[Paul Cadieux]

Hi,

I have four disks which are striped and mirrored. The setup basically goes like this: 3 disks are striped together and then mirrored to a larger fourth disk. A partial copy of lvdisplay shows the mirrored portions. I have included only begining and ending PE's on each disk in order to minimize the amount of data.

****************************
lvdisplay -v /dev/vg00/lvol7 |more
--- Logical volumes ---
LV Name /dev/vg00/lvol7
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 1392
Current LE 87
Allocated PE 174
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c2t5d0 87 87
/dev/dsk/c4t3d0 31 31
/dev/dsk/c4t4d0 29 29
/dev/dsk/c4t5d0 27 27

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
0000 /dev/dsk/c4t3d0 0184 current /dev/dsk/c2t5d0 0184 current
0030 /dev/dsk/c4t3d0 0214 current /dev/dsk/c2t5d0 0214 current
0031 /dev/dsk/c4t4d0 0216 current /dev/dsk/c2t5d0 0471 current
0059 /dev/dsk/c4t4d0 0244 current /dev/dsk/c2t5d0 0499 current
0060 /dev/dsk/c2t5d0 0514 current /dev/dsk/c4t5d0 0004 current
0086 /dev/dsk/c2t5d0 0540 current /dev/dsk/c4t5d0 0030 current


*****************************

c2t5d0 is the large disk. c4t3d0, c4t4d0 and c4t5d0 are the smaller disks. It would sense to me that the 3 smaller disks be striped and then mirrored to the larger disk, BUT....

Take a look at the last two lines of the output. It appears to me that the striping and mirroring is occuring as follows:

c4t3d0, c4t4d0 is striped with a portion of c2t5d0. Lets call this stripe one.

There is another stripe setup (call it stripe two) which includes a portion of c2t5d0 and c4t5d0.

Stripe one is then mirrored with stripe two.

My questions are:
1. Am I correct or incorrect. If no, then can someone please explain to me where I am wrong.
2. Is stripe one the primary stripe from which the OS will read? I'm basing this on how striping in NT works.

Thanks,

Paul.

[Sridhar Bhaskarla]

Hi Paul,

Sorry for the late response. This post should have been in LVM category.

In a nut-shell, you cannot do striping and mirroring together with HP's LVM. The best that you can do is extent level striping which means you will need to have two sets of disks. Primary stripe set and the secondary stripe set that acts as mirror. You will define LVM PVGs to achieve it. The stripe size on extent level striping would be the extent size (4MB minimum). So the benefits of extent level striping are small. But better than regular striping though.

If mirrored with default options, LVM will read from both the disks. So, you can always gain from reads. However, attached to it is write penalty.

In your case, it is not possible to stripe three disks and have them mirrored to one disk

-Sri

[Steven E. Protter]

best bet is to distrubute filesystems on the smaller disks and then mirror them all to the big disk.

Yes, if the big disk goes, you are vulnerable, but you'll get a dmesg in time to replace the disk and rebuild the mirrors.

Striping is normally done for performance, and with your current configuration, it doesn't help.

Mirroring will provide you the ability to survive up to three disks dying, so long as they are the small ones.

SEP

[Colin Topliss]

FYI,

You can decrease the extent size to 1Mb rather than leave it set to the default of 4Mb.

If this is going to be used for an Oracle database, then (according to the recent recommendations I came across) a 1Mb stripe size is not a bad thing (goes against the 64K striping rule we always used to use, but times are changing). Unfortunately I don't have the document reference at the moment - I'll dig it out if you're interested.

[Paul Cadieux]

Thanks for the responses. I am more confused now though. Based on the output given in my first post, one can see that a single logical volume (lvol7) is spread across the four disks. If you look carefully at the breakdown of the logical extents it seems to me that my previous analysis (the lvol lies across 3 disks and is subsequently mirrored) holds true. In the responses everyone is saying that this cannot be. Please explain then what I am looking at based on the output given. If more output is needed please let me know and I will provide it.

Thank you

Paul.

[Sridhar Bhaskarla]

Paul,

What you provided is not pure striping. Look at the lines

Stripes 0
Stripe Size (Kbytes) 0

They tell us that you did not do striping. It is extent based mirroring/striping that I mentioned before. Here the stripe size is equal to the extent size. So, the gains are minimum with this striping. However, you may benefit a little from the mirroring from reads but there will be overhead on writes.

-Sri

[Michael Duthie]

Paul,

Your LE numbering looks strange to me, it should be in sequence even with your 8M extents.

Can you attach a full lvdisplay

[Paul Cadieux]

Sridhar,

I see what you are talking about. No striping. Could one say that these are the equivalent of two volume groups (sorry for the windoze terminology) which are mirrored? If so, please look carefully at the allocation of physical extents on c2t5d0. Does this look like it was setup is a very strange way. BTW I did not set this up. I'm only trying to fix it.

Thanks,

Paul.

[Michael Duthie]

Paul,

That looks better, it was just the way it was cut & pasted. I would say lvol7 (/usr ?) has been extended a couple of times and during the last extend the disks were entered the wrong way round on the lvextend command.

If it is /usr then you have to be careful trying to fix it.

The easiest fix I think would be to remove the mirroring and then move the data from c2t5 to c4t5, then mirror it all back to c2t5.

You need to read up on lvreduce, pvmove & lvextend.

If you need to stripe it you need to blow it away and recreate it, either in a different VG or with Ignite.