LVM Disk Striping

[Tony Peguinho]

Guys,

I have 8x 4gb disks on controller a and 8x 4gb disks on controller b.

These are all to belong to the same volume group and same logical volume in that group.

How can I stripe them so that it forces the 1st disk to be on controller a, the 2nd disk on controller b, the 3rd disk on controller a, the fourth disk on controller b, etc.......

I have never had to do this before as all of our other disks are mirrored.

Thanks

Tony

[Sanjay_6]

Hi Tony,

Look at the thread below.

http://docs.hp.com/cgi-bin/fsearch/framedisplay?top=/hpux/onlinedocs/B2355-90742/B2355-90742_top.html&con=/hpux/onlinedocs/B2355-90742/00/00/48-con.html&toc=/hpux/onlinedocs/B2355-90742/00/00/48-toc.html&searchterms=lvm%20stripe&queryid=20011010-061305

Look for the section, "Setting Up Disk Striping "

Hope this helps.

thanks

[Santosh Nair_1]

Assuming disks 1-4 are on controller1 and 5-8 are on controller2:

From reading Sanjay's link, it looks like you would create the VG:

mkdir /dev/vgNEW
mknod /dev/vgNEW/group c 64 0x0nn0000
vgcreate /dev/vgNEW disk1

then extend the VG using disks from alternate controllers:

vgextend /dev/vgNEW disk5
vgextend /dev/vgNEW disk2
vgextend /dev/vgNEW disk6
...

Then create the LV

lvcreate -L -i 8 -I /dev/vgNEW

this should create the striped LV the way you wanted.


-Santosh

[harry d brown jr]

The term to force striping across disks in a certain pattern is called "extent based striping".

You'll need to write a script to do such, and it will take some time to run it.

The +'s to extent based striping is that you can add another disk to the volume group (but you won't be able to stripe it) without saving/building stripes/and restoring vg.

You might also want to adjust your PE (physical extent) size, depending upon what you are laying down on the vg.

[Sanjay_6]

Hi Tony,

what is it that you want to do. Do you want to mirror disk1 on controller a to disk 1 on controller b and so on. Or do you want that the LV when created should first be created on disk1 on controller a and then on disk1 on controller b, then on disk2 on controller a and so on.

If you want to do the first you should be looking for channel seperation. Refer to section "Setting Up Alternate Links to a Physical Volume" in the thread i've given above.

If it is the second that you are looking for, what you have to do is first create a LV, do a lvextend to cover the area on 1st disk on controller a, then do a lvextend to cover the area on disk1 on controller b, then lvextend to the disk 2 on controller a and so on.

If it is the first, when creating a LV specify the option "-s g" to force the mirror to be created on alternate links.

Hope this helps.

thanks

[Patrick Wallek]

I believe that Santosh's procedure will work as long as the disks you are striping are NOT mirrored. Mirroring and normal striping do not work together in HP-UX (unless you happen to get Veritas Volume Manager and use VxVM filesystems).

If you want to use extent based striping, where your stripe size is the same as your PE size in the VG you can do the following:

Assume we are useing 4 disks, 2 on controller A and 2 on contoller B (the same method can be used for your 16 disks).

Controller A - /dev/dsk/c1t1d0, /dev/dsk/c1t2d0
Controller B - /dev/dsk/c2t1d0, /dev/dsk/c2t2d0

vgextend -g PVG1 vg01 /dev/dsk/c1t1d0
vgextend -g PVG1 vg01 /dev/dsk/c2t1d0
vgextend -g PVG1 vg01 /dev/dsk/c1t2d0
vgextend -g PVG1 vg01 /dev/dsk/c2t2d0

The -g option to vgextend puts the disks in a Physical Volume group (defined in /etc/lvmpvg) which is what controls extent based striping.

You can then create your extent-striped logical volume. The LV will be striped across the 4 disks in the PVG in the order that they are in the PVG and it will go in a round-robin fashion across all disks until the LV is created with the appropriate size. You do need to make sure that you have the latest LVM patch and that your lvcreate supports the '-D' option.

To create a 200MB extent striped LV do the following:

lvcreate -L 200 -D y -s g -n lvol1 vg01


If have some other disks and want to mirror your LV you would put the disks you want to mirror to in a separate physical volume group

vgextend -g PVG2 vg01 /dev/dsk/c3t1d0
vgextend -g PVG2 vg01 /dev/dsk/c4t1d0
vgextend -g PVG2 vg01 /dev/dsk/c3t2d0
vgextend -g PVG2 vg01 /dev/dsk/c4t2d0

And then when you do your lvcreate you would do

lvcreate -L 200 -D y -s g -m 1 -n lvol1 vg01

And you would have a 200MB extent-striped mirrored LV with one copy being stiped across the 4 disks in PVG1 and the other copy striped across in PVG2.

I hope this makes sense.

[Trever Furnish]

Patrick Walleck's response here seems to be exactly what I'm looking for - almost. I'm hoping to do software raid1+0 (aka raid10 aka mirrored stripe sets), in the hopes of gaining improved performance. The performance of raid1+0 improves nearly linearly with the addition of drive pairs.

The only thing worrying me is how to initially create the pvg's, since Patrick's response lists only vgextend commands and not vgcreate commands. If I've already created a volume group without using -g, can I still add it to a "PVG"?

Is this really effectively doing raid0+1 or am I misinterpreting the manual pages?

[Patrick Wallek]

If you want to create PVGs for a VG that already exists then you can just modify / create the /etc/lvmpvg file.

The format is:

VG /dev/vg??
PVG PVG_NAMEa
/dev/dsk/c1t1d0
/dev/dsk/c2t1d0
/dev/dsk/c1t2d0
/dev/dsk/c2t2d0
PVG PVG_NAMEb
/dev/dsk/c3t1d0
/dev/dsk/c4t1d0
/dev/dsk/c3t2d0
/dev/dsk/c4t2d0

[James R. Ferguson]

Hi Tony:

A additional comment to Patrick's excellent examples:

If you ever 'vgexport' a volume group represented in the '/etc/lvmpvg' file (for instance, to move it to a different fiber channel and hence the associated device files are changing) then you find that you will need to recreate the entries for the exported volume group again. You might notice the absence of the "physical volume group" information in a 'vgdisplay -v /dev/vgXX' query after the export until you do.

Regards!

...JRF...

[Trever Furnish]

Ok, thanks! I got the configuration working, realized there was a -g option for vgcreate as well. But now I'm not sure that the benefit is what I was hoping for. How do I benchmark this?

I was hoping for a near-linear write performance increase with the addition of drives to the volume group, since scsi allows disk commands to be sent and executed in parallel rather than in sequence (like IDE), so we would have the reliability of mirroring (since the two stripe-sets are mirrored) and the performance increase of striping (since each mirror is actually a stripe set).

I've seen such a performance change when doing raid0+1 using hardware raid controllers, but I don't seem to be getting it with "extent-based mirrored stripes". For initial benchmarking, I created a mirrored volume as described above and a normal (non-striped) mirrored volume group, and a third volume group with just one drive in it.

- /raidtest2 is extent-based mirrored stripes with four drives in each stripe-set.

- /tmp is two mirrored drives.

- /source is one drive.

For testing, I created a 150MB file on /source and timed (using the time command) a cp of that file to both /raidtest2 and /tmp seperately. To (hopefully) mitigate the effects of disk caching, I first cp'd the file from /source to /dev/null.

On a system with only normal 11.11 running, no applications installed yet, the resulting times vary widely. From 4 seconds to 8 seconds for copying to *either* filesystem. The mirrored stripe sets don't seem to be any faster than the single mirrored drives.

Shouldn't they be? Is this just a really poor benchmark, or a poor implementation of raid0+1?

Suggestions solicited and greatly appreciated.

[Kevin Wright]

Clearly your write to a mirrored stripe set should be much faster than a write to a mirrored set.

In a mirrored set, your write transaction rate will not be faster than a single disk, but your read should be twice as fast.

A striped set is the faster RAID, because you don't have parity of RAID 5.

This leads me to believe your benchmark is inacurrate.

[Trever Furnish]

I'd certainly love to be able to prove that you're right about my benchmark being wrong - but to do so I'll need a better benchmark, one that *DOES* show improvement for the mirrored stripe set.

Any suggestions?

[Sridhar Bhaskarla]

Tony

There are two things to be noticed here

1) LVM striping
2) Extent Level striping

The type of striping you need to do is just dependent on your application needs. If you do a lot of sequential IO, then LVM striping with a non-default stripe size is a good way to go. IF you do a lot of Random IO, both perform the same way.

I personally prefer LVM striping on JBODs and extent level on disk systems like XP's EMCs that have RAID-5.

And obviously you cannot have 0+1 config with lvm striping and you need to go with extent level striping.

You will get good performance with mirroring for reads. But it's a penalty for writes. If you get a chance, try doing the following

1) Create 0+1 - extent level striping and take the numbers for both reads and writes.
2) Create LV striping with the stripe sizes of 8k, 16k, 32k, 64k, 128k so on.. and note the numbers.

YOu can decide on your own. My benchmarks show that the LVM stripe of 256k is the best even with XPs.

-Sri

[Kevin Wright]

You certainly need to adjust your stripe size, as this can fluctuate performance metrics drastically. It also depends, like mentioned above, whether or not your apps use sequential or randam IO.
I would play with the stripe size, and benchmark that to see the difference.