Mirroring and Striping?

[Millar Alan]

I understand that you cannot do both mirroring and striping, using the built-in
LVM striping. However, I heard that you could fake striping a mirrored LV by
extending it onto different physical volumes, one extent at at time. That
would give you stripes the size of your extent size.

Has anyone actually tried it, and is it worth trying? If so, has anyone
written a script to do the loop of multiple lvextends? I'm curious about
people's thoughts on it.

- Alan

[Terja Lange_1]

Lets separate the two things.
Mirroring is to protect you against HW failure, whereas striping CAN give you
performance improvement.

I you wish to stripe, be careful, as performance will depend greatly on the
sizes of the datafiles and how big are the queries. If you get the strip size
wrong, you may run x disks flat chat for each query made to them. With striping
you may also run the risk of disk
failure. If one fails, you then need to recover the
entire fiilesystem (unless mirrored) best rule of thumb
if the avg size of files is less than 4mb , stripe in 4 chunks,otherwise stripe
in the avareage size chuncks your files are in.

To mirror the stripe identically, you will need to mirror as you do the
lvextend.

[Dave Wherry_1]

Alan,
We have done this and it does work. We no longer do it because we moved to an
XP256 and do not need to mirror. We are still doing psuedo, or extent striping
so that we can balance the I/O across our 4 controllers. We created the
logical volumes and then did lvextends, 4MB at a time to each disk in a round
robin. This was recomended to us by the XP256 development team out of
Roseville. Performance has been very good and the load is balanced.

Something to watch out for. If when creating the volume group you change any
of the defaults, the lvextend round robin will be pathetically slow. I've been
burned before by having a volume group which was created with say 4GB disks.
By default the max PE per PV is set to the size of the largest disk. Later I
wanted to add a 9GB drive to the volume group. Only 4GB would be usable as
that was the max. To avoid this again I increased the max PE per PV when
creating my new volume groups. I also changed the Max PV to larger than the
default 16 in case I needed to add more than 16 disks in the future. I was
trying to look ahead. I then tried to create 6 logical volumes, each around
13GB using the extent striping method. At the rate they were being created it
would have taken about 16 hours to create these 6 logical volumes. The higher
we went with the max values, the slower the lvextends ran. HP could not give
us a reason for this and we had to go back to or very close to the defaults.
Watch out for th
at.

There is another post in the forum about mirroring and striping, posted on
4/10. There is a good reply on using physical volume groups to set up extent
striping. Check it out.

Dave

[Millar Alan]

Thanks for the responses. The other message on managing it using physical
groups looks good; I'll definitely look into the new lvcreate syntax.

I never thought of using the striping to balance the array IO across multiple
channels. That's a great idea. Thanks!

- Alan

[Dave Wherry_1]

We are using Fibre Channel interfaces and our load
should not come close to saturating a single interface.
You could saturate a SCSI interface so you should
consider it. It also gives you redundancy and the
ability to use alternate paths in case an interface
fails.
Life is getting easier from the old JBOD days.

[Tim Malnati]

Back at 9.x, extent striping was the only thing available. We used it and saw significant improvements in i/o throughput to our database apps. The only issue that came out of it was a situation where a bad block appeared in the set. I can't remember if it was on the primary or the mirror, but it caused an error message to appear every time we split and resynced mirrors. I seem to remember that the configuration has some problem relocating the bad block. Our solution was to swap out the drive (Jamaica) with an identical and recover the data back to the failing disk. I'm not sure what it would have required if a new drive could not be had.

These days we are using array technology (EMC) and the problem is somewhat of a moot point to us. Striping drives on the EMC drastically improves throughput in a mostly read database but requires full mirroring to operate at peak efficiency. I've also created some using raid 'S' but there are a variety of other issues that come into the picture and throughput is not nearly as good.

[Andy Monks]

if your using lvm striping and an array becareful, you don't stripe a stripe!.

I've seen the following situation.

Raid 5, with lvm striping and Oracle (which striped the data across the raw lvols). The end effect was poor performance.

If your using striping using multiple technologies ensure you do some benchmarking to prove you are actually making things faster.

[Emil Velez]

You will probably need to clean up the shell script but this is a start.
Good luck


#!/bin/ksh
#mk_stripes lvol size ( in extents ) physical_volumes ...
#
lvol=$1
integer drive count=0 limit=$2
shift 2
integer ndisks=$#

for i in $*
do
disks[count]=$i
let count=count+1
done
count=0
while [ $count -lt $limit ]
do
let drive=count%ndisks
let count=count+1
lvextend -l $count $lvol ${disks[drive]}
done
# ./script1 /dev/vg00/lvol2 16 /dev/dsk/c[2345]dt0s0

[Stefan Farrelly]

Almost all of HP's internal Openmail servers worldwide use manual extent based striping & mirroring on Nike's. The whole point of using this is so that you can mix data and mirrorred data across all the SCSI controllers and devices. This was benchmarked against many other solutions and found to be the best.

This has a few advantages over distributed extent based striping (lvcreate -D);

1. doing it manually allows you to start each lvol on a different device for an even better stripe.
2. you can mix a data extent and a mirrored data extent on the same device (but of course for different lvols).
3. consequently you can balance much much better the load across all controllers and devices evenly whereas other solutions only tend to use the mirrored channels+devices when the primary gets too busy or drops.

Stefan
HP OSD

[Dave Wherry]

You can use the lvcreate -D and still get the same effect that Stefan talked about in his first point, which I think is a good idea. By simply editing the /etc/lvmpvg file and changing the order of the disks listed in the physical volume group, you can decide which is the first disk used. That spreads your starting disk for you logical volumes around.
I really prefer the -D option because manually extent striping even with a script, can be very slow. I went through it last year and was not aware of the -D option. It would have saved me a day of work.

[Emmanuel Eyer]

Extent Based Mirrored Stripes. That's what it is called. It has been introduced in LVM in a patch (in 10.20), thus it was not always there. It is now in the LVM command patch (the big cumulative patch).

Before that time, we had to use a script, as shown before. Since then:
- you MUST use PVGs
- use option -D y of lvcreate
- everything is in the man pages

Of course, using a disk array is much better... and more expensive :-(

Have fun!