Extent based striped mirrors

[Greg Hall]

Does anyone have experience with extent based stripes and mirrors? I am building a new system at HPUX v11. The core application is based on Universe, and the data for thses appls will be on 6 x 4gb drives configured as 3 mirrored pairs. I want to stripe these to get an even load. I am also using JFS filesystems. I had thought raid 0+1 was supported but now find it isn't.
- is there a need to reduce the default 4mb extent size to the minimum 1mb?
- are there any known performance issues, particularly with JFS? I seem to recall when striping (raid 0 not extent based striping) with JFS it is recommended to use the largest stripe size (64kb). Is there an issue with an extent based stripe size of 1mb or 4mb?
- any other input or direction to documentation would be appreciated.

thanks, Greg.

[Patrick Wallek]

I have had very good luck useing extent based stripes. I have generally used the default 4 MB PE size and I have not had a problem. I also have online JFS installed and I have noticed no problems with it either. I don't use a lot of the mount options that you have with online JFS. I mainly use the ability to extend a filesystem on the fly.

If I had my choice and a large budget, I would replace my AutoRaid 12Hs with JBOD (Jamaicas) and use extent based striping and mirroring.

[Manju Kampli]

Greg,

You have to Use PVGs for mirroring with striping. See man lvmpvg for more info how to use PVGs.
Once PVGs are ready, Use LVM commands to create VG, LVs. rememer to use PVG name instead of disk device name. Normal LVM does not support mirroring with Stripping.

Hope this Helps
Manju

[Stefan Farrelly]

HP used extent based striping on all their internal email (Openmail) servers European wide (and most in the US). We used a PE size of 4MB and striped+mirrored over as many disks as possible. Performance was fantastic. You couldnt get it any faster unless you replaced your disk subsystem. Its basically RAID10 - using all available disks+controllers.

This was in the days before the -D option on lvcreate. This is probably the preferred method to use as its a bit more structured and easier for admins to see how its setup. Manual extent based striping and mirroring can be a bit confusing to the eye if youre not used to it. Dont forget if you use the -D option to rotate the disks in your /dev/lvmpvg file across controllers so that the stripe+mirror is distributed evenly over all controllers - you dont want the first PE of all your LV's to be on disks on the same controller - the JFS log is there and will kill your performance.

[Dave van Nierop]

We use extent stripes on all of our customers application servers run on HPUX 10.20 and HPUX 11.x and haven't had any problems [incidently, these are also uniVerse servers]. Like Patrick, we use the 4MB PE size and JFS defaults. I know there are a lot of good approaches to accomplishing this goal, but thought I'd attach and example of how we do ours.

[Les Schuettpelz]

Dave Fargo using Les's ITRC login... this should be a hot thread, lots of built-in issues that many are dealing with.

You are looking at a very small configuration, this will force a 'stripe width' of 3 disks, does this map well on the number of controllers available to the volume group? Also, because the total space is so small, you would have the option of using a fairly small LVM PE size, this is not an option as you get into larger VGs unless you go to Veritas VxVM.

With the LVM -D option, you are stuck with the VG PE size as the 'stripe depth', the strip size must be the PE size, that's the price to be paid for the automation you get. So, in LVM, the larger the VG, the more the LVM header size limit forces you to go into larger PE sizes (just to store the LVM metadata), and this relationship is probably not 'naturally optimized'.

Has anyone done performance benchmarking with -D option implemented in varying VG sizes? Assuming relatively simliar seek and rotation factors across varying disk sizes, has anyone found 'rules of thumb' for selecting the PE size for a VG that will be -D stripe/mirrored? It has to be more than just 'the smallest it will let you use'.

Concerning the controller address of LE 0, frequently there will be more than 1 disk on each controller in the stripe group, especially in Fibre Channel and SAN. Are there rules of thumb that would limit how deep you should stripe on a single controller? Some sources indicate that there is still a benefit from doing this, due to the parallel mirror scheduling policy and the ability of the channel to schedule the I/O's faster than the bandwidth of an individual disk. Comments?

More information on the JFS implications would be helpful. What indicators would point to JFS log contention? If you lay things out very 'square', as would happen if you lvcreate -D the striped lvols to the exact physical disk size, giving the same number of lvols as you have mirror pairs, then you would end up with LE 0 not only on the same controller, but also on the same physical disk. Is this really a serious concern, or is it over-rated? Will the stripes and mirrors of multiple active lvols schedule more efficiently if they have a symmetry across pnysicals, or if they are laid out as randomly as possible within the -D format?

[Greg Hall]

Thanks, everyone for your replies. Whew! It seems I need to understand a bit more about striping issues. But sounds like its a reasonable approach.
The disks will be spread evenly across two controllers. I do intend to use the lvcreate -D option. I'm not a big fan of manually administering disk usage.
Stefan, I'm not sure I understand about rotating disks in lvmpvg but I'll do some more research.
Dave, you raise some very good questions. I haven't been able to find much information on performance issues with JFS and striping. There is some in KBRC00006560 if you haven't seen it already. They mention limiting striped VG's to 4 or less disks, due to diminishing performance returns from there on.
There are some complex technical issues indeed. However I also have the impression that some people have just gone ahead with all the defaults and are satisfied with the results. I guess I'll just suck it and see. Yikes, did I really say that.

Points coming soon.

[Greg Hall]

Are there configurable cache issues I should be looking at to alleviate any write overheads?