lvcreate Distributed (-D y) vs. Striping (-i n)

[Greg Wojtak_1]

I am creating some new filesystems and it was recommended by HP to us that we use the Distributed policy instead of striping. After reading the man page, neither I nor my co-workers can figure out what the difference is between the two besides the fact that distributed is ostensibly striping based on extent size rather than a defined stripe size. Two things I've noticed, though, with distributed allocation policy: Performance was terrible and watching glance and sar showed that only the first disk in the logical volume was getting hammered, none of the others were.

Can someone better explain the difference.

BTW, the reason HP recommended distributed was because of the larger "stripe" size -- 16M due to our extent size, but we can get 32M (32768K) using regular striping -- ??

[Patrick Wallek · Accepted solution]

The main difference -- Extent based striping (the -D option) allows you to mirror. With normal striping (-I and -i options) you can not mirror.

I'm really not sure why you would want to stripe with 16M or 32M extents. That almost certainly will be much much to large of a stripe size. I'm not surprised you are seeing issues.

Normal stripes are done in the small KB range (8, 16, 32 or 64 KB generally).

[A. Clay Stephenson]

Extent-based striping stripes over chunks of PE size so that the smallest possible stripe is 1MB and is generally much larger on larger physical volumes. The idea is that PE 0 is written to PV 0, PE1 is written to PV 1, and so on round-robining across each PV in the VG.

Conventional striping stripes in much smaller chunks (e.g. 64KB) across the disks which makeup a given logical volume.

In general, the smallest possible stripe for extent-based stripping (1MB) is still much too large to spread the i/o over multiple paths and devices well and often enough for high throughput.

I find that on most devices and applications a stripe size of 64KB to 128KB is optimal.

[Greg Wojtak_1]

>>I find that on most devices and applications >>a stripe size of 64KB to 128KB is optimal.


Do you find that is the case even with large files, ie databases that keep their extents in 1-2GB chunks?

[A. Clay Stephenson]

In a word, yes. Your performance may vary but the approach I have always found to be the most productive and to yield the best performance is:

Choose the size of your VG (e.g. 120GB). Now create as many equally sized LUN's as your have SCSI path's to your array. If for example you have 3 SCSI paths then create 3 40GB LUN's. LUN 0's primary path should be SCSI PATH 0 (alternate 1); LUN 1's primary PATH should be SCSI PATH 1 (alternate 2); LUN 2's primary path should be SCSI PATH 2 (alternate 0). Now you create each LVOl and stripe across each LUN in 64-128KB chunks. This tends to give the best database performance using vxfs filesystems.

You should also avoid RAID-5 and use RAID 1/0 -- basically trading significantly better performance for significantly poorer storage capacity.

[Greg Wojtak_1]

So is will dstributed mirroring not give me the performance I am hoping for (in your opinion)?

[A. Clay Stephenson]

In my opinion (and based upon my testing in the past), I think you are going to be disappointed in extent-based striping from a performance standpoint --- the smallest possible chunks are simply too big to efficiently spread the i/o. You haven't mentioned the type of array that you are using but the idea is to throw as much at the array over as many SCSI channels as possible and let the array sort out where it's actually going. On most arrays, if you use extent-based striping, you can actually see the i/o's move from one controller to another as the extents round-robin. This means that generally only one channel is actually active at any one moment. On the other hand, if you use conventional LVM striping, all of the controllers will be blinking almost continously and the overall i/o rate will be significantly higher.

By the way, if you reduce to a smaller number of equally sized LUN's, it will APPEAR that you have i/o bottlenecks because all any host-based measurement tool like Glance will know is that a tremendous amount of i/o is going through a small number of devices (never mind that the actual LUN might be many physical disks). You can increase the number of LUN's so that the i/o rate per device APPEARS to go down but in almost all cases there is no actual difference in performance and less LUN's are easier to manage.

The only way to know for sure is to test using your hardware and data but I have yet to see a case where extent-based striping wins.

[Steve Lewis]

My recent comparative testing of distributed versus stripes on an SD32k+xp12k pretty much confirms what ACS says. Especially the bit about watching the i/o change LUNs during the backup.

Distributed allows you to mirror which is good for HA Serviceguard and other manageability.

The best performance for us came from groups of stripes over 4 luns which mapped to different LUSE volumes in the xp. Make sure you spread the load over all controllers for all i/o and increase your scsi_max_qdepth to 32 or even 64. Otherwise sar will show your avq increase and IOs may get blocked in the o/s unnecessarily.

Using distributed reduced performance when we had too many luns in the vg - especially when some of the luns ultimately mapped to the other end of the same disks as others. Beware of this. It may be a disk array but it still pays to plan your storage mapping.

So to summarise, your PE size is too big - you could either use real stripes of 64-128k or re-create the VG with a smaller PE-1,2 or 4Mb at most but I don't think it worth using distributed unless you have a campus HA cluster.

[Greg Vaidman]

One limitation of striping (-i n) besides mirroring not mentioned is that you cannot do a pvmove on a regular striped lvol, whereas you can on an extent-based stripe.