XP Array Storage Layout: Will stripes of LDEV's give me better WRITE throughput than a Single LDEV?

[Alzhy]

A debate rages on how our ORAREDOs need to be laid out on our XP array (which are carved up into 7+1P array groups). One proponent says - striping accross several LDEVs will give better WRITE performance - would this be so? Others are saying a single LDEV from a less used (and shared) array group will perform better than a stripe of LDEVs from other array groups.

Any thoughts?

[A. Clay Stephenson]

The one rule I follow regardless of the array flavor is to divide my VG's into as many PV's as the array and host have SCSI primary paths. Then stripe each LVOL in the VG across all those PV's (LUN's).

For example, let's suppose that I need 200GB total in a VG divided into a 50GB and a 150GB LUN. Let's also suppose that I have 4 external SCSI connections between the host and the array. I would create 4 LUN's of 50GB each specifying a primary path to each LUN over a different physical bus and choosing one of the other buses as an alternate path. I would then create a 50GB LVOL and a 150GB LVOL and stripe over all four PV's typically in 64KB chunks.

The other thing that should be done is to use RAID 1/0 whenever possible.

[Steven E. Protter]

Paraphrase from Keenen's book on Administration.

Raid 5 uses less space but write performance will be slower than Raid 1 or Raid 1/0.

Other soruces:

A rule of thumb for striping is that being spread across more physical disks(forget LUNS) improves performance.

Test write performance would be from Raid 1, assuming unprotected is a bad idea, which it is.

SEP

[Alzhy]

A Clay, my LDEV's are chunks/slices of RAID5 (7+1P) Array Grouwps. Will the write throughput of a single filesystem/raw device created out of that single LDEV be any better than the write throughput of say a stripe of 8 of these "LDEVs" - assuming the number of channels (FC or SCSI) is the same?

[Sundar_7]

Stripe the LDEVs from DIFFERENT Arrage Groups.

For ex, if you need to use 4 LUNS, pick a LUN for 4 different Array Groups. This is what HP recommends for optimal performance.

Also when you dont have SecurePath, you will have balance the use the primary links and alternate links through different controller.

Remember, even LUNS on a single XP port is controlled by a single chip. So dont pick all even luns or all odd luns seen through any single XP port as your primary link.

[Geoff Wild]

I don't have any XP's - only EMC DMX's - we striped the frame across all disks of same size. Example - we use 72 GB and 146 GB drives - so on the frame, we stripe acorss all the 72GB drives - and then setup 8GB luns. We use the 146GB drives the same - except those are 32GB LUN's.

Rgds...Geoff

[Alzhy]

How "WIDE" do you stripe? Accross a minimum of 4 or 8 LUNs or as WIDE as one can get.

On the XP -- if I have - say 100 Array Groups, (25 for each ACP pair) - would it be advisable to stripe accross those 100 LUNS (Ldevs) 1 from each Array Group? And say have a stripe size of say 512 bytes?


On a slower XP512, other array architectures.. I have never striped no more than 8 ways with stripe sizes ranging from 16 to 128 Kbytes.

[Steven E. Protter]

The more physical disks you strip across, the better.

You need to make sure the on the back end that each LUN is striped across the maximum possible number of disks.

SEP

[Alzhy]

Yes but inmy example, does it REALLY make sense to stripe 100-way?

ie. in VxVM (a very wide THIN stripe):

vxassist make stripevol 200G ncols=100 stwidth=512

and in LVM (a 100 way stripe with 64KB stwidth?):

lvcreate -L 204800 -100 3 -I 64 /dev/stripeVG

[Sundar_7]

striping across 100 LUNS of OPEN-E emulation will be a overkill for a single filesystem.

the stripe size, as you know, the bottom line is to match as closely as possible with your application I/O block size.

[Alzhy]

The LVM example should have read:

lvcreate -L 204800 -i 100 -I 64 /dev/stripeVG


If each LUN is coming from say a 4 way raidset, thin this particular LVOL will essentially engage all 400 Physical Disks.



Or would this be allowed at all in LVM?

[Alzhy]

Sundar,

My plan is to utilize 8-way stripes - with each member coming from 2 array groups from each ACP controller (on the XP).. Somehow, we've a number of "consultants" that are saying.. we shuould stripe all accross the almost 100 Array Groups that we have - with each array group's LDEVs shared amongst the connected servers...

My initial plan was to allocate a number of array groups per server with that server having exclusive access to the LDEVs of each array group as what I am used to with the Hitachi flavour of the XP...

[Sundar_7]

LVM would not care. It will stripe the LV across 100 LUNS with stripe size of 64KB.

The size of the LV will possibly be rounded and while creating the VG make sure you set the max PVs to 100.

[Sundar_7]

As I said, HP recommends to stripe the FS across LUNS from different ArrayGroups. But striping the FS across LUNS from 100 AG doesnt sound good to me.

It would be a good idea to make the LDEVS from different AGs visible to each host rather than having each machine see LDEVs exclusively from certain AGs.

[Jean-Luc Oudart]

Hi Nelson,

During a benchmark we had our database on XP with no stripping across LUNs and got qlen > 0
We then changed the layout with 2 array groups in our vg swith stripes 8 way, 64K.
Qlen was at zero !
We use RAID 1+0. There is an internal strip in the XP but I don't have the details.
I suppose READ may impact too and unless you have dedicated path for the redo.
Obviously I would recommend RAID1+0 as this would help.
Too me WRITE will go into cache.

Tuning redo will require the help from your DBA. Statspak report will give you hint to where your combined effort (DBA + Sysadmin) should concentrate.

Regards
Jean-Luc

[Q4you]

Nelson,

we have encountered similar ( and very heated debates over this here) issues and found that there is *no rule of thumb*. Agressive stripping may deter the performance. Lot of folks here may haved used "old disks" of a very very high disk latency and their experince may be based on those old tech wrecks. If you buy a truck, it will never be tuned to a racer car.

XP technology has taken care such things internally using extensive caching technology so we don't see such performace hits, right ? The newer hardrive technology and RAID 1+0 are already at work in the back ground. If you stripe too much, every chance that you will kill XP disk performance and HP sales team will be ready at your door step to sell you more cache me & shared memory :)

Large small accesses across 100 disks will bring 100 tracks in the cache and one single byte mod will cause multiple disk access. To complete the commit, all 100 disks have to comply at the same time, weakest disk will drive the performance ( or lack of it) !

Now you decide : Would you consider an 18 wheeler a better performance vehicle since all 18 wheels are rotating or 4 wheeler with more powerful engine making just 4 wheels spin faster ?


IMHO

-Q