VxVM and VxFS striping on SAN (Oracle Database)

[Soren Morton]

I am looking at setting up a couple of new servers using VxVM and VxFS on a SAN. I want to balance the load over all paths to the SAN as well all disk groups within the storage device.

Since I am using VxVM I am considering using striping, but am not too sure on the stripe size. I have mostly used extent based distribution with the default 4mb size, but I have not seen anyone using a stripe this large.

Unfortunately, I don't have any info about standard read/write sizes, types, or frequency from the application vendor so I know this will be more of a vague answer.

What I am looking for is some idea of what people are using out there and what kind of performance/issues they have run into with different setups. Maybe info about what overhead difference can be seen in relation to the stripe size would be great.

[Steven E. Protter]

1) Fastest Oracle performance in these circumstances would be setting up the LUN on the SAN as Raid 10 Fastest. That is what oracle recommends for high load databases and index files. Also recommended for the rollback space. Good for performance.

2) If you decide the set the SAN for Raid 5, there is no need to use vxvm or lvm striping. That does not improve reliablity, performance in any way.

3) If the LUN's are presenting you unprotected disks, your best course of action is to use two of them and set up Raid 1 mirroring with the add in product for vxvm. If you are presnted with unprotected luns and want to use striping for reliablity, you should have more than two LUNS to stripe on.

The general rule on SAN's is you are better off setting up striping and such on the SAN, not using any OS tools.

SEP

[Soren Morton]

"2) If you decide the set the SAN for Raid 5, there is no need to use vxvm or lvm> striping. That does not improve reliablity, performance in any way.... The general rule on SAN's is you are better off setting up striping and such on the SAN, not using any OS tools."

I am working to distribute the load across multiple disk groups within the storage array as well as across both controllers to get the benefit of the cache in both of them.

Striping will do this most effectively. I have used LVM distribution to do this in the past with 4mb extents and it works great. According to Oracle's document on the SAME philosophy it is best to use a 1mb stipe size but I have not talked with anyone actually doing this. I am hoping to get some input from someone who has worked with this kind of setup.

[Steven E. Protter]

Is your plan to stripe over multiple LUN's?

If you had five LUNs and you striped over them, you'd distribute the load nicely and theoretically have five times the band width.

If the LUNS were set up on the lan as unprotected, unstriped partitions, this would provide you speed and band width.

Most SAN admin's don't do this. They like to spread the data over as many disks as possible and present the disks as raid 5 luns. To use OS striping on top of that will probably impede performance and probably not help much on load balancing.

In the latter scenario the real work is being done on the SAN. That's fine, but I can see no reason to load balance like that.

If you have enough of a load to want to distribute it, you are better off with Oracle
's recommendation for data in a SAN environment. Data and Index Raid 10, redo logs raid 10, everything else RAID 5.

I'm a bit disappointed you didn't get more responses, but perhaps thats the holiday weekend at work.

I think if you look at the detail of our discussion you'll find we agree an the raid 5 issue. So long as the data is not striped on the san, vxvm or lvm striping will help. I'm not sure it will help as much as doing it on the san though.

Good Luck,

Interesting discussion.

SEP

[Victor BERRIDGE]

Hi,
I do some LVM striping on a SAN (HDS9980V), here is an example of a HP5450 with 2 2GB FC:
Class I H/W Path Driver S/W State H/W Type Description
=================================================================
fc 0 0/4/0/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
/dev/td0
fc 1 0/7/0/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
/dev/td1
/var/adm/root $ vgdisplay -v s4vg01|grep dsk
PV Name /dev/dsk/c8t0d1
PV Name /dev/dsk/c4t0d1 Alternate Link
PV Name /dev/dsk/c8t0d5
PV Name /dev/dsk/c4t0d5 Alternate Link
PV Name /dev/dsk/c4t0d0
PV Name /dev/dsk/c8t0d0 Alternate Link
PV Name /dev/dsk/c4t0d4
PV Name /dev/dsk/c8t0d4 Alternate Link
/var/adm/root $ lvdisplay /dev/s4vg01/lvol6
--- Logical volumes ---
LV Name /dev/s4vg01/lvol6
VG Name /dev/s4vg01
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule striped
LV Size (Mbytes) 2000
Current LE 500
Allocated PE 500
Stripes 4
Stripe Size (Kbytes) 64
Bad block on
Allocation strict
IO Timeout (Seconds) default

I tried 32K to 256KB and found 64 is the best for current usage...
I found a gain of 50% compared to non striped

All the best
Victor

[Ralph Haefner]

There was a thread a few days ago about setting up Oracle on a SAN, migrating it from JBOD. If you haven't seen it yet, that might have some more good information for you.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=276015

My answer in that one was that we found the best performance to be setting up Raid 10 luns on the SAN, then striping across 2 of those in VxVM. More details in that thread.

Ralph

[Soren Morton]

Ralph,

The link you gave was quite helpful and contained some good information. Unfortunately, there wasn't any mention of the stripe size. I will probably work with the disk manufacturer (The disk array is a Hitachi 9570) to see if they have a suggestion, but wanted to see what other SM's were using or had tried that didn't work too well.

[Eric Buckner]

Soren,
I have been doing just this for the past couple of years.

I am assuming that whatever product you have in the SAN is taking care of your redundancy.

We are running on an XP512 and I just use the default stripe size of 64 so that I can spread my data across as many LUNS as possible. I also suggest that you set up more than 2 paths to the same device. What we ran into was when we only had 2 paths that we saw some performance problems if one of the paths went down.

Also you want to make sure you plan the number of columns(drives) in your stripe. If you are using a large value for your LUN size this could mean you have to allocate more LUNs than you really need just to grow the disk group. As an example:

My disk groups are maximum of 100G. We have an initial size of 50G using a 13.6G LUN size. If we use a column size of 8 we would have to allocate 108.8G to handle our initial size of 50G. But if we use a column size of 4 it would give us 54.4G, a much closer fit for what we wanted. Of course at a later time we need to increase it by 20G we would be stuck w/ adding an additional 54.4G because that is what we told it the column size would be.

HtH,
Eric

[Alzhy]

Soren,

You have not mentioned what kind of SAN array you're using and what RAID protection the LUNs are at. For Hitachi based arrays (ie. HP XP512,XP1024 and Sun StorEdge 99xx, or the real McCoy..) - no matter what RAID protection your LUNs have, you will always get better performance out of striping accross LUN's distributed accross ACP's (array groups) and accross FC's.

VxVM stripe sizes of 64-128 would mostly suffice for mixed or general purpose loads. For larger read based envrionemnts ie, large DSS/Datawarehouse -- it will depend on the behaviour of your DB and what usual read patterns are to be expected.

I know of sites that use VxVM and VxFS that utilize multiple SAN arrays -- hence, striping is done accross entire arrays and accross a multitude of FC channels. Those environments trying to map out human life would probably employ very large stripe sizes unheard of in normal RDBMS use...

[MANOJ SRIVASTAVA]

Soren

Striping is the right thing to do.
Size of the stripe is determined by the performance when a Xsize amount of data is either written or read on the Y Size of the disk space , so if you read a smaller data on a bigger stripe size it will take more time and bigger size of data on a smaller stripe size .

Veritas / EMC / Oracle Combo we use 128 KB as the stripe size , and there is DMP supported

LVM / HP( HITACHI) / Informix we use 64 KB stripe size no DMP but we use primary alternate magic of LVM.



Manoj Srivastava