performance question

[B. Chapman]

Experts:

I'd like to make a significant change on one of my systems - and I'm wondering if the performance will be *severely* impacted: We run a fairly large Oracle database that is spread across 20 x 8 GB mountpoints. Each mountpoint is ~80% occupied. I'd like to backup the database (cold), rm -rf all of the data mountpoints, lvremove all of the lvols, then build ONE BIG 160GB lvol, call it, say, /BIGMOUNT, then create 20 symbolic links for the "old" directorys to point to newly created directories under /BIGMOUNT (so there will be no disruption in config files, etc). Of course, then kickoff the restore.

Will performance suffer from:
-One big lvol instead of multiple small lvols
-All reads/writes will have to follow the symbolic links

I should've mentioned - all 8.5 GB disk targets are EMC targets over Fibre Channel.

Any advice on this is greatly appreciated!

Thanks,
Ben Chapman
chapmanb@saic.com

[John Poff · Accepted solution]

Hi,

We've tried it both ways, and we prefer to have more, smaller mounts than a single large one. The general performance doesn't seem much different for the application, but backing up the data was a real bear. We use OmniBack and it makes an object for each filesystem. With lots of small filesystems we found that the backup streams much better than with a single large filesystem which winds up being just one object.

We also use the same size PVs on EMC via Fibre Channel.

Any particular reason why you want just one big filesystem?

JP

[Chris Wong]

Wow, lots of work. If going to do all this and the associated downtime, you might want to consider moving to VxVM at the same time. Some questions:
Are you using LVM or VxVM?
Are you using JFS or raw?
Do you have Online JFS?
How many controllers are you using?

- Chris

P.S. The quick short answer is "yes".

[James R. Ferguson]

Hi Ben:

I don't think you will see a measureable difference in performance.

I assume that you are running VxFS filesystems of at least disk layout version-3 so that filesystems can be at least 1TB in size.

I doubt that traversing the symbolic links you will add will add measurable overhead.

Since the filesystems in question are database ones, I presume that the overall number of files and directories within (and therefore inodes) is realatively small. Thus directory searches against one mountpoint will (for all purposes) not be more than an order of magnitude larger than currently; and at that, rather small.

One concern I would have, however, is that by combining all of your logical volumes into one, you loose the ability to do maintenance on subsets of the whole. Too, you will not be able to alter the mount options on any subset of the database structures to optimize performance.

Regards!

...JRF...

[Bill Hassell]

Since the EMC is handling all the disk management, issues such as mirroring and striping are transparent. So the only question concerns one big lvol versus 20 small ones.

The symlink issue is of no concern because once the file is open (ie, when Oracle starts), the symlink is translated to an inode for the file control block in the program and will not be used until the file is closed.

I see the exercise as a waste of time since you may be unable to measure any difference at all. Parallel backups are done very well with Hiback and Omniback so the smaller lvols may be an advantage simply for backups.

[Vincent Fleming]

There's a whitepaper out from Oracle (available on their website) on a concept they call S.A.M.E, or Stripe And Mirror Everything. It's worth reading, and has direct application to your ideas.

The general idea of having one big filesystem for your database is not bad in and of itself, (and previous posters have pointed out some of the ups and downs), but I still like the idea of seperating the logs, indices and tablespaces. If nothing else, if one filesystem goes down, the remaining ones may be used in rebuilding your database.

I would highly suggest using LVM or VxVM striping in building larger filesystems. The load balancing effect can dramatically increase your performance.

Typically, you want to pick your EMC volumes with some care - you want them all to be on different mirror pairs, and evenly balanced over the backend directors (ACP pairs for those with XPs). In other words, take some care in balancing your I/O on the storage array. Try not to use more than one LUN from each mirror pair, if possible. Spread out that load.

Then you will get the most out of the striping.

Like I said above, I like about 3 filsystems; one for logs, one for indices, and one for dataspaces. Each should be a stripeset in LVM or VxVM; striped over several mirror pairs.

Generally, this will give the best of both worlds - simpler backup, and real good performance.

Good luck!

Vince

[A. Clay Stephenson]

Performance-wise I doubt there will be any significant changes BUT if you go to one filesystem then one thing that you do give up is flexibility. You may find that datafiles and indices perform best with mount options which bypass the buffer cache (convosync=direct,mincache=direct) while archive and redo logs perform best with conventional mount options.

I would at least divide into two LVOL's (they cerainly don't need to be the same size) so that you can play with mount options.

[Sridhar Bhaskarla]

Hi Ben,

One of the disadvantages of having one BIGMOUNT is the manageability. Though you have EMC as the backend, sometimes you may want to move the datafiles from heavily accessed luns to idle ones. It is not possible if you have one filesystem.

I would suggest to keep around 3 or more filesystems. Devide them based on the activity. For ex., use one file system exclusively for sequential access, one for random and the other for redo. Let EMC create LUNs for you in the same manner.

-Sri

[B. Chapman]

Many thanks to all who replied! I will assign points right after this closing post.

To sum up (and answer some questions): We're in a slight jam with being overbudgeted on our disk farm allocation - and I was really looking to "free up" some EMC disk targets to toss back into the free pool.

Here is a 'bdf'...

/dev/vg41/lvol4 8835072 8514198 300881 97% /z01
/dev/vg41/lvol5 8835072 8508062 306634 97% /z02
/dev/vg41/lvol6 8835072 8093283 695486 92% /z03
/dev/vg42/lvol1 8830976 7837294 931640 89% /z04
/dev/vg40/lvol1 8835072 8597621 222674 97% /z05
/dev/vg42/lvol2 8830976 8452815 354590 96% /z06
/dev/vg42/lvol3 8830976 8582189 233302 97% /z07
/dev/vg42/lvol4 8830976 8513054 298117 97% /z08
/dev/vg42/lvol5 8830976 8394472 409279 95% /z09
/dev/vg43/lvol1 8830976 7617059 1138107 87% /z10
/dev/vg43/lvol4 11927552 10380179 1450719 88% /z11
/dev/vg42/lvol6 8830976 7447011 1297514 85% /z12
/dev/vg40/lvol2 8835072 5285019 3328233 61% /z14
/dev/vg40/lvol3 8835072 6506134 2183704 75% /z15
/dev/vg40/lvol4 8835072 8707542 119562 99% /z16
/dev/vg40/lvol5 8835072 8195542 599627 93% /z17
/dev/vg40/lvol6 8835072 7756326 1011342 88% /z18
/dev/vg41/lvol1 8835072 6590123 2104640 76% /z19
/dev/vg41/lvol2 8835072 6893885 1819891 79% /z20
/dev/vg41/lvol3 8835072 5888318 2762586 68% /z21
/dev/vg43/lvol5 8830976 7014845 1702624 80% /z22
/dev/vg43/lvol6 5734400 4159549 1476489 74% /z23
/dev/vg43/lvol2 8830976 7331761 1405685 84% /z24

If you were to total the three columns, you'd see that I have 194 G total, 168 G used, and 25 G free. It's the *25 G* free that I was looking to gain.

I'm also a DBA (a lazy one) - and I think I'm going to have to reorganize the datafiles in an attempt to free up *two* mountpoints, which would free up two targets. (What a pain though!). I'll do that by either offlining tablespaces and renaming datafiles, or, I'll create a new controlfile with new file locations.

In closing - I AM NOT GOING TO BUILD THE BIG MOUNTPOINT - mostly because of the OmniBack issue. We *DO* use OmniBack, and that would definitely impact the backup time (I remember facing this problem in the past).

I think another important point is the fact that we do not separate redos, dataspaces, or index spaces - they're all mixed across those 20 mountpoints - and THAT'S where I could pickup performance gain - by separating those - and using different vxfs settings for each category.

Thanks for all the posts - and everyone have a great weekend!

Later,
Ben Chapman
chapmanb@saic.com