File System Performance vs. # of VGs or # of LVs

[George Lenzer]

Recently we had an issue on our production Oracle server which was largely due to the hardware not being enough for the task. However, an Oracle consultant suggested that instead of having our redo logs on the same file system with our data files that we should split them out on their own file systems. Everything is hosted on an HP VA7410 SAN (and we're now moving to an HP EVA4000). As a result I wasn't sure these changes would have any real effect:

Before we made the changes our consultant recommended I had the following layout (just examples):

/dev/vg05/lvol1 mounted as /oradata

The above file system contained all data files, indexes and redo log groups.

After the changes we would up with this:

/dev/vg05/lvol1 mounted as /oradata (Now with only data and index files)

/dev/vg06/lvol1 mounted as /ora_redo1
/dev/vg07/lvol1 mounted as /ora_redo2

The above being for the two redo log groups

/dev/vg08/lvol1 mounted as /ora_archlogs

The last file system being for new archive logs since we enabled archive logging.

As I said, I was assuming that since the file systems were all housed on the same group of disks sliced out of the SAN and going over the same fiber paths, switches, etc... that there would be no change. But the changes did reduce some of the disk I/O load and there was a bit of a performance increase. Enough for us to live until we moved to new servers and the new SAN.

Later on I asked someone from HP how this could have made a difference and it was explained to me that it would have reduced some LVM overhead in kernel space. That answer cleared things up a bit. But now I'm certain I've gone about this all wrong since I've created a completely new volume group for each file system, instead of creating logical volumes. Doing this (on HP-UX 11.11 and 11.23) I run into the maxvg limits which I had to raise on one system to 25 and might wind up passing that.

So would I still keep the LVM kernel overhead down if I move to a layout like the following?

/dev/ora_vg/oracle10g mounted as /oracle10g
/dev/ora_vg/oradata mounted as /oradata
/dev/ora_vg/redo1 mounted as /ora_redo1
/dev/ora_vg/redo2 mounted as /ora_redo2
/dev/ora_vg/archlogs mounted as /ora_archlogs

That way I'd have one VG with multiple LVs. It would also make my management of the LUNs on the SAN a lot easier. (I'm using LVM to stripe across two LUNs with multiple alternate paths for manual load balancing)

Or should I stick with my current multiple VGs for better performance?

[Sharma Sanjeev]

Hi George Lenzer

Its always better to have data, index, redo on diffrent luns & if in archive mode than arch also on diffrent.
we can have max 256 VG's in system abd default is 10

Regards
Sanjeev

[Steven E. Protter]

Shalom,

This seems like a reasonable suggestion.

Years ago, Oracle said to me: index, data and redo need to be on raid 1 or raid 10 disk, because they are write intensive.

How the writes take place to index and data is very different than how redo works. So it might make sense to have rdo on its own disk.

Since a volume group can handle 255 physical disks or LUNs, you can still keep it all in one volume group, yet do a little specialization.

Why raid 1 or raid 10? To do less actual writes. Raid 5 stripes your data over a disk array. Performance increases with the number of disks, how over in a typical raid 5 parity nine configuration a single disk write needs to take place in 45 disk locations.

That takes times and disk heads need to do a lot of moving around to make it happen.


raid 1 or 10 can reduce the actual number of writes needed.

This is worth a try. You can have a single volume group for the database, which you need to have if you go with a high availability SG solution, because your database startup script will also activate your volume group.

How you build the volume group is up to you.

I'd like to see you try this and report results. Do a series of performance tests before the change and after so you have a good benchmark.

>>
So would I still keep the LVM kernel overhead down if I move to a layout like the following?

/dev/ora_vg/oracle10g mounted as /oracle10g
/dev/ora_vg/oradata mounted as /oradata
/dev/ora_vg/redo1 mounted as /ora_redo1
/dev/ora_vg/redo2 mounted as /ora_redo2
/dev/ora_vg/archlogs mounted as /ora_archlogs
<<

I don't think this is an LVM kernel overhead problem. It is more likely write contention between redo, index and data, or poor sql statements hammering the database. So you may do all this work and have to go back to the application developers and have them improve their sql.

Your DBA should be a gatekeeper to keep sql statements from making your system and therefore you unhappy.

SEP

[Olivier Masse]

The problem I've learned the hard way in the past is that since the SCSI queue length is defaulted to 8 on 11.23, any filesystem backed by a huge unique LUN can get bottlenecked quickly on an EVA. Spreading "problem" areas within that filesystem such as the logs to a completely different VG (thus ensuring that a different LUN is used) is an easy fix for this problem.

While the best solution is simply to increase the queue length, you have to be careful because not all devices can benefit from this. I think for example that internal SCSI disks should not have their queue length increased.

11.31 comes with a feature named DDR which lets you assign low-level parameters such as the queue length using a mask, and it will be set automatically depending on the exact model SAN array the OS detects, it's much easier to manage. I simply set the DDR to put automatically the queue length to 32 for EVA arrays, and default for everything else.

Good luck

[butti]

Hi all,

/package1/ors Raid5 (ORACLE_HOME)
/package1/ora_data1 Raid5
/package1/ora_data2 Raid5
/package1/ora_redo1 Raid1
/package1/ora_redo2 Raid1
/package1/ora_exarch Raid1 (export and Archivelogs)

We have the Luns in a similar way you see above.


On HPUX11.23 we have increased the "queue_depth" for alls Disks (LUNS) with following command

scsictl -m queue_depth=16 /dev/rdsk/c4t0d1
scsictl -m queue_depth=16 /dev/rdsk/c4t0d2
scsictl -m queue_depth=16 /dev/rdsk/c4t0d3

greeting,
Butti

[Olivier Masse]

Butti, keep in mind that scsictl does not keep the queue across reboots. You need a startup script that runs it each time you boot your server. As an alternative, you can use scsi_max_qdepth but it will be applied to ALL disks and as I mentioned above this might not be what you want. Using DDR fixes these problems with 11.31.

[Emil Velez]

Before we make alot of assumptions you may want to extract the lvm and disk metrics from measureware and see what the req-queue and

extract -xp -d -f /tmp/disks.txt

look at bydsk_request_queue and the other metrics

extract -xp -z -f /tmp/lv.txt
look at the LV_RED_RATE and write rate

I make no assumptions till I look at the data.

Good luck.

[Torsten.]

Many people can tell you long stories about VA family and performance... can you please post a

# armdsp -a

to check your current config?

[George Lenzer]

The SCSI queue length comments are exactly what I was thinking of. This is what the HP consultant suggested (and I think of as kernel overhead since the drivers are in kernel space *I think*).

I am trying to get GlancePlus installed (we supposedly have a license that we bought but that's another topic). Is Measureware part of that or is it a separate product? I'd like to get some metrics on our real disk I/O performance so that I can determine if we'd benefit from increasing the SCSI queue lengths for our Oracle LUNs.

As much as I'm not enjoying managing ten LUNs for Oracle (originally it was only two LUNs when everything was on one VG) if it is the best route to maintaining the best disk I/O, I'll live with it. If changing the SCSI queue lengths will help squeeze out any more performance, I'll take that too.

Regarding the SQL itself, we don't have a DBA. I'm about as close as we get since I can work at the sqlplus prompt. The application that makes the SQL statements (from a third-party vendor) is not under our control. We can't alter the SQL they make and expect to be supported. We did call in a SQL consultant who looked over the various statements that are frequently used and he seemed to be of the opinion that things are being done "right". But we have no ability to tune the SQL. So the best I can do is provide the best disk I/O possible via a combination of alternate paths, LVM striping (2 stripes per file system) and now possibly tuning the SCSI queue lengths.

If anyone else has any further suggestions regarding LVM and disk I/O I'd love to hear them. If not, I'll close this thread in a few days. Thanks to everyone who responded! I've been on ITRC for a few years but using my supervisor's account. I finally decided to reactivate my own, but as usual ITRC forums do not disappoint!

[Torsten.]

The VA itself could cause performance problems, examples:

- data are converted to RAID5 (not enough free space to keep this mirrored)
- no caching because of bad batteries
- you access LUNs via the alternate controller instead of primary
...

that's why I asked you for the armdsp output.

[George Lenzer]

Torsten, we are currently on an EVA4000. We migrated from the VA7410 so it's been decommissioned for production use. Is there an equivalent to armdsp for the EVA4000? Can I get similar output from SSSU?

[George Lenzer]

Torsten; one other thing. We are not currently experiencing disk I/O problems with our current set up on the EVA4000. I am merely trying to squeeze out the best possible performance to try and extend the longevity of the array's usefulness. When we moved to the newer servers and new array our performance issues went away. However, I don't think things are as optimized as they can be. This is simply the effect of more powerful hardware meeting the needs of the situation. Not better configuration getting the best performance possible, which is where I want to get to. Thanks! :)

[Torsten.]

OK George, my understanding was you currently have performance problems with the VA.