Question over disk performance.

[Ronnie Doggart]

Hi All,

I have an Itanium RX4640 server running 11.23 with mirrored system disk and with two sets of mirrored disks configured as follows:

c5t0d0 - lvol1 mounted /u02
c5t1d0 - lvol1
c5t2d0 - lvol2 mounted /u03
c5t3d0 - lvol2

All the disks are 146GB drives and belong to vg01.

There is and Oracle database sitting on the drives and I have had complaints about performance of the rman backup to disk. I have carried out some tests with the following results.

If I copy a 1GB file from /u03 to /u02, or /u03 to /u02 then the copy takes:

real 5:54.9
user 0.0
sys 2.8

By my calculations this means we are doing 3MB/s writing to the drives.

With sar reporting:

Average c5t0d0 95.93 117.83 367 5728 307.03 20.88
Average c5t1d0 91.53 7.15 355 5684 18.93 19.29
Average c5t2d0 74.49 1.66 354 5662 3.10 14.01
Average c5t3d0 0.48 0.50 0 5 0.00 13.48

Surely 3MB/s is poor.

Can anyone suggest how to investigate further.

Ronnie

[Naveej.K.A]

hi ronnie,

have you got the glance plus installed??
If not, its there in one of your application CDs. you get an initial 60 days evaluation version. Glance plus will be able to probe more into the I/O chokes

with best wishes
naveej

[Naveej.K.A]

hi once again,

Your disks are far too busy. Perhaps time to add more disks /reorient your oracle control files, roll back segments, temporary dbf files or shift your user's home directory.

Two pair of disks (as it is mirrored) and oracle running is severly going to thrash the performance of your database

with best wishes
naveej

[Eric Antunes]

Hi Ronnie,

Are you using the same logical volume for the source and destination?

PS: in my case I do the backup via NFS (to another server logical volume) and I get 5Mb/s...

[Ted Buis]

Since LVM mirrors lvols not disks, depending on how they were set up, you could have the lvols on the same drive. Can you confirm that they are on separate physical drives?
Also, for your external drives, are they on the same UPS as the rx4640? If they are, then you might want to consider taking the extra risk of invoking immediate report mode.
By default, HP-UX will have this turned off, because of the increased change of data corruption and loss if the disk losses power while the server does not. Do not try this unless you fully understand the consequences of changing the kernel parameter, default_disk_ir, from 0 to 1. Also, you can change performance, if you have OnLineJFS, by changing how the JFS logs are handled and how freqently metadata is posted relative to data. You can change these mount options easily from 'sam' to make each file system faster, but naturally at increasing risk.

[Ronnie Doggart]

I have confirmed that the mirrors are not on the same physical disk. Also the copy used for the test is copying from one logical volume to another. The disk that shows the highest service time is always the first disk in the volume not the mirror.

All help appreciated.

[Todd McDaniel_1]

Ronnie,

Above are correct, mere size avail is not a determining factor in how your DB will perform.

multiple disks across multiple controllers etc...

My DB filesystems are setup with 8-way stripe 128k size. On external disks not local to the OS.

Second, you may be experienceing problems with the UNIX cache which is not meant to handle Oracle transactions as well as handling the Unix OS activity.

You have several issues here.

1) you are I/O bound
2) you are cache bound by the limitations of the OS cache.
3) your disk layout, even with the amount of Disk space you have is somewhat lacking in distribution of data since it is only on 2 disks.

In my humble opinion, having several smaller disks is preferable to only 2 large disks.


Unfortunately, This problem will follow you until you can reconfigure your environment.

[KapilRaj]

For now ,lvol1 & 2 striped on c5t0d0 & c5t2d0 and then mirroed on to the next set c5t1d0 & c5t3d0

It would be a good idea if you can wire t1 & t3 via a diffrent controller ... yea it costs a controller .

Regds,

Kaps

[Ronnie Doggart]

I know that the disk layout is not optimal but during the normal DB operation I do not currently have a problem. The issue only occurred when RMAN was introduced for backups and it took such a long time to backup 5GB to disk. That is why I performed the copy tests that showed a 1GB copy taking 5mins, giving disk performance of 3MB/s. What I would like to find out is why my disks are only giving 3MB/s.

[Eric Antunes]

Yes,

You should have at least 2 different controllers (You have just one: c5): one for the OS and another for databases and applicatons. Depending one the complexity of your applications you should considerer also a third controller: one for OS, one for database and another for applications...

[KapilRaj]

I think rman works at database level i.e. For an incremental backup it backups up only the changesd oracle blocks. So it takes lots of I/O operations.

Kaps

[Eric Antunes]

Since you use RMAN and cannot yet adding more controllers you should consider doing the backups to a DLT tape.