Very Busy Mirrored Root Disks

[Ty Roberts]

Pete,
With that amount of Sockets do you see your root disks being hit hard??? We also have a vxfsd process that seems to pop up everyonce in a while and hit the root disks hard.. have not been able to find much information on this process and if there is a way to harness or configure it...

THanks,
Ty

[Stefan Farrelly]

The vxfsd process is a system one which actually does the I/O to JFS filesystems. There is nothing you can do to configure it - its an essential HP-UX process. Its not causing your problem - something else is asking it to do lots of I/O and you need to find out what.

[Pete Randall]

Ty,

Yes, my root disks are getting smacked around quite a bit - but not /var in particular. It's mostly /opt, /usr, and, interestingly enough, / itself. Since we don't have any performance complaints, I'd never really looked at until I started reading your post - now I'm starting to investigate.

Pete

[James R. Ferguson]

Hi Ty:

With regard to the buffer cache. Since you are using Oracle, a large buffer cache is just "double-buffering" for Oracle. As indicated, reduce the buffer cache downwards.

The preferred management scheme is the dynamic buffer cache rather than a static one as controlled by 'bufpages' and 'nbuf'. When you get to 11.11 this works better than in previous releases too.

To enable the dynamic buffer cache you must set 'bufpages' and 'nbuf' to zero. For your environment, I'd set 'dbc_min_pct' to 2% (the minimum allowed) and 'dbc_mac_pct' to no more than 4%.

Regards!

...JRF...

[Stefan Farrelly]

Ty,

we have a similar problem here - but not as bad as you. Our largest Oracle server is hitting the vg00 disk medium hard and Ive been trying to find out what/why. Ive found a good way to narrow down which lvol it is;

1. run gpm
2. produce graph IO by File System
3. sort by io rate

Doing this on ours I can see /usr is the one being hit. So much for /var/spool/sockets ! I will investigate further but you need to do the same yourself.

[Pete Randall]

Ty and Stefan,

From Glance I/O by file system, my top three are:
/ 560 (logical i/o rate)
/usr 330
/opt 313

I'm trying to figure out what is hammering the root file system, but not having much luck. How would I use lsof to try to narrow this down?

Pete

[Ty Roberts]

It seems as though my top three the following..

/var --> High 300's
/usr --> High 300's
/ --> Below 100

With Glance and Lsof I am using GLance to sort the process by IO RT then using LSOF on them to see if there are any similarities...

[Ty Roberts]

QUestion.. though it may be a dumb one.. Should we not be more concerned with the Physical IO's rahter than the Logical Ones?? Since the logical ones would be hitting the Cache?

[James R. Ferguson]

Hi Ty:

The whole idea of a cache (buffer cache) is to do I/O in memory as opposed to from a physical device. Therefore, as you have already deduced, the concern should be more with physical rather than logical I/O, first.

Of course, modified buffers must still be written to disk, and to get a logical I/O one must do a least one physical I/O, so in these senses, when you are looking to minimize I/O and I/O contention, you want to consider all kinds.

Regards!

...JRF...

[Stefan Farrelly]

Hi Ty,

Your last output was very helpful. I think now your heavy boot disk usage is a result of your server being very heavily used - application wise. Its normal! your server is just very busy. If all 3 (/, /usr and /var) are being hit then this matches nicely with a normal user profile;

/ is hit hard because of /dev/tty entries as users log in and our and request more data.
/usr is hit because of all the libraries in /usr/lib which applications need.
/var is hit because of /var/sockets, again from user socket connections (requesting more data etc.) and possibly also /var/tmp.

I dont think there is anything much youre going to be able to do about it. It is a feature of having such a large server with such a large number of users and application(s) being heavily used on it.

There are a few things you can do;
1. Replace the boot disk with a faster one. HP make some really quick 15,000 rpm drives nowdadays with big caches on them
2. Move the boot disk off to an external disk array - something with lots of cache. HP make all sorts of disk arrays which could do this.
3. Add in another disk or 2 to VG00 and recreate /usr and /var striped over >1 disk. This should speed up performance a lot also.

[Pete Randall]

Ty,

I think I've narrowed mine down to /dev/log and /INFORMIXTMP (which Informix insists has to be someplace "safe" - I'm not sure / is my favorite choice, though). I'm inclined to believe this is normal behavior for me, too.

Good luck,
Pete

[Martin Johnson]

Pete,

I would not put /INFORMIXTMP in /. / is normally static (i.e. does not grow). You could fill up / with files in /INFORMIXTMP. That would cause you all sorts of problems.


Marty