Need help interpreting page outs.

[Francis Noël]

Hello forum people.

I am working with 11.31 on Integrity for the first time and I need help interpreting some page out stats.

The server is an rx3600 and all storage is local disk for the moment.

What I am seeing is high page out rates when disk IO occurs. If an Oracle database is running an export, for example, page outs will skyrocket and the vhand process performs IOs like crazy.

Filecache is set tu use 512-1024 MBytes of RAM.

At first this would seem to suggest that the system needs more RAM but there are gigs of it marked free and swapinfo says the device swap never gets used.

So how would you guys tackle this ? Yet more filecache ? I feel like I'm missing the obvious...

Regards

[Tim Nelson]

vhand going crazy is probably the buffer flushing.

As far as "po" metric is concerned. I cannot say I have see this exact situation but keep in mind, "pi"s are processes starting up, maybe in the case "po"s are processes ending and getting flushed.

This is a streatch of an explaination...

[Patrick Wallek]

How much RAM do you have? How much swap space do you have configured? Do you have the swap_mem_on kernel parameter set to 1?

Please post the output of 'swapinfo -tam'.

There isn't quite enough information yet.

[Francis Noël]

Here comes the info.
See attachement for the swapinfo output.

The system has 24 Gigs of ram installed. Out of those 24 Gigs, 15 are free.

I set up the device swap on a 16 Gig lvol.

pseudo-swap seems enabled according to the swapinfo output but it looks like the swapmem_on tunable was renamed/obsoleted in 11.31 as I cannot find it in kctune's output.

Thanks for the help guys.

[Francis Noël]

Okay

Got some more bits toadd here.

I had the DBA launch another export run and this is what I saw.

At the very start of the procedure the system FileCache got cleared. It then filled up in a linear fashion up to its configured upper limit ( 1 Gig ).
Up to this point there were NO page outs.

Once the cache got filled up the page outs started, in a big way. See the attachement for the numbers.

I'm a junior admin so up to now my thinking has pretty much been "page outs = bad" but what I just saw has me thinking this might just be the way 11.31 handles tings ? Or is something broken ?

[Tim Nelson]

In HPUX page outs are different from deactivations.

Deactivations are processes being "swapped" out to disk.

Page outs are programs that have finished and their memory resources being cleared.

I almost mentioned this in my reply before.

Deactivations are bad. Page outs are not neccessarily bad.

Just like page ins.. If you start up 1000 programs then you will have 1000 page ins. If you terminate 1000 programs you will have 1000 page outs.

[Don Morris_1 · Accepted solution]

swapmem_on was obsoleted in v3, yes. (man 5 swapmem_on, btw... obsoleted tunables will always still have man pages for at least 1 release -- so that should be an easy way to check).

Yes, the page outs are the file cache flushing pages in order to provide new pages for the writes I assume Oracle is making as part of the export (on prior releases this would have been I/Os underneath the buffer cache -- the Unified File Cache in v3 puts this all in one place). vhand does some memory pressure handling for UFC (some is done in-line by the requesting process, etc.) -- hence vhand is active because of the UFC pressure.

I'm assuming you left filecache_max at the default of 50%? If not -- you could consider raising it to give the cache more headroom... although if the DB is bigger than the cache could possibly be, you'd obviously still hit some cache pageouts eventually.

I'm not terribly clear if this is causing general system performance problems [other than vhand showing up as using CPU cycles, of course... but if the cache is under pressure, the cache is under pressure]. If it is -- and the DB export is a sequential write out, you may want to look at the fcache_seqlimit_file and fcache_seqlimit_system tunables [lowering fcache_seqlimit_file would cap the cache used by Oracle and prevent the cache as a whole from being under pressure... however, there will still have to be pages flushed from the cache as Oracle moves through it].

I'd also recommend you be up to date on patches (at least 11.31.0709 if you aren't there already) since there were a few post release tweaks to adjust paging behavior when the file cache was under pressure.

[Francis Noël]

Hello again and thank you so much for your reponses.

This explains nicely why I had never seen so many page outs under 11iv1.

Don : Thank you for taking the time to explain so concisely. Sorry for omitting the detail in my original post but I did NOT observe any significant performance impact. The DB export is in fact being done in record time. I just got startled by so many page outs and seeked an explanation, which I now have.

I think I am almost ready to move on, points will be assigned today, but if anyone could point me towards some documentation on UFC that would be great.

[Duncan Edmonstone]

Francis,

There's a WP on UFC here:

http://h71028.www7.hp.com/ERC/downloads/4AA0-7157ENW.pdf

HTH

Duncan

[Francis Noël]

Throught the answers in this thread I learned that under 11.31 filesystem buffer cache flushes are accounted for under the page out metric. This is the way the new UFC works and under previous OS releases this activity was filed under Buffer Cache I/O's.

Thank you all for the help !