poor performance after memory upgrade

[Peter Clarke]

Hi Guys,

Upgraded my memory today but hasnt really improved as much as i thought it would.Had 4GB now got 8GB with 4 way interleving,i had hoped that the page faults would decrease significantly but they havent any ideas??
I will run autogen again tomorrow after 24hrs of use and see what happens.

[Uwe Zessin]

Peter,
you might have to increase process working set sizes to reduce page faulting. Without that and even with more main memory you might just turn your 'hard' faults (to/from disk) against 'soft' faults (to/from memory). Soft faults are still better than hard faults.

You will still have paging activity:
e.g. there is no special 'image loader' - they are brought into memory through the normal demand paging.

[Peter Clarke]

How would i know how much to increase the process working sets by??

The average page fault rate is currently 354.62.
And on average 22.50% of these are hard faults.

Reg
Pete

[Mobeen_1]

I would depend on AUTOGEN to tune the params and run it as often as possible before making any changes

regards
Mobeen

[Jan van den Ende]

Peter,

your number of PAGEFAULTS almost certainly stays nearly the same after just adding more memory (assuming the same workload).
(What WILL change, is the nature of the pagefaults, and there SHOULD be performance gain.)
The first time any page is needed, ofcourse it will generate a fault, and have to be brought in from disk. If a page LEAVES a working set, initially it will stay in memory, and be added to the free page list, First In, First Out. (modified pages go to the modified page list, and from there may also land on the free page list). VMS DOES keep track of them. After some time, all pages on the free list are pages that have been used before. Now if new pages need to be brought in, the oldest pages of the free list get overwritten with the new data, and only now are they truely gone from memory.
This becomes highly relevant when (the data of) a page that was moved to the free list is needed again. In that case, VMS knows it is still in memory, and simply changes some pointers, and the page is in the requiring working set again!. Just changing some pointers in various page-tables is really very, very much faster than reading the page from disk, and then also adjust an equal amount of pointers!
The first is a 'hard' pagefault, the second a 'soft' one.
Now, if you have more memory, many more pages stay in memory, so much less pagefaults are hard.
This even becomes much more pronounced when an IO cache becomes active (XFC in V7.3-x, VIOC in 7.2-x, third party before that). Very simplified, an IO cache does not only know a processes's pages, but knows which diskblocks are where in memory. Therefore, a block that was already read for a completely unrelated process, or even an already-ended one, might already/still be in memory, and no disk IO is needed. Sadly for statistics, for VMS this still represents a (very fast) hard fault, you need the reporting tool of the cache to differentiate those.
If you want to monitor the different fault types, in the Pagefault subscreen of MONITOR SYSTEM a "|" (pipe) separates hard (left of it) from soft (right) faults.
Monitor PAGES shows more detail:
The whole second block is various types of soft faults.
Special case of hard fault: if a process has its maximum working set and needs more, pages 'fall out' of the working set. If they have been modified, they move to the ModifiedPageList. If THAT is filled over its treshold, a large chunck get written to the pagefile, and if it is needed again, that ALSO is a hard fault. (well, the page might still be at the free list, but you get the idea) So, as long as there is enough memory, it is advantageous to have a large ModifiedPageList.

Since you HAVE extra memory, it MIGHT stll be well worth trying to decrase pagefaults however.
Although the overhead by soft faults is MUCH less than by hard faults, it _IS_ overhead.
So, if you can identify processes that DO incur many soft faults, it will be worth the trouble of assigning them larger working sets.
Quick and easy: just give everyone more WSQUOTA & WSEXTEND (check WSMAX as well, because that sets a hard upper limit).
If you suspect a process of excessive soft faulting: SHO PROC/ID=../CONT and watch the working set. If that stays under WSEXTENT, there is no gain. But if it hovers near WSEXTENT, and the pagefaults are running, there may be performance gain available by raising WSEXTENT.


hth.


Jan

[John Eerenberg]

Peter,

> The average page fault rate is currently 354.62.
> And on average 22.50% of these are hard faults.

What concerns me greatly is a hard fault rate of 22.50% or roughly 79/sec. This is extremely high for almost any Alpha environment.

When tuning systems, I look for a hard fault rate over 10/sec (which may or may not be a problem).

After doing what has already been suggested, do you still see a high page fault rate?

If so, hard faults will become the focus for investigating peformance. If you do a
$ monitor page
and run it for a few minutes, this would be a good start toward determining what action needs to be taken.

For the moment, it doesn't really matter if your soft faults are over 1,000/sec. If your hard faults are high, there are techniques to remedy this. Some of them are involved, some are quite easy, just depends.

If your system still isn't were you think it should be, let us know what the above $ monitor page looks like (a $ show memory wouldn't hurt either).

john

[Peter Clarke]

Hi John,

I have attached the output from 'show mem' i have also attached an excel file of the alpha statistics that i have been monitoring over the previous week or so.
In reply to the previous question it seems to be all processes that are page faulting rather than just a few.Should i increase all process working sets?
Hope you can help.

Thanks

Peter

[Peter Clarke]

Here is 'show mem'

[John Eerenberg · Accepted solution]

Peter,

I'm on the road right now so I don't have excel at this time. However, the show mem is interesting. You seem to have plently of memory on the freelist. The modified list might be on the small size.

Out of several posibilities, your modified page list may be too small (check page write IO's). Another posibility is that you have a lot of image activationss. Do you have a lot of people (or processes) logging in/out? Or running .COM over and over?

Sorry I can't provide more. Maybe someone else can chime in . . .

john

[Jan van den Ende]

Sorry John,


there is NO problem there!



DISK$ALPHASYS:[SYS0.SYSEXE]PAGEFILE.SYS
254 524792 524792



Look like wrapping unpleasantly, but anyway:
free pagefile pages = total pagefile pages,
meaning: NO modified pages in pagefile, ie, NO problems there!

(but Peter, I WAS planning to ask for this, so that's covered as well now)


Jan

[Cass Witkowski]

Process creation will generate hard page faults. Do you create a lot of processes?

Also image activation will cause hard page faults. Are the images that get run frequently installed

[Jan van den Ende]

Peter,

One more thing.

I noted


Vols in Full XFC mode 0 Vols in VIOC Compatible mode 6
/snip>

This DOES mean you are on a 7.3(-x).
The VIOC compatibility mode however is mainly for use during rolling upgrade (and you don't want XFC if you are in 7.3 (no -), without the recent patches!

I'm at home now, no docs readily at hand, but (at least!) the Installation/Upgrade manuals has a chapter on moving from VIOC to XFC.
You will want to move to full XFC mode.


Well, I better have another strong look, the longer one tries, the more one finds...

Jan

[Antoniov.]

Hi Peter,
lookin into yours attachments, I guess:
- your system makes a lot of I/O (due DBMS?);
- I/O have some hit in specific days (for example May, 7th and May, 10th morning);
- you have 4 GB of cache (50% of RAM) to increase I/O performance;
- your page file is free;

I think, your bottleneck is not page fault but I/O; if you have a DBMS you could increase working set of DBMS process; monit this process that's critical for you.

@Antoniov

[Hein van den Heuvel]

I think Cass has the most pertinent reply.
The most likely cause for the hard fault rate is Image activation.

Do you have the Oracle images installed, and installed properly to maximize sharing?
What lives on DKA1? Is that $ORACLE_HOME/bin?

What you also may want to do it correlate the known Oracle IOs (statspack or some such)with the measured IO to help explain parts.

I think that the most interesting lines from the Show memory are: "Extended File Cache (Time of last reset: 20-MAY-2004 13:37:53.05)
Allocated (GBytes) 3.46 Maximum size (GBytes) 4.00

So the system decided (probably rightly so), that it had nothing better to do with it's free memory then to give all of it to the XFC cache. For Oracle heavy applications, that is often a waste (double buffering). You are better of giving it to the SGA (which you already did, and nicely so in the reservered memory, but maybe you can now give it more?


Back to the 'frequent image activations'. Coudl that be the case? Can you influence that? Use some parameter in the application to keep connections (to Oracle) open? Use MTS to keep a pool of slaves ready to be used?

Get yourself a 'hot file' package of some sore, or dive into the XFC details showing files in details to understand where (pagefault) IOs are being resolved. SYS$SYSTEM:SET.EXE? LOGINOUT? ORACLE.EXE?....

Finally I would recommend you check the (SY)LOGIN.COM and such that get involved when an Oracle slave is activate. The steps leading to an Oracle slave activation should be made very lean and mean. Junk any 'set proc/name' no terminal testsing/setting, clean out that path... Oracle adds enough steps already!


Hope this helps,
Hein.

[Peter Clarke]

Thanks i will try a few of these ideas.

Just a couple of questions:
How can i take some memory away from the xfc cache and give it to the SGA.

Have atached the agen$params.report file it has a few warnings in are these anything to worry about??

Thanks

Peter

[Peter Clarke]

Here is setparams file aswell just incase anything is set wrong....

[Brian Reiter]

Peter,

Not sure how to allocate memory to Oracle, but you can limit the memory allocation of XFC by setting VCC_MAX_CACHE via SYSGEN. This parameter is dynamic so should kick in without a reboot. The default is half the physical memory.

As previously mentioned try and install frequently used images.

regards

Brian

[Jan van den Ende]

Peter:

Autogen report:

-There are two parameter name error reports, coming from MODPARAMS. They are simple typos, but correct them anyway.
-A lot of parameters have multiple entries.
AUTOGEN overwrites any previous value with the latest (they are just temporary symbols), but for human clarity it is much more handy to have those weeded out a bit.

Apart from that, I noted no immediately obvisous things That's to be expected with uptime < 24 hours; this may (but needs not) change upon longer uptime.

Taking memory from XFC to give to SGA:
it's the other way around. You GIVE memory to SGA, (and FREELIST, and MODIFIED LIST, and whatever..) and XFC is ALLOWED to use the remainder.
I'm not familiar with the latest ORACLEs, but way back when the systemmanager (in cooperation with the DBA) defined the SGASIZE in ORA.INI.
(If that has changed, anyone on current OACLE please correct me; but there will undoubtably be a way to tell ORACLE what size SGA to initialize).

hth


Jan

[Antoniov.]

Hi Peter,
I preliminarly looked into yours attachment and it seems there is no significative update because your system worked too little time for autogen.
Look about SCSSYSTEMID and SCSNODE: you could find this value more times in your MODPARAMS.DAT; however it you node is "EURAXP" and DecNet Address is 1.405
GBLPAGES and GBLSECTIONS will be increased by autogen; I think you will not have some benefit from this.
May be better result from MIN_VIRTULAPAGECNT and MIN_WSMAX values.
I think it's better repeat autogen after 2/3 days of working.

Antonio Vigliotti

[Peter Clarke]

You metion about installing frequently used image's,how can i find out which ones are frequently used and how would i install them??
Sorry if i sound a bit dim but this is all relatively new to me...

Reg

Peter

[Antoniov.]

Installed images are shared library (or DLL in windows terms) used by software; when they have installed, system load then quicky and can reduce PF.
What image you can install is depending by application: reading Oracle documentation you can find somthink about this.
To view installed images type:
$ INSTALL LIST
I think you'll receive a very long list of installed images.

Antonio Vigliotti

[Peter Clarke]

Yes i get a long list of images.Are these the ones that are installed??
How can i find out which ones are being accessed frequently but are not installed so that i could install these ones??

Pete

[Brian Reiter]

Peter,

You would have to accounting. Assuming its running (and looking after the right info) just enter

ACCOUNT/SIN=TODAY

Which, assuming its enabled will give you a list of image activations etc.

To enable it:
$SET ACCOUNTING /ENABLE=(PROCESS,BATCH,INTERACTIVE,IMAGE)

to verify whether its running and what for

$SHOW ACCOUNT

[Peter Clarke]

Ok Thanks Brian,I now have this information so how would i install the most frequent images from this accounting file??

Is there anything to take into consideration when doing this??
Diskspace i quess? anything else?

[Peter Clarke]

Attached is the output from the Accounting utility.
Any suggestions on which images to install and how to install them properly??