topic Re: poor performance after memory upgrade in Operating System - OpenVMS

poor performance after memory upgrade

Peter Clarke — Thu, 20 May 2004 10:13:48 GMT

Hi Guys,

Upgraded my memory today but hasnt really improved as much as i thought it would.Had 4GB now got 8GB with 4 way interleving,i had hoped that the page faults would decrease significantly but they havent any ideas??
I will run autogen again tomorrow after 24hrs of use and see what happens.

Re: poor performance after memory upgrade

Uwe Zessin — Thu, 20 May 2004 11:09:15 GMT

Peter,
you might have to increase process working set sizes to reduce page faulting. Without that and even with more main memory you might just turn your 'hard' faults (to/from disk) against 'soft' faults (to/from memory). Soft faults are still better than hard faults.

You will still have paging activity:
e.g. there is no special 'image loader' - they are brought into memory through the normal demand paging.

Re: poor performance after memory upgrade

Peter Clarke — Mon, 16 Sep 2024 09:18:50 GMT

How would i know how much to increase the process working sets by??

The average page fault rate is currently 354.62.
And on average 22.50% of these are hard faults.

Reg
Pete

Re: poor performance after memory upgrade

Mobeen_1 — Thu, 20 May 2004 12:01:50 GMT

I would depend on AUTOGEN to tune the params and run it as often as possible before making any changes

regards
Mobeen

Re: poor performance after memory upgrade

Jan van den Ende — Thu, 20 May 2004 12:02:03 GMT

Peter,

your number of PAGEFAULTS almost certainly stays nearly the same after just adding more memory (assuming the same workload).
(What WILL change, is the nature of the pagefaults, and there SHOULD be performance gain.)
The first time any page is needed, ofcourse it will generate a fault, and have to be brought in from disk. If a page LEAVES a working set, initially it will stay in memory, and be added to the free page list, First In, First Out. (modified pages go to the modified page list, and from there may also land on the free page list). VMS DOES keep track of them. After some time, all pages on the free list are pages that have been used before. Now if new pages need to be brought in, the oldest pages of the free list get overwritten with the new data, and only now are they truely gone from memory.
This becomes highly relevant when (the data of) a page that was moved to the free list is needed again. In that case, VMS knows it is still in memory, and simply changes some pointers, and the page is in the requiring working set again!. Just changing some pointers in various page-tables is really very, very much faster than reading the page from disk, and then also adjust an equal amount of pointers!
The first is a 'hard' pagefault, the second a 'soft' one.
Now, if you have more memory, many more pages stay in memory, so much less pagefaults are hard.
This even becomes much more pronounced when an IO cache becomes active (XFC in V7.3-x, VIOC in 7.2-x, third party before that). Very simplified, an IO cache does not only know a processes's pages, but knows which diskblocks are where in memory. Therefore, a block that was already read for a completely unrelated process, or even an already-ended one, might already/still be in memory, and no disk IO is needed. Sadly for statistics, for VMS this still represents a (very fast) hard fault, you need the reporting tool of the cache to differentiate those.
If you want to monitor the different fault types, in the Pagefault subscreen of MONITOR SYSTEM a "|" (pipe) separates hard (left of it) from soft (right) faults.
Monitor PAGES shows more detail:
The whole second block is various types of soft faults.
Special case of hard fault: if a process has its maximum working set and needs more, pages 'fall out' of the working set. If they have been modified, they move to the ModifiedPageList. If THAT is filled over its treshold, a large chunck get written to the pagefile, and if it is needed again, that ALSO is a hard fault. (well, the page might still be at the free list, but you get the idea) So, as long as there is enough memory, it is advantageous to have a large ModifiedPageList.

Since you HAVE extra memory, it MIGHT stll be well worth trying to decrase pagefaults however.
Although the overhead by soft faults is MUCH less than by hard faults, it _IS_ overhead.
So, if you can identify processes that DO incur many soft faults, it will be worth the trouble of assigning them larger working sets.
Quick and easy: just give everyone more WSQUOTA & WSEXTEND (check WSMAX as well, because that sets a hard upper limit).
If you suspect a process of excessive soft faulting: SHO PROC/ID=../CONT and watch the working set. If that stays under WSEXTENT, there is no gain. But if it hovers near WSEXTENT, and the pagefaults are running, there may be performance gain available by raising WSEXTENT.

hth.

Jan

Re: poor performance after memory upgrade

John Eerenberg — Thu, 20 May 2004 23:21:47 GMT

Peter,

> The average page fault rate is currently 354.62.
> And on average 22.50% of these are hard faults.

What concerns me greatly is a hard fault rate of 22.50% or roughly 79/sec. This is extremely high for almost any Alpha environment.

When tuning systems, I look for a hard fault rate over 10/sec (which may or may not be a problem).

After doing what has already been suggested, do you still see a high page fault rate?

If so, hard faults will become the focus for investigating peformance. If you do a
$ monitor page
and run it for a few minutes, this would be a good start toward determining what action needs to be taken.

For the moment, it doesn't really matter if your soft faults are over 1,000/sec. If your hard faults are high, there are techniques to remedy this. Some of them are involved, some are quite easy, just depends.

If your system still isn't were you think it should be, let us know what the above $ monitor page looks like (a $ show memory wouldn't hurt either).

john

Re: poor performance after memory upgrade

Peter Clarke — Fri, 21 May 2004 03:18:53 GMT

Hi John,

I have attached the output from 'show mem' i have also attached an excel file of the alpha statistics that i have been monitoring over the previous week or so.
In reply to the previous question it seems to be all processes that are page faulting rather than just a few.Should i increase all process working sets?
Hope you can help.

Thanks

Peter

Re: poor performance after memory upgrade

Peter Clarke — Fri, 21 May 2004 03:19:40 GMT

Here is 'show mem'

Re: poor performance after memory upgrade

John Eerenberg — Fri, 21 May 2004 09:51:49 GMT

Peter,

I'm on the road right now so I don't have excel at this time. However, the show mem is interesting. You seem to have plently of memory on the freelist. The modified list might be on the small size.

Out of several posibilities, your modified page list may be too small (check page write IO's). Another posibility is that you have a lot of image activationss. Do you have a lot of people (or processes) logging in/out? Or running .COM over and over?

Sorry I can't provide more. Maybe someone else can chime in . . .

john

Re: poor performance after memory upgrade

Jan van den Ende — Fri, 21 May 2004 11:52:33 GMT

Sorry John,

there is NO problem there!

DISK$ALPHASYS:[SYS0.SYSEXE]PAGEFILE.SYS
254 524792 524792

Look like wrapping unpleasantly, but anyway:
free pagefile pages = total pagefile pages,
meaning: NO modified pages in pagefile, ie, NO problems there!

(but Peter, I WAS planning to ask for this, so that's covered as well now)

Jan

Re: poor performance after memory upgrade

Cass Witkowski — Fri, 21 May 2004 11:56:16 GMT

Process creation will generate hard page faults. Do you create a lot of processes?

Also image activation will cause hard page faults. Are the images that get run frequently installed

Re: poor performance after memory upgrade

Jan van den Ende — Fri, 21 May 2004 12:00:52 GMT

Peter,

One more thing.

I noted

Vols in Full XFC mode 0 Vols in VIOC Compatible mode 6
/snip>

This DOES mean you are on a 7.3(-x).
The VIOC compatibility mode however is mainly for use during rolling upgrade (and you don't want XFC if you are in 7.3 (no -), without the recent patches!

I'm at home now, no docs readily at hand, but (at least!) the Installation/Upgrade manuals has a chapter on moving from VIOC to XFC.
You will want to move to full XFC mode.

Well, I better have another strong look, the longer one tries, the more one finds...

Jan

Re: poor performance after memory upgrade

Antoniov. — Sat, 22 May 2004 04:03:32 GMT

Hi Peter,
lookin into yours attachments, I guess:
- your system makes a lot of I/O (due DBMS?);
- I/O have some hit in specific days (for example May, 7th and May, 10th morning);
- you have 4 GB of cache (50% of RAM) to increase I/O performance;
- your page file is free;

I think, your bottleneck is not page fault but I/O; if you have a DBMS you could increase working set of DBMS process; monit this process that's critical for you.

@Antoniov

Re: poor performance after memory upgrade

Hein van den Heuvel — Sat, 22 May 2004 12:28:30 GMT

I think Cass has the most pertinent reply.
The most likely cause for the hard fault rate is Image activation.

Do you have the Oracle images installed, and installed properly to maximize sharing?
What lives on DKA1? Is that $ORACLE_HOME/bin?

What you also may want to do it correlate the known Oracle IOs (statspack or some such)with the measured IO to help explain parts.

I think that the most interesting lines from the Show memory are: "Extended File Cache (Time of last reset: 20-MAY-2004 13:37:53.05)
Allocated (GBytes) 3.46 Maximum size (GBytes) 4.00

So the system decided (probably rightly so), that it had nothing better to do with it's free memory then to give all of it to the XFC cache. For Oracle heavy applications, that is often a waste (double buffering). You are better of giving it to the SGA (which you already did, and nicely so in the reservered memory, but maybe you can now give it more?

Back to the 'frequent image activations'. Coudl that be the case? Can you influence that? Use some parameter in the application to keep connections (to Oracle) open? Use MTS to keep a pool of slaves ready to be used?

Get yourself a 'hot file' package of some sore, or dive into the XFC details showing files in details to understand where (pagefault) IOs are being resolved. SYS$SYSTEM:SET.EXE? LOGINOUT? ORACLE.EXE?....

Finally I would recommend you check the (SY)LOGIN.COM and such that get involved when an Oracle slave is activate. The steps leading to an Oracle slave activation should be made very lean and mean. Junk any 'set proc/name' no terminal testsing/setting, clean out that path... Oracle adds enough steps already!

Hope this helps,
Hein.

Re: poor performance after memory upgrade

Peter Clarke — Mon, 24 May 2004 03:37:00 GMT

Thanks i will try a few of these ideas.

Just a couple of questions:
How can i take some memory away from the xfc cache and give it to the SGA.

Have atached the agen$params.report file it has a few warnings in are these anything to worry about??

Thanks

Peter

Re: poor performance after memory upgrade

Peter Clarke — Mon, 24 May 2004 03:37:59 GMT

Here is setparams file aswell just incase anything is set wrong....

Re: poor performance after memory upgrade

Brian Reiter — Mon, 24 May 2004 04:03:06 GMT

Peter,

Not sure how to allocate memory to Oracle, but you can limit the memory allocation of XFC by setting VCC_MAX_CACHE via SYSGEN. This parameter is dynamic so should kick in without a reboot. The default is half the physical memory.

As previously mentioned try and install frequently used images.

regards

Brian

Re: poor performance after memory upgrade

Jan van den Ende — Mon, 24 May 2004 04:09:01 GMT

Peter:

Autogen report:

-There are two parameter name error reports, coming from MODPARAMS. They are simple typos, but correct them anyway.
-A lot of parameters have multiple entries.
AUTOGEN overwrites any previous value with the latest (they are just temporary symbols), but for human clarity it is much more handy to have those weeded out a bit.

Apart from that, I noted no immediately obvisous things That's to be expected with uptime < 24 hours; this may (but needs not) change upon longer uptime.

Taking memory from XFC to give to SGA:
it's the other way around. You GIVE memory to SGA, (and FREELIST, and MODIFIED LIST, and whatever..) and XFC is ALLOWED to use the remainder.
I'm not familiar with the latest ORACLEs, but way back when the systemmanager (in cooperation with the DBA) defined the SGASIZE in ORA.INI.
(If that has changed, anyone on current OACLE please correct me; but there will undoubtably be a way to tell ORACLE what size SGA to initialize).

hth

Jan

Re: poor performance after memory upgrade

Antoniov. — Mon, 24 May 2004 04:12:50 GMT

Hi Peter,
I preliminarly looked into yours attachment and it seems there is no significative update because your system worked too little time for autogen.
Look about SCSSYSTEMID and SCSNODE: you could find this value more times in your MODPARAMS.DAT; however it you node is "EURAXP" and DecNet Address is 1.405
GBLPAGES and GBLSECTIONS will be increased by autogen; I think you will not have some benefit from this.
May be better result from MIN_VIRTULAPAGECNT and MIN_WSMAX values.
I think it's better repeat autogen after 2/3 days of working.

Antonio Vigliotti

Re: poor performance after memory upgrade

Peter Clarke — Mon, 24 May 2004 04:27:00 GMT

You metion about installing frequently used image's,how can i find out which ones are frequently used and how would i install them??
Sorry if i sound a bit dim but this is all relatively new to me...

Reg

Peter