How to retrieve InAct physical memory in tru64

[Danny Ramos_3]

Hi Guys,

I have a GS320 with 25GB memory 14CPU and 75GB swap space. Just recently, we experienced an abnormal shutdown on the database (oracle). The log shows that the system ran out of swap space, resulting to randomly kill processes. Unfortunately, oracle's pmon process was killed resulting for the database to go down.

I've been monitoring the system for 3 days now after an unscheduled system reboot. As we startup the database and begin processing, i've seen fron "COLLECT" command that more than 50% of the physical memory has been immediately utilized. A day after (2nd day) i've noticed that more than 90% of the MEMORY is utilized. Today is the third day and i only have few MBs of mem free. This results to the utilization of swapspace that worries me to trigger another untoward incidence of abnormal shutdown.

Could anyone tell me how to get rid of this phenomenon. I woud like to inquire as well if it's possible for me to regain possesion of the inactive memory devices as show from my collect output below:

# MEMORY STATISTICS
#(<------------ MegaBytes -------------> <--------- Pages/sec -------->)
# Free Swap Act InAc Wire UBC PI PO Zer Re COW SW HIT PP ALL
366 68067 6600 13866 882 2394 22068 0 10238 0 2377 0 25114 0 0

Thanks in advance!
Dan

[Mark Poeschl_2]

Having little or no free memory is not necessarily a bad thing on Tru64. It just means that the memory allocation algorithm is doing its job. We commonly run with under 1% 'free' memory and have no problem at all.

Your output indicates that there's very little 'hard' (to disk) paging, no processes are being swapped, and 68 out of your 75GB swap space is still free. Is the swap space utilization trending upwards over time? Do you have archived collect data from shortly before the crash for comparison?

[Hein van den Heuvel]

As Mark says, 99% memory in use can be OK. The PI/PO rate will tell you whether it really is. The default UBC setting is to allow it to fill up all memory, and just walk away form buffered file data when vm space is needed.

With a swapspace available at 3 times the physical memory I would recommend EAGER (default) swap mode, not lazy. This will prevent further allocation when swap/memory space is low and will not randomly kill processes (as is the risk with lazy mode).

What is your Oracle SGA size? That could readily explain 50% memory used upon startup.
Have you define GH-chunks? and are they indeed being use?

This 25GB is an odd number. Do you really have an odd number of memory module? Not a good idea for performance (reduced interleaving). Do you have a broken memory module?

This GS320 does it have 4 rads (4+4+4+2 cpus) or more? How is the memory distributed? (8 + 8 + 8 + 1 ?! )

hth,
Hein.

[Joerg Schulenburg]

Sounds like my problem ;). Buggy or badly designed VM managment for RADs in the Tru64 kernel.
And always the same answers from HP experts...
Why a run out of swap is better in eager mode , Hein? Because the process which needs memory
at the fullswap state is killed, get an error
or hangs? Nothing will be better than in lazy mode except the lazy mode is badly implemented and causes the growing swap itself.
As Hein wrote you should concern about RADs.
Do a vmstat -R 2 and look at number of free pages. I also admin a GS320 like machine in
lazy mode, only to see when something
is swapped. If you switch to eager, as Hein
suggests, please tell me, what the outcome is. I am very interested.
If one of the RADs has less then 1000 pages free whereas others have enough free and swap
grows you have exact the same problem as I have and the vm_overflow patch together with PK4 and the ubc-patch can help you partly.

[Joerg Schulenburg]

Sorry, I was to rashly with my statements.
Indeed 99% memory using can be ok, but probably
it causes swapping and slow down of your oracle database. And that can be a reason that your application ask for more memory so other memory is swapped too. Do you know which process consumes all the virtual memory?
Can you also show more than one line of collect-output (better would be vmstat -R)?
Is it possible that UBC pages are swapped out?
can you show also output of /sbin/sysconfig -q vm | grep ubc ? Try to reduce ubc_maxpercent
if you observe growing swap again.
Good luck.

[Joerg Schulenburg]

Concerning the title of your thread:
"How to retrieve InAct physical memory in tru64?"
Killing the process which owns the inactiv pages is the only way, as far as I know.
On a normal UNIX system I would expect
that accessing an inactiv page should retrieve this page to active ones if free pages are available. Or am I wrong?

[Hein van den Heuvel]

>> Why a run out of swap is better in eager mode , Hein?

Because the effects will be predictable.
New allocations will fail, existing work can go on.

The alternative is what Dan describes in this topic. When running out in lazy mode, the OS kills a random process.
Dirty Harry / Clint Eastwood: "You've got to ask yourself one question...Do I feel lucky? Well, do ya punk?"

I run all my systems in Lazy mode, but I work in a lab with test systems, and I believe I know how to estimate and control physical memory usage.

If I was responsible for a million dollar setup like Dan's GS320, then I would invest the extra dollars for a little more disk space.

This also avoids having to fight with Oracle and Sap support who normally recommend 3x physical whether you need it or not. It's just not worth the fight.

Admittedly it would be rahter painfull if you actually have to do significant swapping on a multy-GB memory systems. Even at say 50mb/sec, you talk about a minute activity to make a serious dent in free pages.

Hein.

[Joerg Schulenburg]

>>>> Why a run out of swap is better in eager mode , Hein?

>>Because the effects will be predictable.
>>New allocations will fail, existing work can go on.

Fine. If I understand Danny right, he is worrying about a swapping machine. So why not consequently set the swapspace to 95% of physical memory and set to eager mode. Than swapping is impossible, effects are predictable, and Danny can relax.

Danny, feel free to follow the suggestions of Mark & Hein. Tell us if it was helpful. I am curious about it.

I found it better if you give more information (vmstat -R 3600;
ps -o "pid,pcpu,vsz,rss,minflt,majflt,cmd" -a (every hour), vmubc -tty -memory -swap -pagefault -interval 3600 etc.) for analyzing your problem.