HP RP 8400 High memory usage and system hangs

[Syed Fahimuddin]

Hi All,

This server is used as production application server for Oracle Applications11i. We have an average of 40 concurrent users when the systems hangs.

We have noticed the following when the system hangs.

1) Memory usage is 100%
2) Disk i/o is 100%

My question is, is the system suppose to hang when the system goes out of memory. If yes, why is it not using SWAP and avoid hanging.

Any help would be greatly appreciated.



Rgds
Najmal

[RAC_1]

The system should not hange when memory or disk i/o reaches 100 %. If memeory reaches 100 %, swapping should start. Do you have glance installed? Check the which processes are causing high mem and disk utilization.

Also what are your settings for dbc_max_pct and dbc_min_pct? By default the dbc_max_pct is set at 50 %. That means half of you RAM will be used as buffer cache.

Also periodically keep on chekcing swapinfo. And when systems hangs do a TOC from GSP, this would give you crash dump, so that exact cause of hange would be known.

What is your RAM and waht is the swap space configured? Alo make sure you have all patches that oracle recomands.

Anil

[Mark Grant]

I imagine it can't. It has probably run out of swap too. It is difficult to decide what to do in these sitations, some OS's hang, some crash and some start killing off processes. None of the solutions to running out of virtual memeory are particularly nice but at least the killing off processes option gives you some kind of chance of recovery (HPUX doesn't do this).

Does the "hang" go away or do you have to re-boot?

I would check the output of "swapinfo -mt" to see what you swap levels are at and check the system log to see if there are any errors there. In all likelyhood, you have some process that is killing your machine.

[Steven E. Protter]

What is the application?

What is id doing?

swapinfo -tam make sure the swap is actually active.

Is the system properly patched?

Perhaps collec some data. See the script I'm attaching.

SEP

[Shaikh Imran]

Hi,
Please Try Vmstat Command

Measuring Memory Usage with vmstat:
vmstat displays a wealth of information;
use the -n option to make it more readable on an 80-column display.
The column to watch most closely is po. If it is not zero, the system is
paging. If the system is paging consistently, you probably need more
RAM.

Regards

[Syed Fahimuddin]

Hi

Thanks for the quick response.

The story started like. This system is running fine for almost more than a year. Recently there were 25 new users added. It started the problem and we did lot of research and changed some kernel parameters. One of them is dbc_max_pct, which is now 6%.
The RAM on this machine is 2 GB.

Using glance we noticed a lot of things. One such thing was the process f60webmx takes on an average 25 - 30 MB RAM. If it is a order management user, it takes anywhere between 40 - 60 MB RAM.

We have changed parameters
vx_ninode from 0 to 8270 and swapmem_on to 0 from 1.

This has resulted in more memory given to the users and saved from Buf Cache and Sys Memory. So far so good . . .

We applied some Forms patchset on advise from Oracle Corp. Did not help.

We concluded that it requires at least 30MB for each user process, so we decided for a RAM upgrade which is currently under way.

During this time, we applied the Dec, 2003 GOLD PACKS on this machine,

After this the symptoms changed. Now it goes down even when there are no more than 34 users. Suddenly the memory utilization goes up to 100% and the Disk utilization goes up to 100% and the system hangs. At this point of time, the Sys memory shown in Glance is 1.65 GB and the user memory goes down to 300MB.

We cannot salvage the situation. The only way is to connect to the Admin console and give a RS (reset) or TC (TOC).

Already send the dumps to HP for analysis.

Any clues or any help.

This is a very grave and serious situation.

Attaching the latest glance screen output, the sar output, the no. of users output

All the experts in the world are requested to help. . . .

- regards

[RAC_1]

What was the reason to change vx_ninode to xxxx, vxfs is dynamic and creates inodes as required. On my system it is 0. Also setting pseudo swap will help on your machine, with that more users will be able start processess. This paramaeter is designed to take advantage of large RAM. In your situation I would not have set it to 1.

Also 2gb, is not enough for oracle, with the prices of RAM oging down, I would think of adding more RAM to machine.

Anil

[Tom Smith_9]

I agree with Anil, 2 GB is is far too low for Oracle Apps. Assuming that this system is also the DB server.

I have an RP8400 that runs Oracle Apps (Apps and DB on 8400, web server on other tier.)
This system supports 300-400 users peak, we have 32 GB of RAM, and we don't have much room to spare.

The rule of thumb I use for the 8400 is 2 GB / Processor. I have 16 processors, so I installed 32 GB RAM.

Hope this helps.

[Dietmar Konermann]

Two thingss in your last post worry me...

1) fork() failed messages in your sar Output. Have you already noticed that? I would assume that nproc needs to be increased urgently.

2) The system memory consumption reported in Glance is really high. Since you seem to have already a call with HP open... a report of the HP support tool 'kmeminfo' would be very helpful.

Best regards...
Dietmar.

[Syed Fahimuddin]

I have already increased the nproc, based on the output.

Also we are awaiting the response from HP.

We have also ordered 4GB RAM, since all the indications are pointing to lack of enough RAM

Thanks to all the guys for their valuable responses and advise.

We will keep this updated with more information when it becomes available