Mem 100% full

[Pradep]

Greetings,
Could you please help me find the issue with hpux11iv3 over PA-RISC. / OracleR12

B3692A Glance C.04.70.000 15:09:22 uasng076 9000/800 Current Avg High
-------------------------------------------------------------------------------------
CPU Util SSUU | 7% 5% 7%
Disk Util FV V | 6% 5% 9%
Mem Util S SU| 99% 99% 99%
Swap Util U UR R | 37% 37% 37%
-------------------------------------------------------------------------------------
MEMORY REPORT Users= 5
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 6096 12798 1172.3 688.0 1172.3
Page In 214 530 41.1 28.4 328.2
Page Out 0 0 0.0 0.0 0.0
KB Paged In 856kb 2.1mb 164.6 113.9 852.0
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 226 576 43.4 30.9 52.3
VM Writes 0 0 0.0 0.0 0.0
Total VM : 4.4gb Sys Mem : 15.9tb User Mem: 70.3gb Phys Mem : 8.0gb
Active VM: 1.8gb Buf Cache: 6mb Free Mem: 65mb FileCache: 2.2gb
MemFS Blk Cnt: 0 MemFS Swp Cnt: 0


SWAP SPACE Users= 5

Swap Device Type Avail Used Priority
--------------------------------------------------------------------------------
/dev/vg00/lvol2 device 16.0gb 709mb 1
/dev/vg00/lvol10 device 1.1gb 134mb 1
pseudo-swap memory 7.6gb 5.4gb -1



# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16384 709 15675 4% 0 - 1 /dev/vg00/lvol2
dev 1136 134 1002 12% 0 - 1 /dev/vg00/lvol10
reserve - 2949 -2949
memory 7790 5495 2295 71%
total 25310 9287 16023 37% - 0 -

[Pradep]

Oops. Oracle DB version is 11g.

[Dennis Handly]

>Mem 100% full

This is a good thing, what's the problem? Do you want to add more load on the system?

>Page Out 0 0 0.0 0.0 0.0

No page outs.

>memory 7790 5495 2295 71%

At this time, you have 2 Gb free and lots of device swap.

[Pradep]

we had a issue reported by DBA that commands were kind of hung. then we found that Glance is showing 100% Mem.
can you please help me understand why glance shows mem 100%, even though Psuedo-swap which is part of physical memory is 25% free ? what defines the size of pseudo-swap.any kernel parameter ?

how do i interpret Total VM : 4.4gb Sys Mem : 15.9tb User Mem: 70.3gb. Mem is 15.9TB ? and user Mem 70gb ?

Should i decrease size of FileCache: from 2.2gb to 1GB..to create extra available space in Memory ?

[Dennis Handly]

>we had a issue reported by DBA that commands were kind of hung. then we found that Glance is showing 100% Mem.

Did you collect this info when it was still hung?

>what defines the size of pseudo-swap, any kernel parameter?

The size of RAM times a percent. Search for Don Morris' comments on it.

>Should I decrease size of FileCache: from 2.2gb to 1GB to create extra available space in Memory?

That would be helpful if you need it for your SGA.

[Pradep]

Thanks Dennis.
Can you tell me why glance shows mem 100%.
Whats the meaning of Sys Mem : 15.9tb User Mem: 70.3gb
They are far above total swap on server.

[Dennis Handly]

>Can you tell me why glance shows mem 100%.

It may be related to your next inconsistency?

>What's the meaning of Sys Mem: 15.9tb User Mem: 70.3gb? They are far above total swap on server.

Right they do seem incorrect. Have you contacted the Response Center about the problem? Do you have the latest Glance?

[chris huys_4]

Hi,

> Mb Mb Mb PCT START/ Mb
> TYPE AVAIL USED FREE USED LIMIT RESERVE PRI
> NAME
> dev 16384 709 15675 4% 0 - 1 /dev/vg00/lvol2
> dev 1136 134 1002 12% 0 - 1 /dev/vg00/lvol10
The "larger then 0% USED percentage output" for the device swapspace, could be an indication that at one time, the system was actually "using" device swap space as "extra memory", which would result in a not very responsive system..

Check the SGA size of the different databases on the system and see if they are not to "close" to using all of the 8gb of physical memory in the system.

Run also a "kmeminfo;kmeminfo -user;crashinfo -continue" as a root cronjob every 5 minutes and monitor the memory usage.

Greetz,
Chris

[B. Hulst]

How much memory is allocated to Oracle?
What does it tell at the startup in the alert_.log?

You could have too little SGA space or wrong setup of the shared pool,etc memory configuration.

And what is the max.number of Oracle connections possible and how many do you have on average?

[V.P]

Hi Pradep,

Please check whether the swapmemory is on or not?

kctune|grep -i swapmem_on

This should be on (1)

[VK2COT]

Hello,

a) Kernel parameter swapmem_on does not
exist on HP-UX 11.31. Pseudoswap is
enabled by default.

b) Having two paging devices on the same
physical disk(s) is a very bad idea. You
have two of them in volume group vg00 and
most probably they are on the same disk(s).

Please review it and remove one of them.

Your design of paging devices breaks almost
all best practices:

a) One swap device per physical disk;
b) Swap devices should be of preferably of
same size;
c) Spread swap devices across fast disks;
and so on.

c) In regards to 100% RAM utilisation,
others already gave you some good pointers,
starting with Unified File Cache tuning.

Cheers,

VK2COT

[Pradep]

Thanks to all.
I had logged case with HP Response center.
following Params were tuned to resolve it.
because kmeminfo showed like this.

Physical memory = 2096640 8.0g 100%
Free memory = 40007 156.3m 2%
User processes = 785502 3.0g 37% details with -user
Detached SHMEM = 1 4.0k 0% details with -shmem
System = 1268660 4.8g 61%
Kernel = 732732 2.8g 35% kernel text and data
File Cache = 535928 2.0g 26% details with -ufc

#kctune filecache_max=8%
#kctune filecache_min=5%
#kctune vxfs_ifree_timelag=-1
#kctune vx_ninode=40000

SGA is 1GB.

[SoorajCleris]

hmmm

in 11.31 vx_ninode is always suspected when there is a memory issue.

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c01527093

Regards,
Sooraj

[Pradep]

reply above