Memory Issues

[Mark Parsons]

Hi,

We have a number of oracle databases on our rp5470 running hp11.11 and they are displaying the following memory issues (using Big Brother and/or Scom):

Database Link = DEVTROND
Problem Type = 10g mgmt alert
User= Object= memUsedPct (Target is tora03) Memory Utilization is 97%

There are six databases that display a similar message as above on the system as follows:

/dev/vgTSTROUND/lvTSTROUND
146739200 101437168 44948176 69% /oracle/oradata/TSTROUND
/dev/vgDB/lvTSTPS89
30720000 24916548 5712812 81% /oracle/oradata/TSTPS89
/dev/vgDB/lvDMOPSOFT
11776000 11359448 403574 97% /oracle/oradata/DMOPS89
/dev/vgTROUND/lvDEVTROND
21504000 9583096 11734704 45% /oracle/oradata/DEVTROND
/dev/vgDB/lvDEVPS89
29696000 25432880 4196548 86% /oracle/oradata/DEVPS89
/dev/vgDB/lvDEVDSL 30720000 25508984 5129600 83% /oracle/oradata/DEVDSL

There are 4 cpu's and the memory details are:

Physical Memory: 6149.5 MB ³³ ³
³³ ³³Real Memory: ³³ ³
³³ ³³ Active: 3588366.8 KB ³³ ³
³³ ³³ Total: 4773749.5 KB ³³ ³
³³ ³³Virtual Memory: ³³ ³
³³ ³³ Active: 6633042.0 KB ³³ ³
³³ ³³ Total: 8150093.5 KB ³³ ³
³³ ³³Free Memory Pages: 46714 at 4 KB/page ³³ ³
³³ ³³Swap Space: ³³ ³
³³ ³³ Avail: 10240 MB ³³ ³
³³ ³³ Used: 5942 MB

Swap space is as follows:

/dev/vg00/lvol2 Device 1 1695 of 2048 now/on boot ^ ³
³³ /dev/vg00/lvol9 Device 1 7838 of 8192 now/on boot

Any ideas where I should look to fix the memory utilization?

Thanks - Mark P.

[Steven E. Protter]

Shalom Mark,

First step is to review with the DBA's how much memory is allocated to the Oracle SGA. The Oracle SGA grabs memory at database start time and may or may not be using all it is asking for.

Commonly Oracle DBA's ask for and get a lot more than they need. Some analysis of oracle memory use by the DBA's will be required.

Also, Oracle has released a number of bug fixes involving various versions of the database server using too much memory when reports and queries are run. That is an area of investigation as well.

For further analysis: http://hpux.ws/system.perf.sh

SEP

[Jean-Luc Oudart]

Hi

just wondering if you have a baseline (resource, perfomance) before you got such messages.
Resource usage information should be collected (eg statspack for the oracle databases) and memory allocated to the instances.

do you use ASMM (automatic shared memory management) ?

see also this thread.http://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=1425480

Regards
Jean-Luc

[John Guster]

please post the output of commands:
1. swapinf -mat
2. ipcs -mob

[Dennis Handly]

>John: swapinf -mat

Fixing your typo: swapinfo -tam

[Mark Parsons]

Hi,

1) Here is the response to a swapinfo -tam:

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2048 426 1622 21% 0 - 1 /dev/vg00/lvol2
dev 8192 422 7770 5% 0 - 1 /dev/vg00/lvol9
reserve - 5224 -5224
memory 4837 834 4003 17%
total 15077 6906 8171 46% - 0 -

2) Here is the response to ipcs -mob:

IPC status from /dev/kmem as of Thu Jun 10 11:54:40 2010
T ID KEY MODE OWNER GROUP NATTCH SEGSZ
Shared Memory:
m 0 0x411c0846 --rw-rw-rw- root root 0 348
m 1 0x4e0c0002 --rw-rw-rw- root root 1 61760
m 2 0x41202011 --rw-rw-rw- root root 1 8192
m 3 0x411c17bd --rw------- root root 0 4096
m 31236 0xd8013700 --rw-r----- oracle dba 47 616570880
m 5 0x0c6629c9 --rw-r----- root root 2 17883968
m 6 0x06347849 --rw-rw-rw- root root 1 65626
m 7 0xffffffff --rw-r--rw- root root 0 22908
m 8 0x00a5c581 --rw------- sfmdb users 1 9101312
m 9 0x5e18000a --rw------- root root 1 512
m 522 0x27f7a1e0 --rw-r----- oracle dba 38 373301248
m 2059 0x352cecf8 --rw-r----- oracle dba 28 373301248
m 7692 0x4b396504 --rw-r----- oracle dba 28 423632896
m 3085 0x0640d5a8 --rw-r----- oracle dba 27 373301248
m 3086 0xd6d33de0 --rw-r----- oracle dba 40 457187328

3) Please find attached the files created by system.perf.sh

Many thanks,

Mark P.

[Bill Hassell]

From ipcs, it appears that you are using about 3GB of your 6GB of RAM for shared memory. This is not a problem as long as the system is performing well. The warnings about memory are simply informing you that physical RAM is just about used up but from swapinfo, you can add a lot more processes and shared memory without a program failure.

But there is a consequence to running 150% to 200% more memory than you actually have. The whole idea behind virtual memory is that not everything needs to run at the same time, especially with interactive applications. However, that may not always be the case -- if all processes must run at the same time then paging out and in will take place with an enormous impact on performance. To see if this is happening, run vmstat 2 10 when the system is very busy and look at the po (page out) column. If the numbers are less than 15-20, then memory usage is fine and there is no impact to performance. Anything over 100 means severe paging is taking place and you'll need to adjust memory usage.

There are two ways to adjust memory usage:

1. Tell the DBA to reduce the size of the Oracle SGA, perhaps 25%-50% smaller. Of course, this will impact performance too.

2. Add another 4-6 GB of RAM. That will immediately improve performance.

[Mark Parsons]

Well po seems to be fine and at zero:

# vmstat 2 10
procs memory page
faults cpu
r b w avm free re at pi po fr de sr in
sy cs us sy id
2 0 0 1653585 44969 295 28 1 0 0 0 30 1080
9135 493 10 5 85
2 0 0 1653585 45191 723 228 2 0 0 0 0 1174
26206 883 7 24 68
2 0 0 1653585 43754 628 197 3 0 0 0 0 1135
24330 791 5 10 85
1 0 0 1644815 43753 403 125 1 0 0 0 0 1105
16903 621 0 3 97
1 0 0 1644815 43753 260 80 2 0 0 0 0 1108
12267 534 1 2 97
1 0 0 1644815 43962 501 107 1 0 0 0 0 1089
17470 590 11 12 77
1 0 0 1644815 45416 322 68 1 0 0 0 0 1065
12291 488 1 2 97
1 0 0 1644815 45413 207 43 1 0 0 0 0 1047
8839 416 0 2 98
1 0 0 1635470 45413 133 27 0 0 0 0 0 1038
6806 375 1 1 98
1 0 0 1635470 43954 225 19 0 0 0 0 0 1034
6634 378 9 2 89

Sorry for my inexperience but how do I increase ram anyway?

Mark P.

[TTr]

> how do I increase ram anyway?

You need to buy some memory and install it. The rp5470 has 16 memory slots in it and you need to find out if you have any free slots before you buy any memory.

If you have STM installed, you can get the physical memory information.

Try this

echo "selclass qualifier memory; info; wait; infolog" | /usr/sbin/cstm

Depending on what is installed you may be able to add more memory or remove smaller memory SIMMs and add larger ones.