memory usage 97%

[???_185]

my symtem info is like this.
Shared Memory:
m 0 0x410c181e --rw-rw-rw- root root
m 1 0x4e0c0002 --rw-rw-rw- root root
m 2 0x410c04f9 --rw-rw-rw- root root
m 327683 0x0c6629c9 --rw-r----- root root
m 4 0x06347849 --rw-rw-rw- root root
m 32773 0xffffffff --rw-r--r-- root root
m 458758 0x5e0d324c --rw------- root root
m 3112967 0xb540c1cc --rw-r----- oracle dba
m 8552456 0x500c7e3c --r--r--r-- bin bin
m 2555913 0x6a501995 --rw-rw-rw- root root




APPLICATION LIST Users= 4
Num Active CPU AvgCPU Logl Phys Res Virt
Idx Application Procs Procs Util Util IO IO Mem Mem
--------------------------------------------------------------------------------
1 other 1 0 0.0 0.0 0.0 0.0 792kb 864kb
2 network 36 2 0.0 0.0 0.0 2.0 25.8mb 71.2mb
3 memory_management 4 1 0.0 0.1 0.0 12.2 3.3mb 3.9mb
4 Oracle 15 3 1.7 0.3 0.4 0.4 561.3mb 752.9mb
5 other_user_root 105 16 2.3 1.9 365.0 517.2 157.7mb 305.3mb




CPU Util 0% 0% 0%
memory util 97% 97% 97%

how can i solve this problem seqeuntially?

[Arunvijai_4]

Hello, Can you post # vmstat 1 5 and sar -d output ?

-Arun

[Muthukumar_5]

check out which process is using more memory as,

UNIX95= ps -ef -o sz,vsz,pid,comm | sort -rnk 1 | more

--
Muthu

[Muthukumar_5]

can you post os information? Did you tune any kernel parameters?

--
Muthu

[???_185]

-os
shmmax 4289159183 4289159183 Immed

-oracle db ora file.
shared_pool_size=25165824
processes=1000
large_pool_size=10485760
log_buffer=32768
open_cursors=300

any infomation you need?

[Arunvijai_4]

Just find out, # UNIX95= ps -ef -o pcpu="CPU %" -o ruser=USER -o pid,args|awk '{if($1>=40)print $0}'|sort -rk1|head -16

-Arun

[Devender Khatana]

Hi,

What is the value of dbc_min_pct and dbc_max_pct kernel paramaters?

HTH,
Devender

[Frank de Vries]

You have allocated a 3 giga SGA
on a total of what looks like 14giga.
(quite normal I would say)

What do you get with swapinfo ?
swapinfo -amt
and are sar -c (scalls and forks)
unusual ?

What I am trying to question,
can we rely on the 97% memory allocation
or is there some memory leaking ?

[Victor BERRIDGE]

Hi,

Looks like you have less device swap than RAM size...(the size of device swap determines how much memory can be used for processes...)

To confirm we need to see the output of swapinfo -tam
maybe also
grep Physical /var/adm/syslog/syslog.log
and of course dbc_max_pct and if swapmem is on...

All the best

Victor

[???_185]

this is more info for the system.
root@cdr:/]# vmstat 1 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 3 0 141793 36149 4 0 0 0 0 0 0 4075 9844 1855 3 1 96
1 3 0 141793 36070 3 0 0 0 0 0 0 1082 565 128 0 0 100
1 3 0 141793 36070 2 0 0 0 0 0 0 1073 533 125 0 0 100
1 3 0 141793 36070 1 0 0 0 0 0 0 1080 563 126 0 0 100
1 3 0 141793 36070 0 0 0 0 0 0 0 1073 535 126 0 0 100
root@cdr:/]# sar -d 1 5

HP-UX cdr B.11.23 U ia64 01/11/06

09:10:39 device %busy avque r+w/s blks/s avwait avserv
09:10:40 c3t6d0 0.99 0.50 1 16 0.00 8.94
c16t0d1 0.99 0.50 2 63 0.00 0.36
09:10:41 c0t6d0 2.00 0.50 4 82 0.00 3.67
c3t6d0 1.00 0.50 2 18 0.00 9.82
09:10:42 c0t6d0 0.99 0.50 2 32 0.00 10.14
c3t6d0 0.99 0.50 2 32 0.00 8.66
09:10:43 c0t6d0 2.02 0.50 3 36 0.00 8.19
c3t6d0 2.02 0.50 2 32 0.00 6.87
09:10:44 c0t6d0 2.00 0.50 2 20 0.00 9.79
c3t6d0 1.00 0.50 1 16 0.00 9.74

Average c3t6d0 1.20 0.50 2 23 0.00 8.67
Average c16t0d1 0.20 0.50 0 13 0.00 0.36
Average c0t6d0 1.40 0.50 2 34 0.00 7.19
root@cdr:/]# UNIX95= ps -ef -o sz,vsz,pid,comm | sort -rnk 1 | more
31820 131376 2473 ora_arc1_CDR
31820 131376 2471 ora_arc0_CDR
31820 131376 2463 ora_lgwr_CDR
30540 123184 2461 ora_dbw1_CDR
30540 123184 2459 ora_dbw0_CDR
30472 122912 2457 ora_pmon_CDR
30064 120256 2467 ora_smon_CDR
30048 120192 2469 ora_reco_CDR
30028 120112 2465 ora_ckpt_CDR
29894 123392 11923 oracleCDR
29855 123328 18338 oracleCDR
29525 121216 22131 oracleCDR
29255 120192 22139 oracleCDR
29169 119872 22558 oracleCDR
4366 17660 2319 ia64_corehw
2628 10512 1320 midaemon
1659 7664 2931 tnslsnr
1533 7664 12220 tnslsnr
1361 7236 1190 cimserver
766 3976 1351 scopeux
695 4428 1621 rep_server
647 4656 1452 vxsvc
490 2520 22491 sshd:
478 3144 1187 rpcd
438 2216 22412 sshd:
436 2484 1624 alarmgen
406 2380 1105 snmpdm
398 2160 1346 spagent
393 1880 2190 dm_ses_enclosure
372 1880 809 sshd
348 1664 1507 rpc.lockd
329 1744 1723 psmctd
298 3064 1363 swagentd
284 1172 616 ipmon
265 2080 498 utmpd
259 1996 1623 agdbserver
258 1352 1252 diagmond
240 1360 22564 -sh
236 1312 22415 -sh
230 1096 830 rpcbind
224 1232 1472 rpc.mountd
213 852 1 init
207 1408 1953 dm_TL_adapter
205 1108 1729 registrar
202 812 1207 xntpd
201 1832 1563 icodd
195 1200 1300 ttd
194 1240 1540 sh
188 864 2416 sysstat_em
185 1716 1764 cmc_em
175 1536 1329 perflbd
165 1212 2370 memory_ia64
161 1516 1800 cpe_em
155 956 2220 dm_stape
151 832 1132 mib2agt
149 1052 1827 disk_em
145 2032 1501 rpc.statd
140 716 2274 fpl_em
127 608 1553 krsd
126 588 1579 kcmond
119 684 1124 ipv6agt
112 464 22493 -ksh
99 656 1173 fddi4subagt
96 560 1722 diaglogd
76 308 1223 pwgrd
75 480 1148 trapdestagt
73 1040 1557 dtlogin
73 328 1561 p_client
72 292 843 inetd
70 492 1116 hp_unixagt
69 612 1500 nfsd
69 612 1499 nfsd
69 612 1498 nfsd
69 612 1497 nfsd
69 612 1496 nfsd
69 612 1495 nfsd
69 612 1494 nfsd
69 612 1493 nfsd
69 612 1492 nfsd
69 612 1491 nfsd
69 612 1490 nfsd
69 612 1489 nfsd
69 612 1488 nfsd
69 612 1486 nfsd
69 612 1485 nfsd
69 612 1483 nfsd
68 600 1400 emsagent
62 404 495 syncer
58 236 639 syslogd
57 484 1482 nfsd
48 260 22563 ps
48 200 1235 cron
40 220 22565 more
26 264 680 netfmt
25 500 656 hotplugd
21 148 1562 cimserverd
17 492 1554 sfd
9 116 642 ptydaemon
5 84 669 nktl_daemon
4 84 1255 envd
2 92 679 ntl_reader
2 72 1552 getty
SZ VSZ PID COMMAND
0 0 22560
0 0 4381 lvmattachd
0 0 4380 lvmdevd
0 0 124 lvmattachd
0 0 123 lvmdevd
0 0 122 lvmattachd
0 0 121 lvmdevd
0 0 120 lvmattachd
0 0 119 lvmdevd
0 0 118 lvmattachd
0 0 117 lvmdevd
0 0 116 lvmattachd
0 0 115 lvmdevd
0 0 114 lvmattachd
0 0 113 lvmdevd
0 0 112 lvmattachd
0 0 111 lvmdevd
0 0 110 lvmattachd
0 0 109 lvmdevd
0 0 78 rng
0 0 53 vxfsd
0 0 46 kend
0 0 45 sblksched
0 0 44 sblksched
0 0 43 smpsched
0 0 42 smpsched
0 0 41 smpsched
0 0 40 smpsched
0 0 39 cmcd
0 0 38 pagezerod
0 0 37 schedcpu
0 0 36 eventdaemon
0 0 29 ipmid
0 0 28 ttisr
0 0 27 kmemdaemon
0 0 26 progressdaemon
0 0 25 strfreebd
0 0 24 strweld
0 0 23 strmem
0 0 22 supsched
0 0 21 pagetable_init_daemon
0 0 20 lvmattachd
0 0 19 lvmdevd
0 0 18 ksyncer_daemon
0 0 17 lvmschedd
0 0 16 lvmkd
0 0 15 lvmkd
0 0 14 lvmkd
0 0 13 lvmkd
0 0 12 lvmkd
0 0 11 lvmkd
0 0 10 autofskd
0 0 9 nfsktcpd
0 0 8 ioconfigd
0 0 4 unhashdaemon
0 0 3 statdaemon
0 0 2 vhand
0 0 0 swapper

OS info-
CPU info:
Number of CPUs = 4
Clock speed = 1500 MHz


Cache info:
L1 Instruction: size = 16 KB, associativity = 4
L1 Data: size = 16 KB, associativity = 4
L2 Unified: size = 256 KB, associativity = 8
L3 Unified: size = 6144 KB, associativity = 24

Memory = 4079 MB (3.983398 GB)


Platform info:
model string = "ia64 hp server rx7620"
machine serial number = SGH442656H

OS info:
sysname = HP-UX
nodename = cdr
release = B.11.23
version = U (unlimited-user license)
machine = ia64

swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 46 8146 1% 0 - 1 /dev/vg00/lvol2
reserve - 1433 -1433
memory 4079 2873 1206 70%
total 12271 4352 7919 35% - 0 -
# sar -c 1 5

HP-UX cdr B.11.23 U ia64 01/11/06

09:14:53 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s
09:14:54 946 7 4 0.00 0.00 540672 8192
09:14:55 413 8 5 0.00 0.00 24333 24333
09:14:56 449 7 3 0.00 0.00 32768 57344
09:14:57 395 7 3 0.00 0.00 0 0
09:14:58 650 18 13 0.00 0.00 163840 8192

Average 570 9 6 0.00 0.00 151497 19622

dbc_max_pct 20 20 Immed
dbc_min_pct 5 Default Immed

swapmem_on 1 Default

[Frank de Vries]

Hmm,
my analysis would be:

Overall not to bad looking systems,
except the swap memory of 70% is way to high and r/chars on the high side!!

Disks not overall busy, so i/o looks normal.
Top processes are Oracle's, but then i guess that is the idea
R/char is a bit high (so a lot of read activity on system calls)

Very difficult to give an reliable let alone
accurate conclusion, but I would investigate
your system activity overhead.

just user sar -u 5 55 and check your %sys
is proportionaly high (or when it is), (could this be the reason for the exeggerating paging, crosscheck swapinfo)

If yes there is something wrong at kernel
level - may need to look into patches,
wrongly configured parameters, bugs or
something we haven't figured out yet.

(More swap would probably not be a solution,
may just open the floodgates, hard to tell without furhter test, but keep it in mind)

That's my penny's worth.

[Muthukumar_5]

Post this,

# swapinfo -tam

I hope you are machine is requiring more swap space.

--
Muthu

[Victor BERRIDGE]

So most of your RAM is used for swap (70%),20% is used by buffers, doesnt leave much for the rest...
Since you have 2 oracle running I would believe with the RAM you have to fix :
swapmem_on to 0
dbc_max_pct 16
dbc_min_pct 8

And increase your SGA size to something more decent (to a total of 600-800MB max)


All the best
Victor

[???_185]

Total System Global Area 1044876528 bytes
Fixed Size 728304 bytes
Variable Size 503316480 bytes
Database Buffers 536870912 bytes
Redo Buffers 3960832 bytes
what is r/chars?
and if it is configured as swapmem_on to 0,
won't use pseu-do swap anymore. Do it cause some troble,donesn't it?