topic Re: High memory utilization in Operating System - HP-UX

High memory utilization

Jason Judge — Fri, 30 May 2008 12:44:18 GMT

Hi all, I am fairly new to the HP-UX world and have been tasked with taking over our HP-UX environments.
We are in the middle of load-testing a new corporate wide solution and there are memory questions being brought up to me.

We are running HP-UX 11iV2 on 2 7640's. They are both running Oracle RAC 10g. The nPar that assigned to the application have 12 Gig of memory each with one dual-core Itanium proc each. So to clarify.. 2 nPars with 12 Gig each and 2 cores each.

The memory utilization is showing like it is all gone, but there are conflicting reports.

Here is what I get on the worst node...

vmstat -n 1 1
VM
memory page faults
avm free re at pi po fr de sr in sy cs
1819429 225052 570 130 0 0 0 0 4 746 207161 1101
CPU
cpu procs
us sy id r b w
7 17 76 2 1 0
12 12 76

dmesg |grep Physical
Physical: 12533024 Kbytes, lockable: 9261448 Kbytes, available: 10769808 Kbytes

top -w
System: hanpl1v1 Fri May 30 08:36:01 2008
Load averages: 0.11, 0.12, 0.13
337 processes: 293 sleeping, 43 running, 1 zombie
Cpu states: 2 processors
LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0.11 9.1% 0.0% 8.9% 81.9% 0.0% 0.0% 0.0% 0.0%

Memory: 7353860K (6930216K) real, 9382164K (8864496K) virtual, 896164K free Page# 1/6

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
4 ? 51 root 152 20 25128K 22336K run 163:03 2.14 2.13 vxfsd
0 ? 3000 root 152 20 164M 45208K run 13:32 0.73 0.73 crsd.bin
4 ? 17192 oracle 152 20 689M 298M run 13:33 0.61 0.61 java
4 ? 4791 oracle 152 20 604M 185M run 4:13 0.52 0.52 java
4 ? 11699 oracle 152 20 571M 151M run 4:42 0.47 0.47 java
0 ? 4924 oracle 152 0 97132K 21080K run 19:16 0.46 0.46 ocssd.bin
0 pty/ttyp1 17329 oracle 156 10 34864K 3540K sleep 0:14 0.41 0.41 glance
4 ? 2394 root -16 20 40948K 16944K run 523:01 0.34 0.34 midaemon
4 pts/0 3989 root 178 20 11120K 1624K run 0:00 0.44 0.32 top
4 ? 2999 oracle 152 20 99428K 18268K run 2:43 0.30 0.29 evmd.bin
0 ? 2398 root 127 20 47232K 13192K sleep 10:23 0.26 0.26 scopeux
4 ? 3113 root 158 20 4440K 704K sleep 23:20 0.26 0.26 sh
4 ? 3249 oracle 152 20 142M 23136K run 1:14 0.25 0.25 racgimon
4 ? 20053 oracle 152 20 142M 23168K run 0:44 0.25 0.25 racgimon
0 ? 15737 oracle 152 20 140M 21420K run 0:49 0.25 0.25 racgimon
0 ? 2086 root 152 20 175M 63388K run 2:15 0.23 0.23 cimserver
4 ? 578 root 152 20 9524K 2576K run 28:49 0.22 0.22 utmpd
4 ? 38 root 152 20 576K 512K run 8:52 0.19 0.19 schedcpu
0 ? 2645 root 152 20 110M 14084K run 0:41 0.18 0.18 vxsvc
4 ? 2635 root 152 20 129M 25884K run 31:55 0.17 0.17 coda
0 ? 17413 oracle 152 20 97020K 21492K run 3:04 0.17 0.17 emagent
4 ? 2749 oracle 152 20 109M 33360K run 13:25 0.15 0.15 emagent
0 ? 15842 jjudge 154 20 15656K 1280K sleep 0:00 0.15 0.15 sshd:
4 ? 15314 oracle 154 20 2407M 19260K sleep 2:26 0.15 0.15 ora_lms0_HANTEST1
4 ? 2444 root 152 20 45876K 4840K run 2:16 0.14 0.14 ovcd
4 ? 6531 oracle 152 20 16268K 3188K run 0:17 0.14 0.14 ons
4 ? 15316 oracle 154 20 2407M 19244K sleep 2:15 0.12 0.12 ora_lms1_HANTEST1
4 ? 39 root 191 20 432K 384K run 1:30 0.12 0.12 pagezerod
0 ? 2079 root 152 20 25416K 3628K run 0:03 0.12 0.12 rpcd
4 ? 6505 oracle 152 20 117M 11456K run 1:32 0.12 0.12 racgimon
4 ? 15596 oracle 152 20 229M 154M run 15:49 0.12 0.12 emagent
4 ? 19908 oracle 154 20 1154M 15432K sleep 1:17 0.12 0.12 ora_lmon_HANDEV1
4 ? 15310 oracle 154 20 2404M 15432K sleep 1:29 0.12 0.12 ora_lmon_HANTEST1
4 ? 19918 oracle 154 20 1158M 19244K sleep 1:41 0.11 0.11 ora_lms1_HANDEV1
4 ? 3150 oracle 154 20 773M 19240K sleep 2:20 0.11 0.11 ora_lms0_HANWOTST1
4 ? 19956 oracle 154 20 1148M 9636K sleep 0:45 0.11 0.11 ora_cjq0_HANDEV1
4 ? 5542 oracle 154 20 453M 20248K sleep 9:37 0.10 0.10 asm_lmd0_+ASM1
4 ? 3146 oracle 154 20 769M 15432K sleep 1:58 0.10 0.10 ora_lmon_HANWOTST1
4 ? 19912 oracle 154 20 1158M 19244K sleep 1:50 0.09 0.09 ora_lms0_HANDEV1
4 ? 3152 oracle 154 20 773M 19240K sleep 2:18 0.09 0.09 ora_lms1_HANWOTST1
0 ? 19910 oracle 154 20 1158M 19240K sleep 1:02 0.08 0.08 ora_lmd0_HANDEV1
0 ? 18 root 191 20 504K 448K run 2:11 0.08 0.08 ksyncer_daemon
4 ? 2429 root 152 20 22760K 2900K run 0:06 0.08 0.08 swagentd
0 ? 5544 oracle 154 20 453M 20248K sleep 6:32 0.07 0.07 asm_lms0_+ASM1
4 ? 3863 oracle 154 20 2392M 4392K sleep 0:00 0.09 0.07 ora_j000_HANTEST1
4 ? 5540 oracle 154 20 444M 12080K sleep 6:43 0.07 0.07 asm_lmon_+ASM1
4 ? 5536 oracle 154 20 443M 11056K sleep 5:19 0.07 0.07 asm_diag_+ASM1
0 ? 5570 oracle 154 20 445M 13368K sleep 3:15 0.06 0.06 asm_gmon_+ASM1
0 ? 8502 oracle 152 20 55236K 5188K run 2:57 0.06 0.06 tnslsnr
4 ? 16367 rhall 154 20 14032K 4836K sleep 0:01 0.06 0.06 hpterm
4 ? 20254 oracle 154 20 1145M 5516K sleep 0:12 0.06 0.06 oracleHANDEV1
4 ? 3746 oracle 154 20 760M 5504K sleep 0:00 0.08 0.06 ora_j000_HANWOTST1
0 ? 19902 oracle 154 20 1145M 6448K sleep 0:25 0.06 0.06 ora_pmon_HANDEV1
4 ? 1980 root 152 20 9432K 1524K run 3:33 0.06 0.06 snmpdm
4 ? 2534 root 152 20 39496K 4764K run 0:00 0.06 0.06 ovbbccb
4 ? 1 root 152 20 1960K 564K run 4:56 0.06 0.06 init
0 ? 3148 oracle 154 20 773M 19240K sleep 1:20 0.05 0.05 ora_lmd0_HANWOTST1
4 ? 15312 oracle 154 20 2407M 19256K sleep 1:08 0.05 0.05 ora_lmd0_HANTEST1
4 ? 15360 oracle 154 20 2418M 25040K sleep 0:17 0.05 0.05 ora_mmon_HANTEST1

GlancePlus C.04.50.00 08:41:47 hanpl1v1 ia64 Current Avg High
----------------------------------------------------------------------------------------------------------------------------------
CPU Util S SAU U | 10% 10% 10%
Disk Util F FV | 7% 7% 7%
Mem Util S SU UB B | 93% 93% 93%
Swap Util U UR R | 35% 35% 35%
----------------------------------------------------------------------------------------------------------------------------------
PROCESS LIST Users= 3
User CPU Util Cum Disk Thd
Process Name PID PPID Pri Name ( 200% max) CPU IO Rate RSS Cnt
--------------------------------------------------------------------------------
emdctl 14204 14203 179 oracle 5.4/ 5.4 0.1 0.0/ 0.0 84kb 1
glance 14174 15982 158 root 1.3/ 2.2 0.0 0.0/ 1.6 436kb 1
ora_cjq0_HA 3194 1 154 oracle 0.7/ 0.1 82.5 0.0/ 0.0 26.3mb 1
ora_s038_HA 15613 1 154 oracle 0.0/ 0.0 0.3 0.0/ 0.0 32.8mb 1
ora_lmon_HA 15310 1 154 oracle 0.0/ 0.1 89.9 2.1/ 1.1 42.9mb 1
ora_rbal_HA 20061 1 154 oracle 0.0/ 0.0 1.6 0.0/ 0.0 41.0mb 1
racgimon 15737 1 168 oracle 0.0/ 0.1 48.1 0.0/ 0.0 24.3mb 17
ora_s049_HA 15667 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 32.8mb 1
ora_dbw0_HA 15320 1 154 oracle 0.0/ 0.0 12.5 0.0/ 0.9 41.0mb 1
ora_s017_HA 15488 1 154 oracle 0.0/ 0.0 0.3 0.0/ 0.0 32.8mb 1
ora_q001_HA 20838 1 154 oracle 0.0/ 0.0 9.6 0.0/ 0.0 37.4mb 1
swapper 0 0 127 root 0.0/ 0.0 8.5 1.4/ 2.0 64kb 1
ora_mmon_HA 15360 1 154 oracle 0.0/ 0.0 18.0 0.0/ 0.0 52.2mb 1
ora_lgwr_HA 19948 1 154 oracle 0.0/ 0.0 7.4 0.0/ 0.5 50.3mb 1
ora_ckpt_HA 19950 1 154 oracle 0.0/ 0.0 11.1 0.0/ 0.3 42.4mb 1
emagent 17413 9357 154 oracle 0.0/ 0.0 181.5 0.0/ 0.0 26.7mb 6
ora_s032_HA 15576 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 32.8mb 1
ora_lgwr_HA 3186 1 154 oracle 0.0/ 0.0 9.6 0.0/ 0.5 38.1mb 1
oracleHANWO 7219 1 154 oracle 0.0/ 0.0 1.1 0.0/ 0.0 26.8mb 1
ora_j000_HA 13066 1 154 oracle 0.0/ 0.2 0.1 0.0/ 0.0 35.7mb 1
ora_s035_HA 15582 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 32.8mb 1
ora_mmnl_HA 3198 1 154 oracle 0.0/ 0.0 31.9 0.0/ 0.0 21.6mb 1
ora_lck0_HA 3228 1 154 oracle 0.0/ 0.0 19.5 0.0/ 0.0 28.7mb 1
ora_pmon_HA 3140 1 154 oracle 0.0/ 0.0 42.5 0.0/ 0.0 23.2mb 1
ora_s027_HA 15538 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 32.8mb 1
oracle+ASM1 15739 1 154 oracle 0.0/ 0.0 3.5 0.0/ 0.0 23.6mb 1
ora_o001_HA 17956 1 154 oracle 0.0/ 0.0 0.1 0.0/ 0.0 33.4mb 1
ora_s004_HA 15409 1 154 oracle 0.0/ 0.0 1.3 0.0/ 0.0 36.2mb 1
ora_rbal_HA 3270 1 154 oracle 0.0/ 0.0 2.6 0.0/ 0.0 28.7mb 1
ora_asmb_HA 3242 1 154 oracle 0.0/ 0.0 3.9 0.0/ 0.0 27.2mb 1
ora_s045_HA 15631 1 154 oracle 0.0/ 0.0 0.3 0.0/ 0.0 32.8mb 1
ora_lms1_HA 15316 1 154 oracle 0.0/ 0.2 137.0 0.0/ 0.0 46.6mb 1
emagent 15596 12473 154 oracle 0.0/ 0.1 935.5 0.0/ 0.1 156.2mb 6
ora_diag_HA 3142 1 154 oracle 0.0/ 0.0 49.8 0.0/ 0.0 27.7mb 1
ora_s033_HA 15578 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 32.8mb 1
ora_s012_HA 15478 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 33.0mb 1
asm_diag_+A 5536 1 154 oracle 0.0/ 0.1 322.0 0.0/ 0.0 28.1mb 1
ora_s040_HA 15621 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 32.8mb 1
ora_pz98_HA 8350 1 154 oracle 0.0/ 0.0 0.1 0.0/ 0.0 33.5mb 1
ora_mmnl_HA 15362 1 154 oracle 0.0/ 0.0 21.3 0.0/ 0.0 34.2mb 1
ora_s022_HA 15528 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 32.8mb 1
ora_s008_HA 15442 1 154 oracle 0.0/ 0.0 0.3 0.0/ 0.0 33.0mb 1
oracleHANTE 15928 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 36.1mb 1
ora_mmon_HA 3196 1 154 oracle 0.0/ 0.0 16.3 0.0/ 0.0 37.6mb 1
ora_s041_HA 15623 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 32.8mb 1
asm_lgwr_+A 5550 1 154 oracle 0.0/ 0.0 16.7 0.0/ 0.0 24.3mb 1
ora_smon_HA 15354 1 154 oracle 0.0/ 0.0 7.1 0.0/ 0.0 37.1mb 1
ora_s002_HA 15399 1 154 oracle 0.0/ 0.0 21.7 0.0/ 0.0 37.7mb 1
ora_s029_HA 15570 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 32.8mb 1
ora_q001_HA 3988 1 154 oracle 0.0/ 0.0 1.2 0.0/ 0.0 21.3mb 1
racgimon 20053 1 168 oracle 0.0/ 0.1 43.6 0.0/ 0.0 26.0mb 18
oracleHANWO 5047 1 154 oracle 0.0/ 0.0 29.7 0.0/ 0.0 27.9mb 1
ora_s025_HA 15534 1 154 oracle 0.0/ 0.0 0.3 0.0/ 0.0 33.0mb 1
ora_pz99_HA 15916 1 154 oracle 0.0/ 0.1 34.7 0.0/ 0.0 37.2mb 1
ora_s023_HA 15530 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 32.8mb 1
oracleHANWO 5057 1 154 oracle 0.0/ 0.0 57.2 0.0/ 0.0 26.7mb 1
ora_s043_HA 15627 1 154 oracle 0.0/ 0.0 0.2 0.0/ 0.0 34.6mb 1

This is actually better than normal since mem utilization has been hitting 99%.
They want answers, but I don't know what to tell them. Can someone please help me make clear of this and/or suggest how to free the memory????

Re: High memory utilization

Deepak Kulkarni — Fri, 30 May 2008 12:55:37 GMT

Hi,

What is the output of /usr/sbin/swapinfo -tm

Regards
DK

Re: High memory utilization

Craig A. Sharp — Fri, 30 May 2008 12:57:42 GMT

Looks to me like you have some performance tuning to do. You have many Oracle processes that are using up all your memory. If you look at the memory utilization, all of your memory is being chewed up in user space. If you can, try adding additional memory to the npars. The other option is to tune down the memory that is being locked by Oracle. I am not an Oracle person but this is pretty standard across databases.

Craig

Re: High memory utilization

Jason Judge — Fri, 30 May 2008 13:01:11 GMT

Here is the swapinfo:

/usr/sbin/swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16320 0 16320 0% 0 - 1 /dev/vg00/lvol2
reserve - 7551 -7551
memory 12239 2438 9801 20%
total 28559 9989 18570 35% - 0 -

Keep in mind, there is virtually no activity happening right now from users....

Re: High memory utilization

Craig A. Sharp — Fri, 30 May 2008 13:03:39 GMT

By swapping, you are showing serious memory pressure. What happens if you shutdown Oracle (if you can do that)?

Re: High memory utilization

Jason Judge — Fri, 30 May 2008 13:35:27 GMT

here is everything after a shutdown of Oracle....

System: hanpl1v1 Fri May 30 09:31:56 2008
Load averages: 0.09, 0.12, 0.14
172 processes: 135 sleeping, 36 running, 1 zombie
Cpu states: 2 processors
LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0.09 0.4% 0.0% 2.6% 97.0% 0.0% 0.0% 0.0% 0.0%

Memory: 1310504K (1222740K) real, 2746512K (2602384K) virtual, 6940164K free Page# 1/3

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 51 root 152 20 25128K 22336K run 164:22 5.29 5.29 vxfsd
4 ? 17192 oracle 152 20 690M 299M run 13:55 1.64 1.64 java
0 ? 10663 oracle 152 20 564M 114M run 0:15 1.27 1.27 java
4 ? 11699 oracle 152 20 571M 151M run 4:47 0.51 0.51 java
0 pty/ttyp3 25028 oracle 156 10 34592K 3236K sleep 0:00 0.33 0.33 glance
0 ? 2635 root 152 20 129M 25900K run 32:04 0.33 0.33 coda
0 ? 2086 root 152 20 175M 63388K run 2:15 0.23 0.23 cimserver
0 pts/0 3177 root 178 20 11120K 1448K run 0:00 0.23 0.23 top
0 ? 2645 root 152 20 110M 14084K run 0:41 0.18 0.18 vxsvc
4 ? 2394 root -16 20 40948K 16944K run 523:13 0.18 0.18 midaemon
4 ? 15596 oracle 152 20 230M 154M run 15:55 0.17 0.17 emagent
0 ? 38 root 152 20 576K 512K run 8:55 0.16 0.16 schedcpu
4 ? 578 root 152 20 9524K 2576K run 28:57 0.15 0.15 utmpd
4 pty/ttyp2 20330 jjudge 154 10 46164K 5440K sleep 0:04 0.14 0.14 gpm
0 ? 2444 root 152 20 45876K 4840K run 2:17 0.14 0.14 ovcd
4 ? 2749 oracle 152 20 109M 33360K run 13:31 0.12 0.12 emagent
4 ? 39 root 191 20 432K 384K run 1:34 0.12 0.12 pagezerod
0 ? 2079 root 152 20 25416K 3628K run 0:03 0.12 0.12 rpcd
4 ? 15842 jjudge 154 20 15656K 1280K sleep 0:00 0.12 0.12 sshd:
0 ? 17413 oracle 152 20 97788K 22292K run 3:07 0.12 0.12 emagent
4 ? 16335 rhall 154 20 15912K 1488K sleep 0:00 0.11 0.11 sshd:
4 ? 16367 rhall 154 20 14272K 5060K sleep 0:02 0.10 0.10 hpterm
4 ? 2429 root 152 20 22760K 2900K run 0:06 0.08 0.08 swagentd
4 ? 2419 root 152 20 42572K 4904K run 20:51 0.07 0.07 perfalarm
0 ? 2006 root 154 20 9568K 768K sleep 2:20 0.06 0.06 ipv6agt
4 ? 23041 rhall 154 20 13968K 4800K sleep 0:00 0.06 0.06 hpterm
4 pty/ttyp1 16847 oracle 156 20 5128K 400K sleep 0:00 0.06 0.06 ksh
4 ? 18 root 191 20 504K 448K run 2:11 0.06 0.06 ksyncer_daemon
0 ? 1980 root 152 20 9432K 1524K run 3:35 0.06 0.06 snmpdm
0 ? 2534 root 152 20 39496K 4764K run 0:00 0.06 0.06 ovbbccb
4 ? 6083 jjudge 154 20 16040K 1600K sleep 0:00 0.06 0.06 sshd:
4 ? 1968 sfmdb 154 20 21808K 760K sleep 0:03 0.05 0.05 postmaster
0 ? 23037 rhall 154 20 15784K 1408K sleep 0:00 0.04 0.04 sshd:
4 ? 1 root 152 20 1960K 564K run 4:57 0.04 0.04 init
0 ? 40 root 191 20 144K 128K run 1:01 0.04 0.04 cmcd
4 ? 1628 root 152 20 10496K 2416K run 1:05 0.04 0.04 automountd
4 ? 1941 root 154 20 7716K 848K sleep 3:15 0.04 0.04 sendmail:
0 ? 4765 root 154 10 8668K 1244K sleep 4:30 0.04 0.04 psmctd
4 ? 2016 root 154 20 6332K 712K sleep 2:42 0.03 0.03 mib2agt
0 ? 1648 root 154 20 7084K 560K sleep 0:07 0.02 0.02 inetd
4 ? 28645 oracle 168 20 20216K 10052K sleep 1:08 0.02 0.02 perl
0 ? 0 root 127 20 72K 64K sleep 0:20 0.02 0.02 swapper
0 ? 2 root 128 20 72K 64K sleep 0:10 0.02 0.02 vhand
4 ? 3 root 128 20 72K 64K sleep 1:06 0.02 0.02 statdaemon
4 ? 4 root 128 20 72K 64K sleep 0:09 0.02 0.02 unhashdaemon
0 ? 9 root 152 20 72K 64K sleep 0:15 0.02 0.02 nfsktcpd
0 ? 10 root 152 20 288K 256K run 0:00 0.02 0.02 autofskd
0 ? 11 root 147 20 72K 64K sleep 0:17 0.02 0.02 lvmkd
0 ? 12 root 147 20 72K 64K sleep 0:18 0.02 0.02 lvmkd
0 ? 13 root 147 20 72K 64K sleep 0:18 0.02 0.02 lvmkd
0 ? 14 root 147 20 72K 64K sleep 0:18 0.02 0.02 lvmkd
0 ? 15 root 147 20 72K 64K sleep 0:17 0.02 0.02 lvmkd
0 ? 16 root 147 20 72K 64K sleep 0:17 0.02 0.02 lvmkd
0 ? 17 root 148 20 72K 64K sleep 0:40 0.02 0.02 lvmschedd
0 ? 19 root 133 20 72K 64K sleep 0:01 0.02 0.02 lvmdevd
4 ? 26 root -32 20 72K 64K sleep 0:16 0.02 0.02 progressdaemon
0 ? 28 root -32 20 72K 64K sleep 0:00 0.02 0.02 ttisr
4 ? 29 root 152 20 144K 128K run 0:02 0.02 0.02 ipmid
4 ? 41 root 100 20 72K 64K sleep 0:06 0.02 0.02 smpsched
0 ? 42 root 100 20 72K 64K sleep 0:06 0.02 0.02 smpsched

/usr/sbin/swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16320 0 16320 0% 0 - 1 /dev/vg00/lvol2
reserve - 1442 -1442
memory 12239 2431 9808 20%
total 28559 3873 24686 14% - 0 -

dmesg |grep Physical
Physical: 12533024 Kbytes, lockable: 9261448 Kbytes, available: 10769808 Kbytes
root@hanpl1v1[/] > vmstat -n 1 1
VM
memory page faults
avm free re at pi po fr de sr in sy cs
640796 1735437 570 130 0 0 0 0 4 745 206411 1100
CPU
cpu procs
us sy id r b w
7 17 76 1 0 0
12 12 76

GlancePlus C.04.50.00 09:34:56 hanpl1v1 ia64 Current Avg High
----------------------------------------------------------------------------------------------------------------------------------
CPU Util S SA | 8% 8% 8%
Disk Util F F | 4% 4% 4%
Mem Util S SU UB B | 45% 45% 45%
Swap Util U UR R | 14% 14% 14%
----------------------------------------------------------------------------------------------------------------------------------
PROCESS LIST Users= 5
User CPU Util Cum Disk Thd
Process Name PID PPID Pri Name ( 200% max) CPU IO Rate RSS Cnt
--------------------------------------------------------------------------------
vxfsd 51 0 134 root 11.0/ 1.5 9857.7 4.2/ 8.9 21.8mb 34
glance 4320 15982 158 root 1.4/ 1.7 0.0 0.0/ 0.0 1.8mb 1
swapper 0 0 127 root 0.0/ 0.0 8.5 1.4/ 2.0 64kb 1
emagent 2749 28645 154 oracle 0.0/ 0.1 799.1 0.0/ 0.1 38.3mb 6
emagent 17413 9357 168 oracle 0.0/ 0.0 183.9 0.0/ 0.0 27.7mb 6
emagent 15596 12473 168 oracle 0.0/ 0.1 940.7 0.0/ 0.1 157.1mb 6
java 11699 9357 168 oracle 0.0/ 0.2 266.8 0.0/ 0.0 157.6mb 28
vxsvc 2645 1 154 root 0.0/ 0.0 41.6 0.0/ 0.0 53.1mb 18
java 17192 28645 168 oracle 0.0/ 0.4 804.3 0.0/ 0.2 305.0mb 32
coda 2635 2444 168 root 0.0/ 0.3 1924.7 0.0/ 0.0 39.9mb 8
scopeux 2398 1 127 root 0.0/ 0.1 618.5 0.0/ 0.0 20.6mb 1
midaemon 2394 1 -16 root 0.0/ 4.6 31289.1 0.0/ 0.0 22.9mb 2
cimserver 2086 1 168 root 0.0/ 0.0 119.4 0.0/ 0.0 101.3mb 20
java 10663 12473 154 oracle 0.0/ 1.6 18.6 0.0/ 0.3 120.5mb 23

We have another server that is even worse on mem right now, but this is the only one I can play with....

Re: High memory utilization

Craig A. Sharp — Fri, 30 May 2008 13:47:16 GMT

So by this new information, it is the Oracle processes that are eating up your memory. You need to have your dba's look at the configuration and see if they can tune down the amount they are locking. This value should be able to be adjusted. Being that I am not an Oracle person you might also ask them if they can reduce the number of engine processes. All of your memory utilization is user process based as I thought.

Re: High memory utilization

Craig A. Sharp — Fri, 30 May 2008 13:53:49 GMT

Have the kernels been tuned for Oracle? Curious, what is the dbc_max_pct and dbc_min_pct?

Re: High memory utilization

Jason Judge — Fri, 30 May 2008 14:11:59 GMT

kcusage
Tunable Usage / Setting
=============================================
dbc_max_pct 18 / 50
maxdsiz 157081600 / 1073741824
maxdsiz_64bit 54083584 / 2147483648
maxfiles_lim 99 / 4096
maxssiz 1179648 / 134217728
maxssiz_64bit 786432 / 1073741824
maxtsiz 1957888 / 100663296
maxtsiz_64bit 201326592 / 1073741824
maxuprc 225 / 256
max_thread_proc 45 / 1024
maxvgs 1 / 10
msgmni 2 / 4096
msgseg 0 / 32767
msgtql 0 / 4096
nfile 4043 / 63488
nflocks 21 / 4096
ninode 1346 / 34816
nkthread 830 / 7184
nproc 385 / 4096
npty 4 / 60
nstrpty 1 / 60
nstrtel 0 / 60
nswapdev 1 / 10
nswapfs 0 / 10
semmni 37 / 4096
semmns 1034 / 8192
shmmax 2214600704 / 12884901888
shmmni 12 / 512
shmseg 3 / 120

Is there a better command to use. I saw nothing on min...

Re: High memory utilization

trpjr1 — Fri, 30 May 2008 14:14:42 GMT

kmtune | grep dbc

Re: High memory utilization

Jason Judge — Fri, 30 May 2008 14:15:36 GMT

kmtune | grep dbc
kmtune is a wrapper script which exists for compatibility reasons only.
The underlying command used is 'kctune'. New or modified scripts or
procedures should use kctune directly.
About to execute: kctune
dbc_max_pct 50 Default Immed
dbc_min_pct 5 Default Immed

Re: High memory utilization

TTr — Fri, 30 May 2008 14:17:24 GMT

> it is the Oracle processes that are eating up your memory

There is nothing wrong with oracle processes "eating up" the memory. If you don't want memory to be eaten up why pay for it and have it sitting in the server?

The Glance output is what I go with.

> Mem Util 93% 93% 93%
> Swap Util 35% 35% 35%

93% shows that you are getting good use of your memory. And the swap address space (note address space) is only at 35%.

TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16320 0 16320 0% 0 - 1 /dev/vg00/lvol2

There is no disk swapping either, (USED=0).

Buffer cache seems to have been lowered that the default 50% based on the distance between the two Bs in the Glance memory output. You may want to lower it further.

Why are you concerned about memory usage? Are you having performance issues?

Re: High memory utilization

Jason Judge — Fri, 30 May 2008 14:22:17 GMT

The DBA's and consultants are looking at the loads during performance testing and seeing little 'free' memory and high utilization. Since I am new to this area, I can't give them an explanation. I am aware that I don't really understand how UNIX uses it's memory so I can't tell them that there isw not a problem...
The loads we have been running have been on the 40 - 50 user range. The load I showed had like one or two users. The expected usage is about 800 users. So they want to know if we need to buy more RAM. I don't kinow how to answer them because I don't fully understand it myself.
Please excuse my ignorance on this area.....

:)

Re: High memory utilization

TTr — Fri, 30 May 2008 14:36:09 GMT

Your buffer cache is set at 50 and the lower is set to 17. So when the server needs memory it will squeeze the buffer cache down but based on your minimum setting it will not sqeeze it further than 17% of your total physical memory. With 12.5 GB of phys memory, I would set ti to 2 - 4 % only. I have servers where the buffer cache is set to 1-2%.

If you noticed the difference from when oracle was runing to when you shut it down the memory utilization went from 93% to 45% so oracle is really using no more than 48% of your 12.5 GB. I say this is very reasonable these days.

Re: High memory utilization

Jason Judge — Fri, 30 May 2008 14:38:10 GMT

So, is it normal for HP-UX to utalize as much memory as it does when Oracle is not running????

Re: High memory utilization

TTr — Fri, 30 May 2008 14:42:32 GMT

> seeing little 'free' memory

Are you they planning on adding any more users or increase the SGA? If so the you may need additional memory. But if the server during the maximum usage in terms of processing and concurrent users does not change and stays at arount 93%, you do NOT need to add memory. In fact if you reduce the buffer cache setting the memory utilization will decrease.

On the other hand, you may want to allocate more memory to the SGA to improve performance. In that case, yes you should consider buying more memory.

Re: High memory utilization

Don Morris_1 — Fri, 30 May 2008 14:44:28 GMT

Yes, it is. HP-UX favors caching for performance versus large amounts of free memory. (If you have a PC/x86 background, you'll hear similar discussions about Vista RAM use). Until RAM is below a certain point (lotsfree -- see the Memory Management White Paper at: http://docs.hp.com/en/1218/mem_mgt.html ), the system will allow objects and pages containing valid I/O data to remain regardless of size. Only when the system begins to come under memory pressure is this memory released for new allocations. In practice, this means that even an idle system can be sitting at 98 to 99% memory consumption if there was a prior kernel / I/O load causing the caches to fill up.

Free memory is not a good metric for if your load is stressing the box. What you don't want to see is Page Outs in Glance or vmstat [some is normal... but constant pi/po activity indicates paging is taking place and the memory pressure is impacting performance. Cache reclamation will not cause paging... the data in the caches is just discarded and the page is reused for new purposes].

Re: High memory utilization

TTr — Fri, 30 May 2008 15:05:46 GMT

> So, is it normal for HP-UX to utalize as much memory

There are 3 java processes and 3 ema processes all owned by oracle and still running, using another 700MB of memory (reserved to be precise). So hP-UX is using even less memory than that. Then there is some buffer cache that has not been released yet. Look in the respective glance screens by pressing "m", "w".

As Don pointed out, you want to minimize pi/po although depending on the DB activity it may still be significant. Increasing the oracle SGA and changing some internal oracle parameters helps in this case.

You can turn the table around and ask the DBAs to look into this as well and tune the SGA internal areas and parameters to improve performance.