not able to recover memory

[kkdurrani]

Hello all,

I am sure the same question has been posted again and again. After researching so much still I don't see where the memory is lost. Can you please help.

I have pasted the outputs from the top, collect and vmstat -P. We have 6G memory with 3 engines of sybase running. The shared memory allocated within sybase is 1.7G. Even though the free memory ranges between 200M to 14MB. It is 14M when users complain of performance. Where is my rest of the memory is gone. is there any way I can retrieve it. The info on this forum suggests reducing the UBC. what is the side effects reducing UBC. The UBC cache hit ratio is 45%. so does it make sense to reduce the max or min % ?.

One of the things I noticed in top is that the kernel takes 3.6G even after the reboot. Is there anyway to reduce the Kernel VSZ? we are running 4.0F.

Any suggestions are appreciated.


/>> top -n 100
load averages: 0.14, 0.82, 0.85 17:41:52
80 processes: 1 running, 28 sleeping, 51 idle

Memory: Real: 4191M/6024M act/tot Virtual: 77M/9205M use/tot Free: 206M


PID USERNAME PRI NICE SIZE RES STATE TIME CPU COMMAND
1315 root 44 0 9952K 6578K sleep 20:04 0.70% psdc
11131 root 24 -20 5504K 3833K sleep 9:48 0.40% collect
18097 sybhrms 44 0 1748M 1497M sleep 136:16 0.10% dataserver
20652 root 44 0 5664K 2072K run 0:00 0.00% top
30700 sybhrms 44 0 1747M 1375M sleep 438:53 0.00% dataserver
30701 sybhrms 44 0 1747M 1406M sleep 256:47 0.00% dataserver
31540 pshrms 44 0 9008K 4800K sleep 4:36 0.00% PSRUN
1161 root 43 -1 6088K 2228K sleep 3:35 0.00% advfsd
11754 root 44 0 2552K 270K sleep 1:10 0.00% os_mibs
34 root 44 0 1704K 90K sleep 1:05 0.00% update
1117 root 42 0 6768K 2580K sleep 0:57 0.00% insightd
11706 root 44 0 1872K 180K sleep 0:35 0.00% snmpd
31427 pshrms 44 0 13M 10M sleep 0:08 0.00% WSH
31425 pshrms 44 0 11M 8822K sleep 0:08 0.00% WSH
31426 pshrms 42 0 7872K 5013K sleep 0:06 0.00% WSH
1 root 42 0 496K 49K sleep 0:05 0.00% init
17334 root 44 0 7312K 5439K sleep 0:04 0.00% sys_check
1123 root 43 -1 2520K 417K sleep 0:01 0.00% pmgrd
20495 root 45 0 7312K 5398K sleep 0:00 0.00% sys_check
31290 pshrms 44 0 5336K 1613K sleep 0:00 0.00% BBL
31424 pshrms 44 0 3920K 802K sleep 0:00 0.00% WSL
11761 root 44 0 2968K 663K sleep 0:00 0.00% cpq_mibs
17343 root 44 0 4808K 614K sleep 0:00 0.00% sshd
20310 root 44 0 4808K 614K sleep 0:00 0.00% sshd
20354 root 44 0 2096K 294K sleep 0:00 0.00% ksh
11760 root 44 0 2368K 155K sleep 0:00 0.00% svrMgt_mib
189 root 44 0 1744K 155K sleep 0:00 0.00% syslogd
384 root 44 0 1712K 0K sleep 0:00 0.00% nfsd
20642 root 24 -20 3800K 2072K sleep 0:00 0.00% collect


Virtual Memory Statistics: (pagesize = 8192)
procs memory pages intr cpu
r w u act free wire fault cow zero react pin pout in sy cs us sy id
5 142 29 669K 27K 74K 9M 1M 4M 156K 991K 4355 50 11K 1K 15 4 80
6 142 29 670K 26K 74K 44K 10K 6522 0 4303 0 16 10K 1K 26 27 47
6 142 29 648K 48K 74K 42K 10K 6062 0 4143 0 28 14K 1K 33 28 39


# Process Statistics (RSS & VSZ in KBytes)
# PID User %CPU RSS VSZ UsrTim SysTim IBk OBk Maj Min Command
20770 root 1.9 5.4M 7.5M 0.002 0.098 0 3 0 27 sys_check
0 root 1.6 292M 3.6G 0.000 0.023 0 0 0 0 kernel idle
30700 sybhrms 1.2 1.4G 1.8G 0.027 0.001 0 0 0 0 dataserver
21706 root 0.7 237K 1.4M 0.001 0.006 0 0 0 9 cliscript4
1315 root 0.5 6.6M 10M 0.000 0.011 0 0 0 17 psdc
30701 sybhrms 0.1 1.4G 1.8G 0.015 0.007 0 0 0 0 dataserver
21561 root 0.1 3.8M 5.6M 0.010 0.028 0 0 0 57 collect
21737 root 0.1 131K 1.7M 0.000 0.000 0 0 0 2 sleep
18097 sybhrms 0.0 1.5G 1.8G 0.001 0.001 0 0 0 0 dataserver
11131 root 0.0 3.8M 5.6M 0.001 0.006 0 0 0 8 collect
1292 root 0.0 909K 3.5M 0.000 0.000 0 0 0 0 nsrmmd
31540 pshrms 0.0 4.8M 9M 0.006 0.001 0 0 0 0 PSRUN
1161 root 0.0 2.2M 6.2M 0.000 0.000 0 0 0 0 advfsd
31440 pshrms 0.0 28M 36M 0.001 0.000 0 0 0 0 PSAPPSRV
12955 pshrms 0.0 25M 34M 0.002 0.001 0 1 0 0 PSAPPSRV
31429 pshrms 0.0 26M 35M 0.001 0.000 0 0 0 0 PSAPPSRV
11754 root 0.0 270K 2.6M 0.000 0.000 0 0 0 0 os_mibs
34 root 0.0 90K 1.7M 0.000 0.000 0 0 0 0 update
1117 root 0.0 2.6M 6.9M 0.001 0.000 0 0 0 0 insightd
1270 root 0.0 5.3M 7.7M 0.000 0.000 0 0 0 0 nsrmmdbd
11706 root 0.0 180K 1.9M 0.000 0.000 0 0 0 0 snmpd
1317 root 0.0 4.5M 8.2M 0.000 0.000 0 0 0 0 Xdec
31427 pshrms 0.0 10M 14M 0.000 0.000 0 0 0 0 WSH
31425 pshrms 0.0 8.8M 12M 0.003 0.000 0 0 0 0 WSH
30930 sybhrms 0.0 10M 15M 0.000 0.000 0 0 0 0 backupserver
31426 pshrms 0.0 5.0M 8.1M 0.000 0.000 0 0 0 0 WSH
1 root 0.0 49K 507K 0.000 0.001 0 0 0 0 init
17334 root 0.0 5.4M 7.5M 0.000 0.000 0 0 0 0 sys_check
1261 root 0.0 1.6M 4.6M 0.000 0.000 0 0 0 0 nsrd
31441 pshrms 0.0 4.8M 13M 0.000 0.000 0 0 0 0 PSSAMSRV
13232 pshrms 0.0 4.9M 13M 0.000 0.000 0 0 0 0 PSSAMSRV
31318 pshrms 0.0 1.8M 11M 0.000 0.000 0 0 0 0 PSAUTH
9859 root 0.0 237K 4.8M 0.000 0.000 0 0 0 0 sshd
1123 root 0.0 417K 2.6M 0.000 0.000 0 0 0 0 pmgrd
1278 root 0.0 1.4M 4.5M 0.000 0.000 0 0 0 0 nsrindexd
31442 pshrms 0.0 1.9M 11M 0.000 0.000 0 0 0 0 PSAPISRV
31233 root 0.0 1.0M 5.0M 0.000 0.000 0 0 0 0 p_ctmat
31290 pshrms 0.0 1.6M 5.5M 0.000 0.000 0 0 0 0 BBL
1210 root 0.0 245K 3.5M 0.000 0.000 0 0 0 0 cron
31424 pshrms 0.0 802K 4.0M 0.000 0.000 0 0 0 0 WSL
11418 root 0.0 573K 5.0M 0.000 0.000 0 0 0 0 sshd
11761 root 0.0 663K 3.0M 0.000 0.000 0 0 0 0 cpq_mibs
189 root 0.0 155K 1.8M 0.000 0.000 0 0 0 0 syslogd
11759 root 0.0 172K 2.4M 0.000 0.000 0 0 0 0 svrSystem_mib
1242 root 0.0 516K 2.4M 0.000 0.000 0 0 0 0 lpd
1229 root 0.0 352K 2.2M 0.000 0.000 0 0 0 0 lpd
1177 root 0.0 131K 1.8M 0.000 0.000 0 0 0 0 inetd
1277 root 0.0 532K 2.9M 0.000 0.000 0 0 0 0 nsrexecd
11760 root 0.0 155K 2.4M 0.000 0.000 0 0 0 0 svrMgt_mib
20310 root 0.0 614K 4.9M 0.000 0.000 0 0 0 0 sshd
17343 root 0.0 614K 4.9M 0.000 0.000 0 0 0 0 sshd
1065 root 0.0 180K 2.4M 0.000 0.000 0 0 0 0 sendmail
379 root 0.0 155K 2.3M 0.000 0.000 0 0 0 0 portmap
1297 root 0.0 245K 4.5M 0.000 0.000 0 0 0 0 dtlogin
17292 root 0.0 638K 5.0M 0.000 0.000 0 0 0 0 sshd
1276 root 0.0 262K 2.8M 0.000 0.000 0 0 0 0 nsrexecd
16348 root 0.0 1.4M 9M 0.000 0.000 0 0 0 0 dtgreet
3 root 0.0 475K 1.0M 0.000 0.000 0 0 0 0 kloadsrv
11419 klee 0.0 294K 2.1M 0.000 0.000 0 0 0 0 ksh
17299 root 0.0 294K 2.1M 0.000 0.000 0 0 0 0 ksh
17350 root 0.0 294K 2.1M 0.000 0.000 0 0 0 0 ksh
20354 root 0.0 294K 2.1M 0.000 0.000 0 0 0 0 ksh
16349 root 0.0 770K 6.5M 0.000 0.000 0 0 0 0 dxconsole
193 root 0.0 106K 2.8M 0.000 0.000 0 0 0 0 binlogd
17274 mdurrani 0.0 294K 2.1M 0.000 0.000 0 0 0 0 ksh
1316 root 0.0 106K 483K 0.000 0.000 0 0 0 0 getty
20311 mdurrani 0.0 294K 2.1M 0.000 0.000 0 0 0 0 ksh
17346 mdurrani 0.0 294K 2.1M 0.000 0.000 0 0 0 0 ksh
16331 root 0.0 466K 5.8M 0.000 0.000 0 0 0 0 dtlogin
382 root 0.0 311K 2.5M 0.000 0.000 0 0 0 0 mountd
386 root 0.0 196K 3.4M 0.000 0.000 0 0 0 0 proplistd
1313 root 0.0 90K 2.9M 0.000 0.000 0 0 0 0 psdc
1226 root 0.0 65K 1.9M 0.000 0.000 0 0 0 0 lpd
20315 mdurrani 0.0 303K 2.1M 0.000 0.000 0 0 0 0 ksh
17298 mdurrani 0.0 303K 2.1M 0.000 0.000 0 0 0 0 ksh
393 root 0.0 147K 2.5M 0.000 0.000 0 0 0 0 rpc.lockd
17345 mdurrani 0.0 303K 2.1M 0.000 0.000 0 0 0 0 ksh
391 root 0.0 73K 2.4M 0.000 0.000 0 0 0 0 rpc.statd
18095 sybhrms 0.0 196K 2.3M 0.000 0.000 0 0 0 0 RUN_A3PSPHR
30929 sybhrms 0.0 155K 2.3M 0.000 0.000 0 0 0 0 RUN_A3PSPHR_back
384 root 0.0 0K 1.8M 0.000 0.000 0 0 0 0 nfsd
21696 root 0.0 139K 1.7M 0.000 0.000 0 0 0 3 adm_lk
21704 root 0.0 1.0M 7.5M 0.000 0.000 0 0 0 1 sys_check
388 root 0.0 65K 1.7M 0.000 0.000 0 0 0 0 nfsiod

/>> vmstat -P

Total Physical Memory = 6144.00 M
= 786432 pages


Physical Memory Clusters:

start_pfn end_pfn type size_pages / size_bytes
0 256 pal 256 / 2.00M
256 655360 os 655104 / 5118.00M
786432 917491 os 131059 / 1023.90M
917491 917504 pal 13 / 104.00k

Physical Memory Use:

start_pfn end_pfn type size_pages / size_bytes
256 266 unixtable 10 / 80.00k
266 284 scavenge 18 / 144.00k
284 705 text 421 / 3.29M
705 790 data 85 / 680.00k
790 915 bss 125 / 1000.00k
915 920 cfgmgmt 5 / 40.00k
920 921 locks 1 / 8.00k
921 2786 unixtable 1865 / 14.57M
2786 2799 pmap 13 / 104.00k
2799 7295 vmtables 4496 / 35.13M
7295 262144 managed 254849 / 1991.01M
262144 266177 vmtables 4033 / 31.51M
266177 524288 managed 258111 / 2016.49M
524288 526305 vmtables 2017 / 15.76M
526305 655360 managed 129055 / 1008.24M
*
786432 788449 vmtables 2017 / 15.76M
788449 917491 managed 129042 / 1008.14M
============================
Total Physical Memory Use: 786163 / 6141.90M

Managed Pages Break Down:

free pages = 26449
active pages = 86228
inactive pages = 128442
wired pages = 79136
ubc pages = 450830
==================
Total = 771085

WIRED Pages Break Down:

vm wired pages = 4549
ubc wired pages = 34893
meta data pages = 31446
malloc pages = 5840
contig pages = 648
user ptepages = 1503
kernel ptepages = 240
free ptepages = 9
==================
Total = 79128
/>> ipcs -a

Message Queues:
T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME
q 0 0x416e018c --rw------- root system root system 0 0 16384 3 23267 17:44:11 17:44:11 18:35:58
q 2 0xb6a4 -Rrw-rw---- pshrms psoft pshrms psoft 0 0 16384 0 0 no-entry no-entry 18:51:33
q 3 0 -Rrw-rw---- pshrms psoft pshrms psoft 0 0 16384 31427 31318 17:39:12 17:39:12 18:51:34
q 4 0 --rw-rw---- pshrms psoft pshrms psoft 0 0 16384 0 0 no-entry no-entry 18:51:34
q 5 0 -Rrw-rw---- pshrms psoft pshrms psoft 0 0 16384 0 0 no-entry no-entry 18:51:41
q 6 0 -Rrw-rw---- pshrms psoft pshrms psoft 0 0 16384 31429 31425 17:44:15 17:44:15 18:51:41
q 7 0 -Rrw-rw---- pshrms psoft pshrms psoft 0 0 16384 31440 31426 17:44:24 17:44:24 18:51:42
q 8 0 -Rrw-rw---- pshrms psoft pshrms psoft 0 0 16384 12955 31427 17:39:20 17:39:20 18:51:42
q 9 0 -Rrw-rw---- pshrms psoft pshrms psoft 0 0 16384 31426 31440 17:44:17 17:44:17 18:51:42
q 10 0 --rw-rw---- pshrms psoft pshrms psoft 0 0 16384 0 0 no-entry no-entry 18:51:42
q 139 0 -Rrw-rw---- pshrms psoft pshrms psoft 0 0 16384 31425 13232 17:42:28 17:42:28 18:51:49
q 12 0 --rw-rw---- pshrms psoft pshrms psoft 0 0 16384 0 0 no-entry no-entry 18:51:44
q 13 0 --rw-rw---- pshrms psoft pshrms psoft 0 0 16384 0 0 no-entry no-entry 18:51:47
q 14 0 --rw-rw---- pshrms psoft pshrms psoft 0 0 16384 31426 31441 17:17:13 17:17:13 18:51:49
q 15 0 -Rrw-rw---- pshrms psoft pshrms psoft 0 0 16384 0 0 no-entry no-entry 18:51:51
q 16 0 --rw-rw---- pshrms psoft pshrms psoft 0 0 16384 0 0 no-entry no-entry 18:51:51
q 82 0 --rw-rw---- pshrms psoft pshrms psoft 0 0 16384 31425 13232 17:42:29 17:42:29 23:20:12

Shared Memory:
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
m 0 0x3253bc5c --rw-rw-rw- root system root system 2 648 1088 11754 20:05:14 20:05:14 18:37:13
m 1 0xc8c11552 --rw------- sybhrms sybase sybhrms sybase 3 1818189824 18097 30701 18:50:13 18:50:13 18:47:31
m 2 0xb6a4 --rw-rw---- pshrms psoft pshrms psoft 12 301956 31290 12954 23:20:12 23:20:13 18:51:31
m 3 0 --rw------- pshrms psoft pshrms psoft 4 1096 31424 31427 18:51:42 18:51:42 18:51:41
m 4 0x616e0869 --rw------- root daemon root daemon 0 46128 31907 31907 18:53:32 18:53:33 18:53:32

Semaphores:
T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME
s 0 0x416e018c --ra------- root system root system 1 18:35:58 18:35:58
s 1 0xb6a4 --ra-ra---- pshrms psoft pshrms psoft 3 17:44:34 18:51:31
s 18 0 --ra-ra---- pshrms psoft pshrms psoft 24 15:47:48 18:51:31
s 3 0 --ra-ra---- pshrms psoft pshrms psoft 24 no-entry 18:51:31
s 4 0x616e0869 --ra------- root daemon root daemon 1 18:53:33 18:53:32



/>> sysconfig -q vm
vm:
ubc-minpercent = 10
ubc-maxpercent = 100
ubc-borrowpercent = 20
ubc-maxdirtywrites = 5
vm-max-wrpgio-kluster = 32768
vm-max-rdpgio-kluster = 16384
vm-cowfaults = 4
vm-mapentries = 16384
vm-maxvas = 1879048196
vm-maxwire = 16777216
vm-heappercent = 7
vm-vpagemax = 229376
vm-segmentation = 1
vm-ubcpagesteal = 24
vm-ubcdirtypercent = 10
vm-ubcseqstartpercent = 10
vm-ubcseqpercent = 10
vm-csubmapsize = 1048576
vm-ubcbuffers = 256
vm-syncswapbuffers = 128
vm-asyncswapbuffers = 4
vm-clustermap = 1048576
vm-clustersize = 65536
vm-zone_size = 0
vm-kentry_zone_size = 16777216
vm-syswiredpercent = 80
vm-inswappedmin = 1
vm-page-free-target = 1828
vm-page-free-swap = 924
vm-page-free-hardswap = 29248
vm-page-free-min = 20
vm-page-free-reserved = 10
vm-page-free-optimal = 924
vm-page-prewrite-target = 3656
dump-user-pte-pages = 0
kernel-stack-guard-pages = 1
vm-min-kernel-address = 18446744065119617024
malloc-percpu-cache = 1
vm-aggressive-swap = 0
vm_page_color_private = 0
vm-map-index-count = 64
vm-map-index-rebalance = 128
vm-map-index-enabled = 1
vm-map-index-hiwat = 4
vm-map-index-lowat = 2
new-wire-method = 1
vm-segment-cache-max = 50
vm-page-lock-count = 64
gh-chunks = 0
gh-min-seg-size = 8388608
gh-fail-if-no-mem = 1
private-text = 0
private-cache-percent = 0
/>>

[Michael Schulte zur Sur]

Hi,
ubc-maxpercent = 100
This is too much. Reduce it to 70 or what Sybase suggests if at all.
greetings,
Michael

[kkdurrani]

Hi Michael,

I have changed the value to 70% from dxkerneltuner. Apart from the reboot does it need any recompile of the kernal?
since I am using 4.0F, I guess there noway I can do it dynamically.

Karim

[Michael Schulte zur Sur]

Hi Karim,
no kernel build is necessary.
The value is in /etc/sysconfigtab
greetings,

Michael