topic Re: MEMORY BOTTLENECK ------ URGENT in Operating System - HP-UX

MEMORY BOTTLENECK ------ URGENT

shridhar_1 — Thu, 01 Jun 2006 00:36:21 GMT

dear sirs

can i tune my kernel and free some system memory which can be used by my application.

any suggestions

# ./kmeminfo /stand/vmunix /dev/kmem
kmeminfo (5.03)
unix: /stand/vmunix 11.11 64bit PA2.0
core: /dev/kmem live
link: Thu Jan 12 06:16:38 IST 2006
boot: Tue May 30 05:51:29 2006
dump: Wed May 31 15:44:55 2006
nbpg: 4096 bytes

----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 1048064 4.0g 100%
Free memory = 3189 12.5m 0%
User processes = 780798 3.0g 74% details with -user
System = 260750 1018.6m 25%
Kernel = 208348 813.9m 20% kernel text and data
Dynamic Arenas = 109676 428.4m 10% details with -arena
M_TEMP = 77475 302.6m 7%
M_SWAP = 10480 40.9m 1%
M_SPINLOCK = 5720 22.3m 1%
VFD_BT_NODE = 3764 14.7m 0%
ALLOCB_MBLK_LM = 1786 7.0m 0%
Other arenas = 10451 40.8m 1% details with -arena
Super page pool = 28772 112.4m 3% details with -kas
Static Tables = 55656 217.4m 5% details with -static
pfdat = 23876 93.3m 2%
nbuf = 8784 34.3m 1% bufcache headers
htbl2_0 = 8192 32.0m 1%
pfn_to_virt = 3979 15.5m 0%
text = 2710 10.6m 0% vmunix text section
Other tables = 8114 31.7m 1% details with -static
Buffer cache = 52402 204.7m 5% details with -bufcache
#

thanx in adv

shridhar

Re: MEMORY BOTTLENECK ------ URGENT

Chauhan Amit — Thu, 01 Jun 2006 01:09:44 GMT

Hi Shridhar,

Looking at the output:

Physical memory = 4.0g 100%
Free memory = 12.5m 0%
User processes = 3.0g 74%
System = 1018.6m 25%

M_TEMP = 77475 302.6m 7%

Observations:
o User processes are occupying 74% of memory , Check which is the process and is this normal. Need top output to confirm the same...
o kernel is using 20% of the memory which is on the higher side.

Suggestions:
o You can tune kernel parameter "vx_ninode" ( If it is presently set to zero)to free up some memory which is being used up by "M_TEMP" , You can limit it to 100MB , that will free up 200MB of memory

o Permanent Solution would be to Increase the physical memory

Can you paste "kmtune" and "top" output to check further.

-Amit

Re: MEMORY BOTTLENECK ------ URGENT

RAC_1 — Thu, 01 Jun 2006 01:24:11 GMT

UNIX95= ps -ef -o vsz,pid,ppid,ruser,args | sort -nk1

Will tell you what processes are using most of the memory. Also glance -m and look for system and buffer and user memory. buffer cahe should be wisely used.

Re: MEMORY BOTTLENECK ------ URGENT

shridhar_1 — Thu, 01 Jun 2006 05:11:31 GMT

dear RAC

I do not have glance

UNIX95 command is giving error
-ef not found

we have sap r/3 with oracle 9.0.2.0

now there is lot of paging
and high io on root VG
from last 2 days

is it requied to tune kernel
i have 20 GB swap on root VG
thanx for your reply

shridhar

# ./kmeminfo -user /stand/vmunix /dev/kmem
kmeminfo (5.03)
unix: /stand/vmunix 11.11 64bit PA2.0
core: /dev/kmem live
link: Thu Jan 12 06:16:38 IST 2006
boot: Tue May 30 05:51:29 2006
dump: Wed May 31 15:46:04 2006
nbpg: 4096 bytes

----------------------------------------------------------------------
Summary of processes memory usage:

List sorted by physical size, in pages/bytes:

virtual physical swap
pid ppid pages / bytes pages / bytes pages / bytes command
13498 13467 1404063 5.4g 30324 118.5m 91925 359.1m disp+work
13514 13467 1397495 5.3g 28211 110.2m 85341 333.4m disp+work
13507 13467 1399608 5.3g 28144 109.9m 87564 342.0m disp+work
13528 13467 1393828 5.3g 27175 106.2m 81770 319.4m disp+work
13529 13467 1395872 5.3g 27076 105.8m 83819 327.4m disp+work
13519 13467 1395128 5.3g 27069 105.7m 83073 324.5m disp+work
13501 13467 1400931 5.3g 24938 97.4m 88785 346.8m disp+work
13494 13467 1403328 5.4g 24842 97.0m 91294 356.6m disp+work
13495 13467 1401695 5.3g 24814 96.9m 89551 349.8m disp+work
13532 13467 1390097 5.3g 24066 94.0m 80330 313.8m disp+work
13513 13467 1398647 5.3g 23601 92.2m 86496 337.9m disp+work
13506 13467 1400039 5.3g 23529 91.9m 87889 343.3m disp+work
13520 13467 1398936 5.3g 23315 91.1m 86891 339.4m disp+work
13533 13467 1387363 5.3g 22784 89.0m 77591 303.1m disp+work
13536 13467 1385091 5.3g 21800 85.2m 75313 294.2m disp+work
13558 13467 1382247 5.3g 21730 84.9m 72462 283.1m disp+work
13559 13467 1385639 5.3g 20976 81.9m 75862 296.3m disp+work
13546 13467 1381259 5.3g 19714 77.0m 71475 279.2m disp+work
13539 13467 1384323 5.3g 19710 77.0m 74543 291.2m disp+work
25188 13467 1385730 5.3g 18685 73.0m 73654 287.7m disp+work
6595 13467 1391074 5.3g 18665 72.9m 79011 308.6m disp+work
21214 13467 1399522 5.3g 14591 57.0m 87480 341.7m disp+work
6597 1 265259 1.0g 9432 36.8m 12650 49.4m oracle
13467 13438 1162517 4.4g 9414 36.8m 60741 237.3m disp+work
4784 1 263137 1.0g 9206 36.0m 10531 41.1m oracle
4786 1 263137 1.0g 8445 33.0m 10531 41.1m oracle
5008 1 263211 1.0g 8096 31.6m 10597 41.4m oracle
5013 1 262187 1.0g 8056 31.5m 9570 37.4m oracle
4839 1 265259 1.0g 8027 31.4m 12650 49.4m oracle
5004 1 262187 1.0g 8008 31.3m 9570 37.4m oracle
5006 1 262187 1.0g 7993 31.2m 9570 37.4m oracle
4819 1 265259 1.0g 7992 31.2m 12650 49.4m oracle
4817 1 265259 1.0g 7974 31.1m 12650 49.4m oracle
4904 1 265259 1.0g 7938 31.0m 12650 49.4m oracle
4906 1 265259 1.0g 7928 31.0m 12650 49.4m oracle
5011 1 263211 1.0g 7917 30.9m 10597 41.4m oracle
4939 1 262187 1.0g 7876 30.8m 9570 37.4m oracle
4950 1 262187 1.0g 7857 30.7m 9570 37.4m oracle
4902 1 263211 1.0g 7848 30.7m 10597 41.4m oracle
4900 1 265259 1.0g 7834 30.6m 12650 49.4m oracle
4952 1 262187 1.0g 7765 30.3m 9570 37.4m oracle
4936 1 262187 1.0g 7686 30.0m 9570 37.4m oracle
4837 1 261675 1022.2m 7631 29.8m 9057 35.4m oracle
4774 1 263137 1.0g 7584 29.6m 10531 41.1m oracle
4772 1 262059 1023.7m 7509 29.3m 9442 36.9m oracle
4776 1 265185 1.0g 7500 29.3m 12584 49.2m oracle
4778 1 263137 1.0g 7441 29.1m 10531 41.1m oracle
4782 1 261665 1022.1m 7381 28.8m 9055 35.4m oracle
21216 1 265259 1.0g 7379 28.8m 12650 49.4m oracle
4780 1 262177 1.0g 7377 28.8m 9568 37.4m oracle
4941 1 262187 1.0g 7377 28.8m 9570 37.4m oracle
25190 1 262187 1.0g 7377 28.8m 9570 37.4m oracle
4908 1 261611 1021.9m 7376 28.8m 8993 35.1m oracle
13493 13467 102836 401.7m 3414 13.3m 25075 97.9m icman
2416 2412 3325 13.0m 2933 11.5m 2418 9.4m cmcld
13492 13467 67849 265.0m 2196 8.6m 10095 39.4m gwrd
10377 1 4550 17.8m 1282 5.0m 3570 13.9m saposcol
2421 2419 1403 5.5m 735 2.9m 460 1.8m cmlvmd
10639 10630 8327 32.5m 697 2.7m 4419 17.3m igsmux_mt
2004 1 6042 23.6m 412 1.6m 3688 14.4m vxsvc
13590 13581 7615 29.7m 392 1.5m 3667 14.3m igsmux_mt
4832 1867 3916 15.3m 299 1.2m 1671 6.5m rep_server
13148 1 4983 19.5m 287 1.1m 846 3.3m tnslsnr
3446 764 2052 8.0m 234 936.0k 884 3.5m sshd
4840 1867 3242 12.7m 228 912.0k 1029 4.0m agdbserver
4841 4840 3346 13.1m 225 900.0k 1091 4.3m alarmgen
10641 10630 16927 66.1m 175 700.0k 4130 16.1m igspw_mt
13466 13438 3252 12.7m 158 632.0k 1455 5.7m msg_server
13469 13438 1261 4.9m 149 596.0k 349 1.4m rslgsend
13591 13581 16943 66.2m 149 596.0k 4147 16.2m igspw_mt
10640 10630 16947 66.2m 148 592.0k 4151 16.2m igspw_mt
13592 13581 16943 66.2m 146 584.0k 4147 16.2m igspw_mt
13468 13438 1264 4.9m 125 500.0k 349 1.4m rslgcoll
1881 1 2681 10.5m 120 480.0k 884 3.5m swagentd
764 1 1326 5.2m 116 464.0k 188 752.0k sshd
3483 1 2111 8.2m 106 424.0k 1469 5.7m fpl_em
5742 10377 475 1.9m 104 416.0k 85 340.0k sh
1253 1 3773 14.7m 102 408.0k 975 3.8m cimserver
1867 1 3009 11.8m 100 400.0k 770 3.0m perflbd
3525 1 937 3.7m 100 400.0k 278 1.1m ia64_corehw
3268 1 852 3.3m 94 376.0k 213 852.0k dm_stape
2375 1773 995 3.9m 93 372.0k 370 1.4m psmctd
1191 1 1384 5.4m 91 364.0k 297 1.2m mib2agt
1249 1 3125 12.2m 90 360.0k 1588 6.2m dced
1206 1 1051 4.1m 78 312.0k 139 556.0k cmsnmpd
1169 1 3111 12.2m 69 276.0k 640 2.5m snmpdm
5744 5742 435 1.7m 68 272.0k 78 312.0k lanadmin
2218 1 1376 5.4m 64 256.0k 263 1.0m mibmond
2191 1 842 3.3m 63 252.0k 282 1.1m psmmon
2262 1 1386 5.4m 62 248.0k 265 1.0m mibmond
2227 1 1376 5.4m 61 244.0k 263 1.0m mibmond
2145 1 537 2.1m 55 220.0k 97 388.0k p_client
1159 1 868 3.4m 53 212.0k 157 628.0k sendmail
1754 1 5479 21.4m 51 204.0k 1921 7.5m pdclientd
1746 1 4870 19.0m 51 204.0k 1606 6.3m basicdsd
1773 1 833 3.3m 47 188.0k 213 852.0k diagmond
1712 1 459 1.8m 46 184.0k 95 380.0k pwgrd
3507 3469 646 2.5m 46 184.0k 104 416.0k ksh
886 1 738 2.9m 44 176.0k 145 580.0k inetd
2419 2416 451 1.8m 40 160.0k 69 276.0k cmsrvassistd
1267 1 440 1.7m 40 160.0k 78 312.0k rbootd
2373 1773 742 2.9m 37 148.0k 267 1.0m diaglogd
2146 1 908 3.5m 33 132.0k 162 648.0k cimserverd
2129 1 426 1.7m 32 128.0k 69 276.0k prngd
863 1 1407 5.5m 29 116.0k 852 3.3m automountd
3747 1 844 3.3m 29 116.0k 207 828.0k sysstat_em
618 617 945 3.7m 29 116.0k 251 1004.0k netfmt
2901 1 916 3.6m 28 112.0k 267 1.0m disk_em
617 1 438 1.7m 27 108.0k 86 344.0k ntl_reader
494 1 487 1.9m 25 100.0k 115 460.0k syncer
1272 1 1093 4.3m 25 100.0k 508 2.0m xfs
4406 886 419 1.6m 24 96.0k 66 264.0k tftpd
2133 1 82 328.0k 22 88.0k 33 132.0k krsd
3126 1 933 3.6m 21 84.0k 249 996.0k dm_core_hw
2420 2419 449 1.8m 20 80.0k 69 276.0k cmlogd
2382 886 570 2.2m 20 80.0k 127 508.0k registrar
23730 18023 642 2.5m 19 76.0k 100 400.0k ksh
3662 1 1076 4.2m 19 76.0k 300 1.2m lpmc_em
3189 1 832 3.2m 18 72.0k 203 812.0k dm_memory
1182 1 1158 4.5m 17 68.0k 182 728.0k hp_unixagt
607 1 420 1.6m 12 48.0k 66 264.0k nktl_daemon
582 1 428 1.7m 12 48.0k 67 268.0k syslogd
585 1 491 1.9m 12 48.0k 73 292.0k ptydaemon
1776 1 419 1.6m 11 44.0k 69 276.0k envd
2374 1773 915 3.6m 11 44.0k 298 1.2m memlogd
2062 2039 677 2.6m 11 44.0k 153 612.0k nfsd
2055 2038 677 2.6m 11 44.0k 153 612.0k nfsd
2061 2038 677 2.6m 11 44.0k 153 612.0k nfsd
2060 2039 677 2.6m 11 44.0k 153 612.0k nfsd
2065 2038 677 2.6m 11 44.0k 153 612.0k nfsd
2059 2038 677 2.6m 11 44.0k 153 612.0k nfsd
2058 2038 677 2.6m 11 44.0k 153 612.0k nfsd
2057 2038 677 2.6m 11 44.0k 153 612.0k nfsd
2066 2039 677 2.6m 11 44.0k 153 612.0k nfsd
2056 2038 677 2.6m 11 44.0k 153 612.0k nfsd
5676 3525 8 32.0k 8 32.0k 9 36.0k ia64_corehw
5737 10377 8 32.0k 8 32.0k 9 36.0k sh
1822 1 798 3.1m 8 32.0k 246 984.0k ttd
824 1 8 32.0k 8 32.0k 9 36.0k biod
825 1 8 32.0k 8 32.0k 9 36.0k biod
826 1 8 32.0k 8 32.0k 9 36.0k biod
827 1 8 32.0k 8 32.0k 9 36.0k biod
828 1 8 32.0k 8 32.0k 9 36.0k biod
829 1 8 32.0k 8 32.0k 9 36.0k biod
830 1 8 32.0k 8 32.0k 9 36.0k biod
831 1 8 32.0k 8 32.0k 9 36.0k biod
832 1 8 32.0k 8 32.0k 9 36.0k biod
833 1 8 32.0k 8 32.0k 9 36.0k biod
834 1 8 32.0k 8 32.0k 9 36.0k biod
835 1 8 32.0k 8 32.0k 9 36.0k biod
836 1 8 32.0k 8 32.0k 9 36.0k biod
837 1 8 32.0k 8 32.0k 9 36.0k biod
823 1 8 32.0k 8 32.0k 9 36.0k biod
822 1 8 32.0k 8 32.0k 9 36.0k biod
821 1 8 32.0k 8 32.0k 9 36.0k biod
808 1 8 32.0k 8 32.0k 9 36.0k biod
809 1 8 32.0k 8 32.0k 9 36.0k biod
810 1 8 32.0k 8 32.0k 9 36.0k biod
811 1 8 32.0k 8 32.0k 9 36.0k biod
812 1 8 32.0k 8 32.0k 9 36.0k biod
813 1 8 32.0k 8 32.0k 9 36.0k biod
2119 1 136 544.0k 8 32.0k 48 192.0k dtrc
814 1 8 32.0k 8 32.0k 9 36.0k biod
815 1 8 32.0k 8 32.0k 9 36.0k biod
816 1 8 32.0k 8 32.0k 9 36.0k biod
817 1 8 32.0k 8 32.0k 9 36.0k biod
818 1 8 32.0k 8 32.0k 9 36.0k biod
819 1 8 32.0k 8 32.0k 9 36.0k biod
820 1 8 32.0k 8 32.0k 9 36.0k biod
838 1 8 32.0k 8 32.0k 9 36.0k biod
2424 886 1150 4.5m 7 28.0k 154 616.0k cmclconfd
2412 886 1162 4.5m 7 28.0k 166 664.0k cmclconfd
2418 886 1158 4.5m 7 28.0k 162 648.0k cmclconfd
10630 1 4247 16.6m 6 24.0k 1535 6.0m igswd_mt
1200 1 1133 4.4m 6 24.0k 179 716.0k trapdestagt
13438 1 1222 4.8m 6 24.0k 339 1.3m sapstart
13581 1 4247 16.6m 6 24.0k 1535 6.0m igswd_mt
2064 2038 677 2.6m 5 20.0k 153 612.0k nfsd
847 1 864 3.4m 5 20.0k 339 1.3m rpc.statd
1234 1 498 1.9m 5 20.0k 95 380.0k fddi4subagt
2063 2038 677 2.6m 5 20.0k 153 612.0k nfsd
2147 1 413 1.6m 5 20.0k 61 244.0k getty
2144 2119 873 3.4m 4 16.0k 122 488.0k dtlogin
1758 1 499 1.9m 4 16.0k 77 308.0k cron
853 1 863 3.4m 3 12.0k 323 1.3m rpc.lockd
2422 2419 641 2.5m 3 12.0k 84 336.0k cmtaped
784 1 782 3.1m 3 12.0k 260 1.0m rpcbind
2067 2038 677 2.6m 3 12.0k 153 612.0k nfsd
2072 2039 677 2.6m 3 12.0k 153 612.0k nfsd
2073 2039 677 2.6m 3 12.0k 153 612.0k nfsd
8104 886 1829 7.1m 3 12.0k 449 1.8m rpc.ttdbserver
2074 2039 677 2.6m 3 12.0k 153 612.0k nfsd
2054 2038 677 2.6m 3 12.0k 153 612.0k nfsd
2049 2038 677 2.6m 3 12.0k 153 612.0k nfsd
2051 2039 677 2.6m 3 12.0k 153 612.0k nfsd
2043 2038 677 2.6m 3 12.0k 153 612.0k nfsd
2045 2039 677 2.6m 3 12.0k 153 612.0k nfsd
2071 2039 677 2.6m 3 12.0k 153 612.0k nfsd
2022 1 799 3.1m 3 12.0k 278 1.1m rpc.mountd
2070 2039 677 2.6m 3 12.0k 153 612.0k nfsd
1921 1 461 1.8m 3 12.0k 80 320.0k emsagent
2044 2038 677 2.6m 3 12.0k 153 612.0k nfsd
2037 1 645 2.5m 3 12.0k 121 484.0k nfsd
2038 1 677 2.6m 3 12.0k 153 612.0k nfsd
2039 2038 677 2.6m 3 12.0k 153 612.0k nfsd
2040 2038 677 2.6m 3 12.0k 153 612.0k nfsd
2041 2039 677 2.6m 3 12.0k 153 612.0k nfsd
2042 2039 677 2.6m 3 12.0k 153 612.0k nfsd
2048 2039 677 2.6m 3 12.0k 153 612.0k nfsd
2069 2039 677 2.6m 3 12.0k 153 612.0k nfsd
2053 2039 677 2.6m 3 12.0k 153 612.0k nfsd
1782 1 404 1.6m 3 12.0k 55 220.0k hubd
2134 1 82 328.0k 2 8.0k 33 132.0k sfd
3469 3446 140 560.0k 0 0.0k 52 208.0k sh
18023 1 140 560.0k 0 0.0k 52 208.0k sh
physical swap
pages / bytes pages / bytes
Total: 778507 3.0g 2300701 8.8g
#

Re: MEMORY BOTTLENECK ------ URGENT

Pete Randall — Thu, 01 Jun 2006 05:24:46 GMT

The "-ef: not found" error is because you left out the space between the equal sign and "ps".

UNIX95= ps -ef -o vsz,pid,ppid,ruser,args | sort -nk1

Pete

Re: MEMORY BOTTLENECK ------ URGENT

Bill Hassell — Thu, 01 Jun 2006 07:00:55 GMT

It appears that you have just 4Gb of RAM but dozens of large processes running. Assuming that these processes aree working correctly, there is nothing you can do in the kernel to reduce memory usage except to kill some of those big processes (not recommended unless they are not working correctly). You need another 4Gb of RAM to properly size your server for this workload.

Re: MEMORY BOTTLENECK ------ URGENT

shridhar_1 — Thu, 01 Jun 2006 23:52:01 GMT

dear bill

thanks for the reply

the max memory i can extend is 2 GB
as i have RP 3410 which supports
only 6GB max

I m planning to increase ram
which will require some time

so i was intrested to tune oracle /sap
for better performance

how sap uses memory
any suggestions

regards

shridhar

Re: MEMORY BOTTLENECK ------ URGENT

Bill Hassell — Sat, 03 Jun 2006 22:11:00 GMT

The only thing you can do is to reduce the number of users and processes running a4t the same time. 4Gb is adequate for single instance of Oracle but add SAP and even 6Gb may be too small. You can reconfigure SAP and Pracle to use less space in the shared memory areas but this will usually make the applications run slowwer.

Re: MEMORY BOTTLENECK ------ URGENT

Hein van den Heuvel — Sat, 03 Jun 2006 22:45:19 GMT

shridhar,

The kernel memory use seems a bit much relatively speaking, so you shoudl try to nickle and dime that somewhat.

Reduce NPROC / NFILE as much as possible.
Reduce filesytem params as much as possible as SAP/Oracle systems tend to use little file system activity and do their own caching.

What's with all those NFS/IO deamons.
It this an SAP box or a file server?!
Tune accordingly.

Bu mostly you should review both Oracle and SAP setting to try and understand whether the memory given to those is reasonable.

Both have good stats to help you with.
For example, Oracle STATSPACK has a BUFFER POOL and SHARED POOL sizing estimate table, indicating the cost/benefit of reducing/increasing each. by step of 10% or so.

Maybe you have a 700 MB Oracle buffer pool and reducing to 50 MB is only expect to increase IO by 10%? Ditto for the Oracle shared pool. Several SPA applications have a relatively low number of fixed queries and thus need not too much shaed pool.

For SAP, have a look at ST02 output.
What is the "Max Use" reported under "Extended Memory"? Revisit tuning of em/initial_size_MB and the likes.
The GBs virtual memory used vs physical availble is a little scary but you are surviving, so maybe some buffers are simply oversized. Less SAP extended memory, and more ROLL/PF ?

But perhaps more importantly revisit the big SAP knobs: rdisp/wp_no_dia, rdisp/wp_no_btc, rdisp/wp_no_vb.
It looks like you have a good few disp+work's running. I have seen those oversize all too often, and with that you also get too many Oracle slaves of course.
The reasoning sometimes is: Hmm, our dialogue response time is low, lets add a few more dialogue servers. Well, if the system was simply over-loaded, or undersized, then adding more servers will only increase contention and memory use causing a slow down from a systm wide perspective.

Please don't try to solve this from a system admin perspective only. Please work with your SAP / Oracle staff to evaluate load and tuning opportunities.

Good luck,
Hein.
HvdH Performance Consulting.