Memory usage discrepancy

[karthikbalu]

Please help me understand where my memory has been consumed?

I have 16GB of physical memory and the OS is 11iv2.  From basic analysis i can see  ~7gb of memory has been consumed by my app ad OS. But i wonder where the memory is taking... is that chache? buffer? how this can be analyzed?

ProcList CPU Rpt  Mem Rpt  Disk Rpt            NextKeys SlctProc   Help     Exit

 Glance C.04.70.000             08:52:36            ia64       Current  Avg  High

-----------------------------------------------------------------------------------

CPU  Util                                                                    |  0%    1%    3%

Disk Util   F                                                               |  2%    2%    3%

Mem  Util   S                 SU                        UB     B  | 99%   99%   99%

Networkil   U             UR                R                      | 62%   62%   62%

-----------------------------------------------------------------------------------

                                   PROCESS LIST                        Users=    1

                         User      CPU %    Thrd Disk        Memory       Block

Process Name     PID     Name   ( 400% max) Cnt  IOrate      RSS/VSS      On

--------------------------------------------------------------------------------

siebmtsh                 2987 sadmin         0.0    28   0.0   930.8mb  980.9mb SLEEP

siebmtshmw         13101 sadmin        0.0    33   0.0   756.9mb  800.6mb SLEEP

siebmtshmw         10176 sadmin        0.0    21   0.1   715.8mb  784.6mb SLEEP

siebprocmw          2912 sadmin           0.0     8   0.0   637.0mb  666.4mb OTHER

siebmtshmw          2942 sadmin         0.0    33   0.0   446.3mb  488.5mb SLEEP

siebmtshmw          3055 sadmin         0.0    30   0.0   310.8mb  387.1mb SLEEP

siebmtshmw          3081 sadmin         0.0    26   0.0   292.9mb  374.4mb SLEEP

siebmtshmw          3121 sadmin         0.0    63   0.0   273.8mb  330.9mb SLEEP

siebmtshmw          3041 sadmin         0.0    13   0.0   269.9mb  316.6mb SLEEP

siebmtshmw          3098 sadmin        0.0    42   0.0   268.6mb  330.1mb OTHER

siebprocmw          2901 sadmin        0.0     8   0.0   268.0mb  297.5mb SLEEP

siebmtshmw          3070 sadmin        0.0    26   0.0   257.9mb  333.2mb SLEEP

siebmtshmw          3129 sadmin        0.0    62   0.0   255.1mb  306.2mb SLEEP

siebmtshmw          2956 sadmin        0.0    32   0.0   244.0mb  288.9mb SLEEP

siebmtshmw          3103 sadmin        0.0    32   0.0   243.5mb  288.6mb SLEEP

siebmtsh                 2902 sadmin        0.0    10   0.0   223.1mb  252.3mb OTHER

siebmtshmw          2922 sadmin        0.0    15   0.0   187.9mb  229.8mb SLEEP

siebmtshmw          2923 sadmin        0.0    13   0.0   187.1mb  228.2mb SLEEP

siebmtsh                 2911 sadmin        0.0    28   0.0   161.6mb  186.6mb SLEEP

siebprocmw          3020 sadmin         0.0     7   0.0   152.2mb  378.7mb OTHER

siebmtsh                2836 sadmin         0.0    54   0.0    90.0mb  143.6mb OTHER

vxsvc                      2332  root              0.0    18   0.0    54.3mb   84.2mb SLEEP

siebmtshmw          3029 sadmin        0.0    16   0.0    34.5mb   63.4mb OTHER

ProcList CPU Rpt  Mem Rpt  Disk Rpt            NextKeys SlctProc   Help     Exit

###############################

:#./kmeminfo

tool: kmeminfo 8.00 - libp4 9.306 - HP CONFIDENTIAL

unix: /stand/current/vmunix 11.23 64bit IA64 on host "xxxxxxx"

core: /dev/kmem live

link: Sun Feb 20 15:19:30 GMT 2011

boot: Thu Nov 28 14:27:43 2013

time: Tue Feb 18 08:54:54 2014

nbpg: 4096 bytes

----------------------------------------------------------------------

Physical memory usage summary (in page/byte/percent):

Physical memory          =  4186552   16.0g 100%

Free memory                 =    41978  164.0m   1%

User processes              =  1991841    7.6g  48%  details with -user

System                             =  2133805    8.1g  51%

  Kernel                            =  1542320    5.9g  37%  kernel text and data

    Dynamic Arenas        =  1089010    4.2g  26%  details with -arena

      M_TEMP                    =   920887    3.5g  22%

      vx_global_kmcac   =    37664  147.1m   1%

      spinlock                     =    28923  113.0m   1%

      vm_pfn2v_arena  =    16630   65.0m   0%

      VFD_BT_NODE       =    11096   43.3m   0%

      Other arenas           =    73810  288.3m   2%  details with -arena

    Super page pool       =   152101  594.1m   4%  details with -kas

    Static Tables             =   228158  891.2m   5%  details with -static

      pfdat                         =    98122  383.3m   2%

      nbuf                          =    66304  259.0m   2%  bufcache headers

      vhpt                          =    32768  128.0m   1%

      text                           =     8226   32.1m   0%  vmunix text section

      bufhash                    =     8192   32.0m   0%  bufcache hash headers

      Other tables           =    14545   56.8m   0%  details with -static

  Buffer cache               =   591485    2.3g  14%  details with -bufcache

  UFC meta mrg            =        0     0.0   0%

  UFC file mrg                =        0     0.0   0%

#ipcs -mob

IPC status from /dev/kmem as of Tue Feb 18 08:55:00 2014

T            ID               KEY        MODE        OWNER     GROUP NATTCH      SEGSZ

Shared Memory:

m          0          0x411c09a8 --rw-rw-rw-      root      root      0        348

m          1           0x4e0c0002 --rw-rw-rw-      root      root      2      61760

m          2           0x41200084 --rw-rw-rw-      root      root      2       8192

m       32774  0x06347849 --rw-rw-rw-      root       sys      1      65544

m          7           0x491404b0 --rw-r--r--      root      root      0      22908

[Dennis Handly]

Your kmeminfo output has the breakdown of where system memory is allocated.

The lines with "details with" show the options to get more details.

[karthikbalu]

Any idea? How to tune this area's??

System                        =  2153362    8.2g  51%

  Kernel                        =  1575271    6.0g  38%  kernel text and data

    Dynamic Arenas    =  1245145    4.7g  30%  details with -arena

      M_TEMP                =   920883      3.5g  22%

[Bill Hassell]

Check your kernel parameter dbc_max_pct:

kctune -s dbc_max_pct
Tunable      Value  Expression  Changes
dbc_max_pct     10  10          Immed

 50% is the (very bad) default for the buffer cache.

[karthikbalu]

This is what i have for the buffer cache. This value was 20% and i reduced to 15, but still  memory is busy.

#kctune -s dbc_max_pct
Tunable Value Expression Changes
dbc_max_pct 15 15 Immed

I suspect a memory leak here. Also i wonder how to tune the M_TEMP which is showing on 'kmeminfo'.

System = 2095248 8.0g 50%
Kernel = 1543670 5.9g 37% kernel text and data
   Dynamic Arenas = 1086737 4.1g 26% details with -arena
         M_TEMP = 920890 3.5g 22%

[Emil Velez_2]

Looks like you have a lot of user processes with large amounts of resident memory. Suspect applications allocating memory and either needing it or not freeing it when it does not.

[Dennis Handly]

>a lot of user processes with large amounts of resident memory.

But karthikbalu was worried about the 8.1g used by "system".

[karthikbalu · Accepted solution]

Issue resolved after installing the below patch and tuning the below param.

PHKL_32425 has been released which addresses MEMORY_LEAK.

PHKL_41807 s700_800 11.23 mmap,madvise,mlock,msync, read corruption patch.

kctune vxfs_ifree_timelag=-1