90GB of sys mem usage after clean boot

[Vince Pooley]

Hi,

I've been presented with this server suffering from a system memory usage so high that the installed database software can't start up. The server has 96GB of physical memory and the system is using  89GB of it after a cold boot.

I've had a look at the tuneables and there's nothing I can see that is out of order and there are no services running after a fresh boot. I've brought down maxdsiz, maxdsiz_64 & base_pagesize back to default but after a reboot, no difference.

Here's what glance shows for memory stats,

# glance

 Glance 11.00.044               11:26:44 xxxxx016     ia64                                                                          Current  Avg  High
------------------------------------------------------------------------------------------------------------------------------------------------------
CPU  Util                                                                                                                            |  1%    1%    3%
Disk Util                                                                                                                            |  0%    1%   17%
Mem  Util   S                                                                                                              SUF       | 95%   95%   95%
Swap Util   U                                                                                                    URR                 | 87%   87%   87%
------------------------------------------------------------------------------------------------------------------------------------------------------
                                                                    MEMORY REPORT                                                         Users=    3
Event         Current   Cumulative   Current Rate   **bleep** Rate   High Rate
--------------------------------------------------------------------------------
Page Faults       120         1910        80.0       67.0       682.8
Page In            38          265        25.3        9.2       130.0
Page Out            0            0         0.0        0.0         0.0
KB Paged In       8kb        1.0mb         5.3       37.1      3154.0
KB Paged Out      0kb          0kb         0.0        0.0         0.0
Reactivations       0            0         0.0        0.0         0.0
Deactivations       0            0         0.0        0.0         0.0
KB Deactivated    0kb          0kb         0.0        0.0         0.0
VM Reads            0            0         0.0        0.0         0.0
VM Writes           0            0         0.0        0.0         0.0
Total VM :   1.7gb   Sys Mem  :  89.6gb   User Mem:   1.3gb   Phys Mem :  96.0gb
Active VM:   1.1gb   Buf Cache:     0mb   Free Mem:   5.0gb   FileCache:   935mb
MemFS Blk Cnt:                0   MemFS Swp Cnt:                0

... and swapinfo-tam

                 Mb          Mb           Mb   PCT      START/          Mb
TYPE          AVAIL        USED         FREE  USED       LIMIT     RESERVE  PRI  NAME
dev            8192           0         8192    0%           0           -    1  /dev/vg00/lvol2
reserve           -        1511        -1511
memory        93472       86667         6805   93%
total        101664       88178        13486   87%       -           0    -

... and the top ten processes by mem usage

# UNIX95=1 ps -e -o vsz,pid,ppid,args | sort -rn | head -10

 238016 19659     1 /opt/perf/bin/scopeux
 233212 19748     1 /opt/perf/bin/perfd
 231672 19689     1 /opt/perf/bin/midaemon
 217960  4784  4776 /opt/wbem/lbin/cimprovagt 0 4 9 root SFMProviderModule
 199036 19703 19696 /opt/OV/lbin/perf/coda
 117268 19893 19696 /opt/OV/lbin/agtrep/agtrep -start
  97932 19862 19696 /opt/OV/lbin/eaagt/opcmona
  91172  4777  4776 cimservermain --executor-socket 4
  89528  4803  4776 /opt/wbem/lbin/cimprovagt 0 4 9 root HPUXStorageNativeProviderModule
  85636  4800  4776 /opt/wbem/lbin/cimprovagt 0 4 9 root HPUXFCNativeIndicationProviderModule

... and the output of kcusage

# kcusage

Tunable                 Usage / Setting      
=============================================
filecache_max       109944832 / 4900631961
maxdsiz              97189888 / 2147483647
maxdsiz_64bit        50593792 / 34359738368
maxfiles_lim              111 / 63488
maxssiz               1064960 / 134217728
maxssiz_64bit          131072 / 1073741824
maxtsiz               9797632 / 1073741824
maxtsiz_64bit         6119424 / 8589934592
maxuprc                    19 / 27000
max_thread_proc           101 / 3001
msgmbs                      0 / 8
msgmni                      9 / 30000
msgtql                      0 / 30000
nflocks                    43 / 60000
ninode                   1318 / 165888
nkthread                  906 / 250000
nproc                     264 / 30000
npty                        0 / 200
nstrpty                     3 / 200
nstrtel                     0 / 60
nswapdev                    1 / 32
nswapfs                     0 / 32
semmni                     45 / 32767
semmns                    167 / 120000
shmmax              199163840 / 4398046511104
shmmni                     23 / 30000
shmseg                      2 / 512

I'm stumped as to why the system is using so much memory, any ideas/tips?

Thanks

Vin

[Vince Pooley]

Found the culprit, feel a bit silly I hadn't found it before posting but here's the solution.

Turns out that the server in question was a HPVM host in a previous life and was being repurposed, I just had not been informed of this and wasn't aware of the server's history.

The HPVM software/kernel module was reserving all that memory for non-existant VMs.

One could just disable HPVM by amending the HPVM_ENABLE variable in /etc/rc.config.d/hpvmconf but as it was not required any more on this server I resolved the issue by removing the virtualisation software.