HPUX 11.31 with Oracle 11g (running ASM) high memory usage

[Mauro Livi]

Hi all,

We have a new rx2800 i4 with 128GB physical memory running HP-UX 11.31 and Oracle 11g (running ASM). We've recently seen high memory usage through glance  consistantly running at about 99%. Given that this is a 128GB system, running at 99% of the available physical memory makes me a little uncomfortable (though I'm not getting any page outs)

Now in glance I see 55GB being used by File Cache which seems to be at the root of the problem. My filecache_max setting is at the default since I read that it should probably be left at its dynamic setting and to let HP-UX manage itself. However, I believe that can also allocate up to 50% of the physical memory which certainly seems to be happening in this case.

So given the KCTUNE output below, do you folks think that I should perhaps shrink the filecache_max setting and restricting this number to say 20% or so? I'm coming from HP-UX 11.11 so this tunable is a bit new to me.

Any other suggestions based on my kctune below?

Incidently after a little more research, I found the following document which seems to support my thinking of shrinking filecache_max...but I would like some input as well.

http://support.hp.com/sg-en/document/c01646142

As always thank you in advance, and any input will be greatly appreciated (and kudos awarded of course).

Thanks

Mauro 

Tunable                                 Value  Expression   Changes
NSTREVENT                                  50  Default
NSTRPUSH                                   16  Default
NSTRSCHED                                   0  Default
STRCTLSZ                                 1024  Default
STRMSGSZ                                    0  Default
acctresume                                  4  Default
acctsuspend                                 2  Default
aio_iosize_max                              0  Default      Immed
aio_listio_max                            256  Default      Immed
aio_max_ops                              2048  Default      Immed
aio_monitor_run_sec                        30  Default      Immed
aio_physmem_pct                            10  Default      Immed
aio_prio_delta_max                         20  Default      Immed
aio_proc_max                                0  Default      Immed
aio_proc_thread_pct                        70  Default      Immed
aio_proc_threads                         1024  Default      Immed
aio_req_per_thread                          1  Default      Immed
allocate_fs_swapmap                         0  Default
alwaysdump                                  0  Default      Immed
audit_memory_usage                          5  Default      Immed
audit_track_paths                           0  Default      Auto
base_pagesize                               4  Default
btree_prune_min_interval                   90  Default      Immed
close_wake_blkdflks                         0  Default
cmpt_restrict_tl                            0  Default      Immed
copy_on_write                               1  Default      Immed
core_addshmem_read                          0  Default      Immed
core_addshmem_write                         0  Default      Immed
create_fastlinks                            0  Default
default_disk_ir                             0  Default
desfree_pct                                 0  Default      Immed
diskaudit_flush_interval                    5  Default      Immed
dlpi_max_clones                          3992  Default      Immed
dlpi_max_ub_promisc                         1  Default      Immed
dma32_pool_size                     268435456  Default
dnlc_hash_locks                           512  Default
dontdump                                    0  Default      Immed
dst                                         1  Default
dump_compress_on                            1  Default      Immed
dump_concurrent_on                          1  Default      Immed
executable_stack                            0  Default      Immed
expanded_node_host_names                    0  Default      Immed
fcache_fb_policy                            0  Default      Immed
fcache_seqlimit_file                      100  Default      Immed
fcache_seqlimit_scope                       0  Default      Immed
fcache_seqlimit_system                    100  Default      Immed
fcache_vhand_ctl                            0  Default      Immed
fcd_disable_mgmt_lun                        0  Default      Immed
fclp_ifc_disable_mgmt_lun                   0  Default      Immed
fcoc_ifc_disable_mgmt_lun                   0  Default      Immed
fcq_disable_mgmt_lun                        0  Default      Immed
filecache_max                     65301360640  Default      Auto
filecache_min                      6530134016  Default      Auto
fr_rulecache                                0  Default      Immed
fr_statemax                            800000  Default      Immed
fr_tcpidletimeout                       86400  Default      Immed
fr_tcptimewait                            120  Default      Immed
frnat_tcptimewait                         120  Default      Immed
fs_async                                    0  Default
fs_symlinks                                20  Default      Immed
ftable_hash_locks                          64  Default
gvid_no_claim_dev                           0  Default
hires_timeout_enable                        0  Default      Immed
hp_hfs_mtra_enabled                         1  Default
intr_strobe_ics_pct                        80  Default      Immed
io_ports_hash_locks                        64  Default
ipf_icmp6_passthru                          0  Default      Immed
ipl_buffer_sz                            8192  Default      Immed
ipl_logall                                  0  Default      Immed
ipl_suppress                                1  Default      Immed
ipmi_watchdog_action                        0  Default      Immed
ipnat_enable                                1  Default      Immed
kmem_aggressive_caching                     0  Default      Immed
kmem_do_largepage_cache_demotion            1  Default      Immed
kmem_va_cache_enable                        1  Default
ksi_alloc_max                           33600  Default      Immed
ksi_send_max                               32  Default
lcpu_attr                                   0  0            Imm (auto disabled)
lotsfree_pct                                0  Default      Immed
max_acct_file_size                    2560000  Default      Immed
max_async_ports                          4096  Default      Immed
max_mem_window                              0  Default      Immed
max_thread_proc                          1200  1200         Immed
maxdsiz                            1073741824  Default      Immed
maxdsiz_64bit                      4294967296  Default      Immed
maxfiles                                63488  63488
maxfiles_lim                            63488  63488        Immed
maxrsessiz                            8388608  Default
maxrsessiz_64bit                      8388608  Default
maxssiz                             134217728  134217728    Immed
maxssiz_64bit                      1073741824  1073741824   Immed
maxtsiz                             100663296  Default      Immed
maxtsiz_64bit                      1073741824  Default      Immed
maxuprc                                  3686  3686         Immed
mca_recovery_on                             1  Default      Auto
mdep_reduce_rse_size                        0  Default
mpas_readonly_text                          0  Default      Immed
mprotect_reduce_protid_on                   0  Default      Immed
msgmbs                                      8  Default      Immed
msgmnb                                  16384  Default      Immed
msgmni                                   4096  4096         Immed
msgtql                                   4096  4096         Immed
ncdnode                                   150  Default
nclist                                   8292  Default
ncsize                                  35840  35840
nflocks                                  4096  Default      Auto
nfs2_max_threads                            8  Default      Immed
nfs2_nra                                    4  Default      Immed
nfs3_bsize                              32768  Default      Immed
nfs3_do_readdirplus                         1  Default      Immed
nfs3_jukebox_delay                       1000  Default      Immed
nfs3_max_commit_threads                     8  Default      Immed
nfs3_max_threads                            8  Default      Immed
nfs3_max_transfer_size                1048576  Default      Immed
nfs3_max_transfer_size_cots           1048576  Default      Immed
nfs3_nra                                    4  Default      Immed
nfs4_bsize                              32768  Default      Immed
nfs4_max_threads                            8  Default      Immed
nfs4_max_transfer_size                1048576  Default      Immed
nfs4_max_transfer_size_cots           1048576  Default      Immed
nfs4_nra                                    4  Default      Immed
nfs_portmon                                 0  Default      Immed
ngroups_max                                20  Default      Immed
ninode                                  34816  34816
nkthread                                 8416  Default      Immed
nproc                                    4200  Default      Immed
npty                                       60  Default
nstrpty                                    60  Default
nstrtel                                    60  Default
nswapdev                                   32  Default
nswapfs                                    32  Default
numa_mode                                   0  Default
numa_policy                                 0  Default      Immed
numa_sched_launch                           1  Default      Immed
override_umask                              0  Default      Immed
pa_maxssiz_32bit                     83648512  Default      Immed
pa_maxssiz_64bit                    536870912  Default      Immed
pagezero_daemon_enabled                     1  Default      Immed
patch_active_text                           1  Default      Immed
pci_eh_enable                               1  Default
pci_error_tolerance_time                 1440  Default      Immed
process_id_max                          30000  Default      Auto
process_id_min                              0  Default      Auto
pwr_idle_ctl                                0  Default      Immed
remote_nfs_swap                             0  Default
rng_bitvals                        9876543210  Default
rng_sleeptime                               2  Default
rtsched_numpri                             32  Default
rusage_hires_enable                         0  Default      Immed
sched_thread_affinity                       6  Default      Immed
scroll_lines                              100  Default
secure_sid_scripts                          1  Default      Immed
semaem                                  16384  Default
semmni                                   4096  4096
semmns                                   8192  8192
semmnu                                   4092  4092
semmsl                                   2048  Default      Immed
semume                                    100  Default
semvmx                                  32767  Default
shlib_debug_enable                          0  Default      Immed
shmmax                            51539607552  51539607552  Immed
shmmni                                   4096  4096         Immed
shmseg                                    512  512          Immed
streampipes                                 0  Default
swchunk                                  2048  Default
sysv_hash_locks                           128  Default
tcphashsz                                   0  Default
timeslice                                  10  Default
timezone                                  420  Default
uname_eoverflow                             1  Default      Immed
vnode_cd_hash_locks                       128  Default
vnode_hash_locks                          128  Default
vps_ceiling                                16  Default      Immed
vps_chatr_ceiling                     1048576  Default      Immed
vps_pagesize                               16  Default      Immed
vx_dexh_sz                             524287  Default      Immed
vx_era_nthreads                             5  Default
vx_maxlink                              32767  Default
vx_ninode                                   0  Default      Immed
vxfs_bc_bufhwm                              0  Default      Immed
vxfs_ifree_timelag                          0  Default      Immed

[Steven Schweda]

> Now in glance I see 55GB being used by File Cache which seems to be at
> the root of the problem. [...]

   I know nothing, but if I understand the idea of a file-system cache,
then the biggest problem here may be that you believe that you have a
problem.

   If you limit the size of the file cache, so that you then have some
free memory, who wins?  Of what use (to anyone) is free (that is,
unused) memory?

   If the file-cache system failed to give up memory when it's needed by
some real-work process (or if it didn't do it quickly enough), _then_
you might have a problem.  If your real-work jobs don't consume all the
available memory, then where's the harm in letting the file-cache system
use what's left?

> http://support.hp.com/sg-en/document/c01646142

   Ok, you might also get more I/O than you want, but "99% of the

available physical memory" is not an I/O problem.  (It may be a
perception problem.)

[Mauro Livi]

Thanks for your input Steven,

Actually I misspoke, I do see some page outs, though nothing really huge.

Well a couple things concern me one is the recommendation from HP in the article that I cited. Note the recommendations for systems with >32GB

 http://support.hp.com/sg-en/document/c01646142

Moreover, we checked with Oracle, and they pretty much made similar recommendations. So while it might be an issue of "perception", I'm questionining if it's "prudent" to allow caching to use that much as in the event that more system or user memory is required, that would go out to paging since there is no more available (I would think).

Thanks again for your input

[Dennis Handly · Accepted solution]

> If you limit the size of the file cache, so that you then have some free memory, who wins?

The idea is that Oracle (and its SGA) can manage the memory better than the OS.

 >If the file-cache system failed to give up memory when it's needed by some real-work process

Oracle allocates a fixed size SGA and never tries to increase it, so the file cache isn't pressured.

>we checked with Oracle, and they pretty much made similar recommendations. 

Oracle should explain this better.  I.e. needed since the whole purpose of the box is to host Oracle DB.

In any case, you WANT high memory usage, you paid for it.  You don't want high swapping.

[Mauro Livi]

Hi Dennis,

> In any case, you WANT high memory usage, you paid for it.  You don't want high swapping

Yes, I agree with that. In our case the system the system generally hums along pretty happily with the way Oracle SGA has been configured. It's just that when we did some of the things outside of oracle (Fbackup/Frestore for example...we use that for a couple things), the system will allocate a ridiculous amont of cache to the files.

I had HP Support troubleshoot and the recommendation was for filesize_max to be set at 10% and filesize_min at 5%. We'll see how that goes.

Again, always thankful and appreciative of everyone's input.

Mauro

[Dennis Handly]

>>you WANT high memory usage, you paid for it.  You don't want high swapping

>I agree with that. In our case the system the system generally hums along pretty happily

One possible problem with high memory usage is if you decide to do something different and you have to swap out to change gears.  By reducing your filesize_max, you could dedicate more to the Oracle SGA.