Increase lockable memory

[Stephen Andreassend]

Those figures are after I changed those 3 kernel parameters and has stopped the Oracle instance from crashing and swapinfo no longer goes to 100%.

[Stephen Andreassend]

This can be contrasted with the figures before I changed the kernel:


# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 95 -95
memory 12860 931 11929 7%
total 16956 1026 15930 6% - 0 -

Now with this configuration, if I was to start Oracle, then start the applications on the other server connecting to Oracle, the Oracle instance crashes with the memory error, and HPUX hangs.

[Sridhar Bhaskarla]

Well.. obviously the changes (low buffer cache) you made have lowered the kernel's memory which would also affect the pseudoswap. You are still low swap and possibly low on memory as well. Your total swap is equal to physical memory but it is a good idea to have around 25% of headroom on swap to avoid any surprized presented by the application. I would still suggest you add little more device swap - probably around 2GB secondary swap to be out of danger zone or reduce the SGA slightly. Oracle crashes if it can't start the processes it needs to.

-Sri

[Stephen Andreassend]

So in actual fact, if I rollback the kernel changes, then I should see 100% utilisation on /dev/vg00/lvol2?

I'll give it a go.

[Stephen Andreassend]

I restored the original kernel to reproduce the problem.

Here is the swapinfo with Oracle running and all the applications connected.

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 4096 -4096
memory 12860 12628 232 98%
total 16956 16724 232 99% - 0 -

Not much spare memory it seems, and that 4GB segment due to swapmem_on=1 appears to be unused.

Now if I start SQL*Plus and do a large query with a large sort_area_size of 200MB, this is the swapinfo:

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 4096 -4096
memory 12860 12860 0 100%
total 16956 16956 0 100% - 0 -

The query crashes with this message:

ERROR at line 1:
ORA-04030: out of process memory when trying to allocate 8192 bytes (sort
subheap,sort key)

As you can see, that 4GB of RAM is unused and the dev swap is 0% used.

# echo total_lockable_mem/D | adb -k /stand/vmunix /dev/mem
total_lockable_mem:
total_lockable_mem: 3285279
=12833MB

I would therefore expect disabling swapmem_on and allocating sufficient disk swap would be beneficial for Oracle.

Does swapmem_on require that disk swap size >= RAM or is this just a recommendation?

[Sridhar Bhaskarla]

Stephen,

//Does swapmem_on require that disk swap size >= RAM or is this just a recommendation?//

The total swap space you have should be *atleast* (requirement) or exceed (recommendation) the total memory you have on the system. Though some threads say to configure swap with only 25% of memory, I alwasy suggest to configure atleast 50-75% to avoid suprizes. I wouldnt' worry about lockable memory.

Look at the following threads for more discussion. I would basically be posting the same response.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=744365
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=743917


-Sri

[Stephen Andreassend]

Quote from your 2nd URL:

You have swap space greater than physical memory size, so you can turn off pseudo swap. You can set the kernel parameter swapmem_on to 0. This will allow to better use of memory.

Regards,
Zygmunt

[Sridhar Bhaskarla]

hmmm... that was not by me...I wouldn't disable swapmem_on.

Bottom line is to keep the 'total used' in 'swapinto -t' output below 100%. 'swapmem_on' parameter will give you 75% of the swap space. Configure another 50-75% of device swap ( primary+secondary swap or filesystem swap) to be on safer side.

I would suggest that you just add another 2GB of device swap as secondary swap and you will be fine. Make your dbc_max_pct and dbc_min_pct as 3% (~500MB) and 2% respectively. Keep watching 'swapinfo -t'. Never let it go beyond 95%. No need to disable swapmem_on.

-Sri

[Stephen Andreassend]

I have again re-tuned the kernel:
nproc 1024
db_max/min_pct 1 (mincache=direct can be used for Oracle IO)

And the machine is stable with apps running against it.

# vmstat 1 2 | egrep -v 'procs|free' | tail -1 | awk '{ print $5*4/1024 }'
2857.75

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 4096 -4096
memory 13313 12448 865 94%
total 17409 16544 865 95% - 0 -

So swapmem_on=1 can survive for now.

Thanks for your help (so far)