Increase lockable memory

[Stephen Andreassend]

16GB RAM, 64bit HPUX 11 with 64Bit Oracle.

swapmem_on=1 means that only 12GB is lockable due to the 75% rule.

How can I increase the size of the lockable memory?

[Sridhar Bhaskarla]

Stephen,

The swap allocated with 'swapmem_on' parameter is affected by lockable memory but doesn't affect lockable memory.

The kernel parameter "unlockable_mem" determines how much memory can be locked by the processes. Look at the following document on more information on this parameter.

http://docs.hp.com/hpux/onlinedocs/TKP-90202/TKP-90202.html

-Sri

[Stephen Andreassend]

DESCRIPTION

In previous versions of HP-UX, system configuration required sufficient physical swap space for the maximum possible number of processes on the system. This is because HP-UX reserves swap space for a process when it is created, to ensure that a running process never needs to be killed due to insufficient swap.
This was difficult, however, for systems needing gigabytes of swap space with gigabytes of physical memory, and those with workloads where the entire load would always be in core. This tunable was created to allow system swap space to be less than core memory. To accomplish this, a portion of physical memory is set aside as 'pseudo-swap' space. While actual swap space is still available, processes still reserve all the swap they will need at fork or execute time from the physical device or file system swap. Once this swap is completely used, new processes do not reserve swap, and each page which would have been swapped to the physical device or file system is instead locked in memory and counted as part of the pseudo-swap space.

Who Is Expected to Change This Tunable?

Anyone.

Restrictions on Changing

Changes to this tunable take effect on the next reboot.

When Should the Tunable Be Turned On?

If this tunable is set to 0 (off), and a fork() or exec() process is failing with an ENOMEM error message and has sufficient system memory but insufficient unreserved swap space, then setting this tunable to 1 (on) will allow these processes to use pseudo-swap space and execute.

What Are the Side Effects of Turning the Tunable On?

Some physical memory is set aside for pseudo-swap, but since the kernel can steal pages from this allocation if needed (for locked memory or kernel memory), and the rest of the pages will only be used when physical swap is completely filled anyway, this is quite harmless.

When Should the Tunable Be Turned Off?

This tunable can be turned off if there are sufficient physical swap devices or file systems such that the system workload never fails to reserve swap. However, turning this feature off really does not gain the system anything.

What Are the Side Effects of Turning the Tunable Off?

Processes will be limited to physical swap devices or file systems for reserving their swap on a fork() or an exec().

What Other Tunables Should Be Changed at the Same Time?

None.

[Sanjay_6]

Hi Stephen,

Take a look at this link,

http://docs.hp.com/hpux/onlinedocs/939/KCParms/KCparam.UnlockableMem.html

Total Memory - Unlockable Memory = Lockable Memory

Hope this helps.

Regds

[Stephen Andreassend]

I have reduced the fs buffer cache (1% min/max) and reduced nproc, and it has helped

[Sanjay_6]

Hi,

Reducing nproc is not a very good idea unless it was set an abnormally high value.

http://docs.hp.com/hpux/onlinedocs/939/KCParms/KCparam.Nproc.html

Hope this helps.

regds

[Sridhar Bhaskarla]

Hi (Again),

Yes. The document pasted by you confirms what I said.

It's better we take an example. Let's consider some raw numbers and hypothetical scenarios to make it easy.

Consider 100MB system with 100MB of device swap with pseudoswap on. The total swap is 175MB initially. Say, there are processes that locked 80MB of memory. So, 75MB which was alloted from pseudoswap would be gone and you will be effectively left with 100MB-(reserved by other processes) of swap space. You can keep locking the memory further (< unlockable_mem) and it may cause unlocked processes to page out if the available memory is critically low.

However, case is different if you don't have enough device swap. Say you configured only 10MB of device swap. After 10MB worth of processes are brought up, system will start locking the processes in memory that further come up and eventually the system may run of swap space. Under that situation, you will not only be able to lock the memory but also will not be able to bring up more processes though you may see some free memory on the system.

-Sri

[Stephen Andreassend]

unlockable_mem is defaulting to 65536, its not set (0) in SAM.

nproc is 1024, monitoring sar -v.

The server is running only Oracle, problem starts when applications are started and connect to the db over SQL*Net, swapinfo -tam races to 100% and Oracle instance aborts with an out of memory error. Oracle configured to use 12GB shared memory, so there should be plenty left over for oracle shadow processes.

[Sridhar Bhaskarla · Accepted solution]

Stephen,

It's not the memory but the 'total swap' the decides whether you can bring up more processes are not. If 'Total Available' in 'swapinfo -t' is less than 16GB, then though there is free memory, you will not be able to bring up more processes. You will have to add extra swap.

-Sri

[Stephen Andreassend]

# swapinfo -t
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
reserve - 4194304 -4194304
memory 13632956 12730424 902532 93%
total 17827260 16924728 902532 95% - 0 -

# swaminfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 4096 -4096
memory 13313 12373 940 93%
total 17409 16469 940 95% - 0 -

[Stephen Andreassend]

Those figures are after I changed those 3 kernel parameters and has stopped the Oracle instance from crashing and swapinfo no longer goes to 100%.

[Stephen Andreassend]

This can be contrasted with the figures before I changed the kernel:


# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 95 -95
memory 12860 931 11929 7%
total 16956 1026 15930 6% - 0 -

Now with this configuration, if I was to start Oracle, then start the applications on the other server connecting to Oracle, the Oracle instance crashes with the memory error, and HPUX hangs.

[Sridhar Bhaskarla]

Well.. obviously the changes (low buffer cache) you made have lowered the kernel's memory which would also affect the pseudoswap. You are still low swap and possibly low on memory as well. Your total swap is equal to physical memory but it is a good idea to have around 25% of headroom on swap to avoid any surprized presented by the application. I would still suggest you add little more device swap - probably around 2GB secondary swap to be out of danger zone or reduce the SGA slightly. Oracle crashes if it can't start the processes it needs to.

-Sri

[Stephen Andreassend]

So in actual fact, if I rollback the kernel changes, then I should see 100% utilisation on /dev/vg00/lvol2?

I'll give it a go.

[Stephen Andreassend]

I restored the original kernel to reproduce the problem.

Here is the swapinfo with Oracle running and all the applications connected.

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 4096 -4096
memory 12860 12628 232 98%
total 16956 16724 232 99% - 0 -

Not much spare memory it seems, and that 4GB segment due to swapmem_on=1 appears to be unused.

Now if I start SQL*Plus and do a large query with a large sort_area_size of 200MB, this is the swapinfo:

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 4096 -4096
memory 12860 12860 0 100%
total 16956 16956 0 100% - 0 -

The query crashes with this message:

ERROR at line 1:
ORA-04030: out of process memory when trying to allocate 8192 bytes (sort
subheap,sort key)

As you can see, that 4GB of RAM is unused and the dev swap is 0% used.

# echo total_lockable_mem/D | adb -k /stand/vmunix /dev/mem
total_lockable_mem:
total_lockable_mem: 3285279
=12833MB

I would therefore expect disabling swapmem_on and allocating sufficient disk swap would be beneficial for Oracle.

Does swapmem_on require that disk swap size >= RAM or is this just a recommendation?

[Sridhar Bhaskarla]

Stephen,

//Does swapmem_on require that disk swap size >= RAM or is this just a recommendation?//

The total swap space you have should be *atleast* (requirement) or exceed (recommendation) the total memory you have on the system. Though some threads say to configure swap with only 25% of memory, I alwasy suggest to configure atleast 50-75% to avoid suprizes. I wouldnt' worry about lockable memory.

Look at the following threads for more discussion. I would basically be posting the same response.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=744365
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=743917


-Sri

[Stephen Andreassend]

Quote from your 2nd URL:

You have swap space greater than physical memory size, so you can turn off pseudo swap. You can set the kernel parameter swapmem_on to 0. This will allow to better use of memory.

Regards,
Zygmunt

[Sridhar Bhaskarla]

hmmm... that was not by me...I wouldn't disable swapmem_on.

Bottom line is to keep the 'total used' in 'swapinto -t' output below 100%. 'swapmem_on' parameter will give you 75% of the swap space. Configure another 50-75% of device swap ( primary+secondary swap or filesystem swap) to be on safer side.

I would suggest that you just add another 2GB of device swap as secondary swap and you will be fine. Make your dbc_max_pct and dbc_min_pct as 3% (~500MB) and 2% respectively. Keep watching 'swapinfo -t'. Never let it go beyond 95%. No need to disable swapmem_on.

-Sri

[Stephen Andreassend]

I have again re-tuned the kernel:
nproc 1024
db_max/min_pct 1 (mincache=direct can be used for Oracle IO)

And the machine is stable with apps running against it.

# vmstat 1 2 | egrep -v 'procs|free' | tail -1 | awk '{ print $5*4/1024 }'
2857.75

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 4096 -4096
memory 13313 12448 865 94%
total 17409 16544 865 95% - 0 -

So swapmem_on=1 can survive for now.

Thanks for your help (so far)