Memory full on hux 11.11

[vaman]

I am getting "out of memory HP-UX Error: 12: Not enough space " error , memory utilisation is almost 96%.
the swapinfo o/p is like
swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16384 2748 13636 17% 0 - 1 /dev/vg00/lvol2
dev 16000 2507 13493 16% 0 - 1 /dev/vg00/swap
reserve - 24748 -24748
memory 25299 24384 915 96%
total 57683 54387 3296 94% - 0 -
Pl suggest.

[RAC_1]

Most probablly there is no room to start the a process. Can you post

glance -m

Also when do you get this error?? Look at this --

grep -i 12 /usr/include/sys/errno.h
#define ENOMEM 12 /* Not enough core */

Anil

[Pete Randall]

Either add more memory or add more swap space. Swap space is the easier/cheaper of the two but your performance is going to be horrible while all that swapping is going on. Check dbc_max kernel parameter to see if you're using a lot of buffer cache. Ideally you want to hold it down to 300-400 Mb - something like 5 to 10 percent.


Pete

[Geoff Wild]

As well as the posts above, find out what is using all that ram:

# cat memtop
#!/bin/sh
# memtop - show top memory users and pid
# VSZ is in KB
#
echo "VSZ(KB) PID RUSER COMMAND"
UNIX95= ps -e -o 'vsz pid ruser args' |sort -nr|head -30

Rgds...Geoff

[Sridhar Bhaskarla]

Hi Vaman,

The 'memory' line in 'swapinfo' tells you how much memory *is not available* for reservation anymore. It may be closely associated with utilization but not utilizatin itself.

Not enough space means, you don't have enough 'swap space'. You could add some more swap to facilitate to bring more processes up. However, the system may degrade in performance as the system seems to be already paging. See "MB Used" Column. Run 'vmstat' and observe 'po' column. If you see it non-zero, then it's better to add more memory if there are no memory leaks.

Look at your dbc_max_pct value. "kmtune -l -q dbc_max_pct". If it is left at it's default which is 50%, it can be a waste of memory. Reduce it to get around 500-700MB.

-Sri

[Steven E. Protter]

Follow on to Sri

dbc_max_pct we set that to 7
dbc_min_pct we set that to 5. Changing this figure is expensive in cpu terms, so we tune it tight to avoid this.

Overall you don't have enough memory. Extending swap will probably allow your system to work, though it will be very slow.

A memory upgrade on the otherhand might slow down or stop paging which will make the system much more usable.

Its not cheap, but nowadays if you are paging, you should almost always be planning a memory buy.

SEP

[Bill Hassell]

You have lots of memory (24 Gb) and lots of swap space (32Gb) although a lot is actually used (16Gb). Your problem may be due to maxdsiz in your kernel. The kernel limits the maximum size that a program can grow with this parameter. It is used to keep a runaway program from using all of memory.

If maxdsiz is 64megs, (a long time default), then programs can't grow any larger. If this value is more than 1Gb, then the program that is out of memory needs to be chatr'ed to take advantage of quadrant 1 and 2, assuming that this is a 32bit program and the program does not need more than 1700megs of data space. Now if the program wants gigabytes of space, it must be recompiled as a 64bit program since a 32bit program is severely constrained on maximum memory addresses. If this is actually a 64bit program, then the parameter maxdsiz_64 will have to be bumbed up.

[Zygmunt Krawczyk]

You have swap space greater than physical memory size, so you can turn off pseudo swap. You can set the kernel parameter swapmem_on to 0. This will allow to better use of memory.

Regards,
Zygmunt

[Con O'Kelly]

Hi Vaman

I think you are making an assumption about the 'memory' line in swapinfo that is not correct. This refers to pseudo-swap and is not indicative of actual memory use. The amount of pseudo memory is roughly equal to 75% of phys memory so you have about 30GB of phys memory.

You need to check the amout of free memory on the system. Use glance (option 'm') or:
# vmstat 1 2 | egrep -v 'procs|free' | tail -1 | awk '{ print $5*4/1024 }'.
(This gives free memory in MB)

If you have plenty of free memory and pageout (po) rate is low, then your problem is very likely related to a kernel parameter setting as Bill has pointed out.

Cheers
Con

[vaman]

here is o/p ..
kmtune -l -q dbc_max_pct | more
Parameter: dbc_max_pct
Current: 10
Planned: 10
Default: 50
Minimum: -
Module: -
Version: -
Dynamic: No
kmtune -l -q maxdsiz
Parameter: maxdsiz
Current: 4294963200
Planned: 4294963200
Default: 0x10000000
Minimum: -
Module: -
Version: -
Dynamic: No

& glance -m (pl pase it in notepad to view)

CPU Util S SAR RU U | 76% 71% 90%
Disk Util F FVV | 8% 13% 33%
Mem Util S SU UB B | 93% 93% 93%
Swap Util U UR R | 95% 95% 95%
MEMORY REPORT Users= 17
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 2185 625006 364.1 2037.8 10144.6
Page In 1763 375918 293.8 1225.6 3540.3
Page Out 0 0 0.0 0.0 0.0
KB Paged In 92kb 19.7mb 15.3 65.6 304.4
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 26 3542 4.3 11.5 74.8
VM Writes 12 744 2.0 2.4 10.4

Total VM : 39.1gb Sys Mem : 6.68gb User Mem: 20.4gb Phys Mem: 32.0gb

In swapinfo What I think is there are 2 dev swaps but only 17% & 18% been used. but in memory line I get 95% used.

[Jairo Campana]

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=605128

[vaman]

hello Anil (RAC)

here is the o/p
grep -i 12 /usr/include/sys/errno.h
#define ENOMEM 12 /* Not enough core */
so is it indicate the process does not have any physical memory to start?

[Sridhar Bhaskarla · Accepted solution]

Hi Vaman,

Not having memory doesn't stop you from running any commands. It will just slow things down.. may be terribly if it too bad..

But not having 'enough swap' will stop the system from bringing more processes. Everytime a process starts, system will allot some space for it in the swap area and if there is not enough swap space, then the process will fail to come up.

Swap areas are *physically* used only when there is memory pressure. Otherwise they are used just for reservations.

Because HP-UX works in virtual memory realm you can always bring up processes than your actual physical memory can allow. Looks like some processes were started for which system tried to reserve swap space making it to reach 100%. You would need to find those processes. If this is reoccuring again and again, then I suggest you put a cronjob that runs every 5 mins or so that will capture the processes and the memory used by them.

UNIX95= ps -e -o 'vsz pid args'|sort -n > /tmp/memory/$(date +%Y%m%d%H%M)

Keep looking at these files to determine the rogue processes. You may find some with huge vsz values when this occurs.

To temporarily get out of this situation, add more swap space. But adding more memory will improve performance. However, you will need to work with your developers/appadmins to see if they can fix anything to reduce the memory usage.

-Sri

[vaman]

Thanks Sri,
Just to clarify doubt
the swapinfo -tm command shows two device swaps with utilisation with only 16% & 17%, but the memory line in swapinfo shows 90% used how? (is it mixed with phy. memory)
In glance -m, swap utilisation is also shown above 90% with memory also above 90% utilisation.
Also last row of this command gives the memory divisions. In that my phy mem is 32gb,sys mem is 6.68, user mem =19.9, buf cache is 3.20gb, free mem is 2.15gb.
But my doubt is the total vm is showing 38.4gb & active vm is 27.1 gb so this vm figures are comming.
I am observing couple of oracle processes with aroud 1.2 gb each. (highest utilisation)

[Sridhar Bhaskarla]

Hi Vaman,

//the swapinfo -tm command shows two device swaps with utilisation with only 16% & 17%, but the memory line in swapinfo shows 90% used how? (is it mixed with phy. memory)
//

There is a subtle difference between 'swap used' for the device areas and memory . "MB used" from the swap 'devices' is really the pages sitting on the swap that got paged out either due to memory pressure or memory mapped files.

"MB Used" from the memory line is different from the above. It is to be understood as 'memory not available' for swap reservations anymore. Dynamically alloted memory by the kernel, the memory locked by user processes etc., will appear in this figure. So, the device swap has to offset for that calculations.

I suggest you go to docs.hp.com and read 'memory management white papers' to get a better picture.

-Sri