topic Re: Out of memory in Operating System - Linux

Out of memory

Alberto Rocha — Thu, 20 Oct 2005 11:05:18 GMT

Hello, we have a C++ application running on top HP-UX 11i compiled w/

# what /opt/aCC/bin/aCC
/opt/aCC/bin/aCC:
$Revision: 92453-07 linker linker crt0.o B.11.37 040218 $
HP aC++ B3910B A.03.63
HP aC++ B3910B A.03.52 Language Support Library

# PHSS_33263 1.0 HP aC++ -AA runtime libraries (aCC A.03.65)

Unfortunately it crashes regularly w/ an uncomfortable 'Out of memory' message:

I report bellow the last few lines of the application log:

GraphicObject::~GraphicObject(20050408103340:336)
PgrafRootObject::clear_old_state(0,MetaObject,20050408103340:336)
Out of memory!
GraphicObject::~GraphicObject(20050408103339:330)
PgrafRootObject::clear_old_state(0,MetaObject,20050408103339:330)
[ 11:52:52.372 pgraf_1_0@172.20.255.86 End of IIOP Connection (144.64.188.202:1769) ]
Callback called exit.

Considering this information, can anyone give any hints? I send also info about the 'core' file:

$ ident core
core:
$Header:/home/gestSoft/sources/COMUNS/GW_SNMP_CORBA/MD000PRC027_SNMP_TRAP_DAEMON/V-1.3.02$
$Header:/home/gestSoft/sources/COMUNS/MD000LIB045_ORBIX_TOOLS/V-1.0.02$
$Header:/home/gestSoft/sources/COMUNS/MD000LIB125_UTILITIES/V-2.3.02$
$Header:/home/gestSoft/sources/COMUNS/MD000LIB044_API_SNMP/V-3.1.03$
$Header:/home/gestSoft/sources/COMUNS/GW_SNMP_CORBA/MD000LIB047_IDL_CORBA_SNMP/V-1.3.05$
$Revision: 1.9.214.1 $
$Date: 96/10/08 13:20:01 $
$Revision: 1.9.214.1 $
$Date: 96/10/08 13:20:01 $
$Revision: libxti.2: STREAMS: PATCH_11.11 (PHNE_27703) Fri Sep 27 13:18:08 PDT 2002 $
$Revision: libpthread.1: @(#) depot-32pa R11.11_BL2004_0413_6 PATCH_11.11 PHCO_30544 Tue Apr 13 22:31:36 PDT 2004 $
$Revision: 1.9.1.1 $
$Header: U_qfcpy.s,v 66.1 90/07/13 12:21:03 pratap Exp $
$Header: U_qfabs.s,v 66.1 90/07/13 12:12:17 pratap Exp $

$ what core

SNMP++ $Id: address.cpp,v 1.12 2005/02/08 21:24:29 katz Exp$
SNMP++ $Id: counter.cpp,v 1.5 2005/01/19 22:18:38 katz Exp $
SNMP++ $Id: gauge.cpp,v 1.4 2004/03/03 23:11:21 katz Exp $
SNMP++ $Id: octet.cpp,v 1.8 2005/02/08 21:19:25 katz Exp $
SNMP++ $Id: pdu.cpp,v 1.11 2005/02/08 19:49:44 katz Exp $
SNMP++ $Id: ctr64.cpp,v 1.6 2005/01/17 21:11:43 katz Exp $
SNMP++ $Id: eventlist.cpp,v 1.5 2004/03/03 23:11:21 katzExp$
SNMP++ $Id: msec.cpp,v 1.6 2005/01/17 21:10:07 katz Exp $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revisio@SFÿ
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $
hosts $Revision: 1.9.214.1 $ $Date: 96/10/08 13:20:01 $

Re: Out of memory

Steven E. Protter — Thu, 20 Oct 2005 11:28:38 GMT

swapinfo -tam

How much memory does the box have?

How is it laid out?

Perhaps reboot and see if the problem persists.

Perhaps the C++ compiler and/or environment is out of date.

ipcs

What does shared memory look like.

It would be helpful to know your system type and how much memory it has.

SEP

Re: Out of memory

Alberto Rocha — Thu, 20 Oct 2005 11:59:22 GMT

>> SYSTEM TYPE: rp5470 w/ 4 CPU's 850MHz and 4GB mem
# model
9000/800/L3000-5x

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 9000 2575 6425 29% 0 - 1 /dev/vg00/lvol2
reserve - 2604 -2604
memory 3068 747 2321 24%
total 12068 5926 6142 49% - 0 -

# ./how_much_memory.sh
4096 MB

>> Reboot:
the problem persists after a week of usage --> it points to an app memory leak

>> C++ compiler/environment out of date
- the production machine is considerable up to date w/ most recent patches
- the development/compilation machine couldn't be (i'll check)

>> IPCS
# ipcs -m
IPC status from /dev/kmem as of Thu Oct 20 17:50:36 2005
T ID KEY MODE OWNER GROUP
Shared Memory:
m 0 0x411c18cf --rw-rw-rw- root root
m 1 0x4e0c0002 --rw-rw-rw- root root
m 2 0x4120004f --rw-rw-rw- root root
m 3075 0x0c6629c9 --rw-r----- root root
m 4 0x06347849 --rw-rw-rw- root root
m 1541 0x4919172a --rw-r--r-- root root
m 9734 0x5e100a3e --rw------- root root
m 30215 0x00000000 D-rw------- seq
m 520 0x896b0f10 --rw-r----- oracle dba
m 9 0x000003e8 --rw-rw---- seq

# ipcs -q
IPC status from /dev/kmem as of Thu Oct 20 17:50:44 2005
T ID KEY MODE OWNER GROUP
Message Queues:
q 0 0x3c1c226c -Rrw--w--w- root root
q 1 0x3e1c226c --rw-r--r-- root root

# ipcs -s
IPC status from /dev/kmem as of Thu Oct 20 17:50:51 2005
T ID KEY MODE OWNER GROUP
Semaphores:
s 0 0x411c18cf --ra-ra-ra- root root
s 1 0x4e0c0002 --ra-ra-ra- root root
s 2 0x4120004f --ra-ra-ra- root root
s 3 0x00446f6e --ra-r--r-- root root
s 4 0x00446f6d --ra-r--r-- root root
s 5 0x01090522 --ra-r--r-- root root
s 6 0x411c0518 --ra-ra-ra- root root
s 7 0x61200bdd --ra-ra-ra- root root
s 8 0x73200bdd --ra-ra-ra- root root
s 9 0x70200bdd --ra-ra-ra- root root
s 10 0x69200bdd --ra-ra-ra- root root
s 11 0x75200bdd --ra-ra-ra- root root
s 12 0x63200bdd --ra-ra-ra- root root
s 13 0x64200bdd --ra-ra-ra- root root
s 14 0x66200bdd --ra-ra-ra- root root
s 15 0x6c200bdd --ra-ra-ra- root root
s 16 0x6d200bdd --ra-ra-ra- root root
s 17 0x6f200bdd --ra-ra-ra- root root
s 18 0x4919172a --ra-r--r-- root root
s 19 0x410c0544 --ra-ra-ra- root root
s 20 0x00000001 --ra-ra-ra- root root
s 21 0x52200bdd --ra-ra-ra- root root
s 12310 0xdc1d9cc8 --ra-r----- oracle dba
s 23 0x000003e8 --ra-ra---- seq

Re: Out of memory

Bill Hassell — Thu, 20 Oct 2005 12:22:25 GMT

Hopefully, your programmer knows how much memory is normally requested by the program. Unfortunately, the programmer did not gracefully handle a failed memory request so we have to guess whether this is local memory or shared memory. Unless ytou've changed maxdsiz, your program will fail once it requests more than 64megs of RAM (256 megs for recent versions of HP-IX 11i). maxdsiz is a kernel parameter that protects your RAM from runaway programs, but can be changed with SAM. Start by setting maxdsiz to 900 megs and see if the program runs without a problem.

To watch your program's size, use ps as in:

UNIX95=1 ps -p #### -o vsz,pid,ppid,args

where #### is the processID for your program. The VSZ column is the local memory size (Kbytes) of your program. If it is approaching 64megs and then the program crashes, then maxdsiz is the limit it is hitting.

It might also be a shared memory limit - just like maxdsiz, there are low limits placed on the number and size of shared memory segments, and can be changed with SAM.

Re: Out of memory

B. Hulst — Thu, 20 Oct 2005 12:36:58 GMT

Additionally you might want to inspect the output of:

$ strings core | more

:-)

Re: Out of memory

Alberto Rocha — Thu, 20 Oct 2005 13:50:22 GMT

>> Unless you've changed maxdsiz

In fact we are aware of that need. It's current value is:
# kmtune | grep maxdsiz
maxdsiz 18446744071562067968 - (0X04000000+0X07C000000)
maxdsiz_64bit 3221225472 - (0X0000000040000000+0X80000000)

NOTE: surprisingly, 'maxdsiz' reports a very large number! On the other hand, SAM reports maxdsiz = 2147483648 (a more regular one and in accordance with the one i had set)

# ./progsize.sh 26073
VSZ PID PPID COMMAND
11324 26073 1 /home/appuser/exes/MD402PRC108_SVR_PGRAF

>> might also be a shared memory limit
# kmtune | grep shm
shmem 1 - 1
shmmax 1072693248 Y (0X4000000+0X3BF00000)
shmmni 512 - (200+312)
shmseg 512 Y (120+392)

Re: Out of memory

Bill Hassell — Thu, 20 Oct 2005 14:12:50 GMT

maxdsiz is quite out of whack and maxdsiz_64 is dependent on maxdsiz. Set maxdsiz to 2Gb and maxdsiz_64 is fine at 13 Gb.

Re: Out of memory

Alberto Rocha — Mon, 24 Oct 2005 03:50:28 GMT

>> maxdsiz is quite out of whack and maxdsiz_64 is dependent on maxdsiz. Set maxdsiz to 2Gb and maxdsiz_64 is fine at 13 Gb.

Why did you said "maxdsiz is quite out of whack"?

Let me see if i follow your settings: maxdsiz equals the maximum available memory and maxdsiz_64bit equals the total amount of memory (physical=4M + swap=9M), right?

Re: Out of memory

Bill Hassell — Mon, 24 Oct 2005 08:01:31 GMT

maxdsiz is a fence or limit for 32bit programs only. A 32bit program cannot address more than 900 megs of local data space unless it has been compiled with the EXEC MAGIC feature. If that is true, the program can then address no more than 1750 megs of memory. If you are running 11.00 or higher (witgh latest patches), then you can use chatr or the compiler to add quadrant 3 support and the 32bit program can get almost 2700 megs of RAM. And with 11.11 and higher (patches may be required), quadrant 4 may also be included for a maximum of 3500 megs or so. A 32bit programm cannot address any memory beyond about 3750 megs, so the maxdsiz number: 18446744071562067968 is quite out of range (unless it was a cut-n-paste error).

Now the hex values: 0X04000000+0X07C000000 are 67108864 and 2080374784 (67 megs and 2080 megs respectively), and these are both within range (I don't know why there are 2 values), but even adding them together is a valid setting (2147 megs apx) for maxdsiz. NOTE: maxdsiz (and maxdsiz_64) set memory limits for managing large programs. You can set the values low and programs will abort because they can't get the amount of RAM they request, or set the limits very high and the programs can get what they want, as long as the program can actually address that much RAM.

maxdsiz_64 has been set to 3Gb which is fine because it is larger than maxdsiz (if it were less, then the lower number would be the limit for 32bit and 64bit programs). You can set maxdsiz_64 to 100Gb if you like. The amount or RAM and swap space then limit the largest 64bit program.

Again, these are just limits and are not directly connected to RAM or swap. If a program exceeds a maxdsiz limit or a RAM+sweap limit, the result is exactly the same: out of RAM or program too big. For 32bit programs, there are many restrictions on the maximum amount of RAM that can be addressed, even when maxdsiz is very large.

Re: Out of memory

Alberto Rocha — Wed, 26 Oct 2005 13:23:35 GMT

Thank you all for your usefull help. AR