Issue with ENOMEM - what could be causing it ??

[Jaideep Krishnan]

Could someone please provide me some insights related to a situation I am encountering with the failure of mmap() ("ENOMEM" ) while executing a process, as viewed in the Truss output?!

I installed a product on an HP-UX machine, configured it, and tried to start its daemon. It responded with the message : 26556 Memory fault Server not running!! Failed to start ns-slapd process.

I am using a machine that about 2GB unused memory, while the product I installed needs about 128MB only.

I observed from the Truss output that mmap() was failing with ENOMEM message, as below:

mmap(NULL, 188416, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 10, 0x16000) .....
... ERR#12 ENOMEM

I also noticed the following statements in Truss:

Received signal 11, SIGSEGV, in user mode, [SIG_DFL], partial siginfo
Siginfo: si_code: I_NONEXIST, faulting address: 0x7ada001c, si_errno: 0
PC: 0xc23e9e3f, instruction: 0x44990038
exit(11) [implicit] ............................................................
... WIFSIGNALED(SIGSEGV)
waitpid(-1, WIFSIGNALED(SIGSEGV), WUNTRACED) ...................................
... = 1074
sigvec(SIGCLD, 0x7f7f0e70, 0x7f7f0e80) .........................................
... = 0
sigprocmask(SIG_BLOCK, 0x7f7f0e30, 0x7f7f0e10) .................................
... = 0
sigprocmask(SIG_SETMASK, 0x7f7f0e10, NULL) .....................................
... = 0
sigprocmask(SIG_BLOCK, 0x7f7f11b0, 0x7f7f1190) .................................
... = 0
sigprocmask(SIG_SETMASK, 0x7f7f1190, NULL) .....................................
... = 0

Is there a way to check and update "maxdsiz", "maxdsiz_64", "maxssiz" settings, IF THEY ARE THE CAUSE OF THE PROBLEM? (How to envisage that?) .

Current settings are the following:

/etc/conf/master.d/core-hpux:

*range maxdsiz<=0xfffff000
*range maxdsiz>=256*1024
*range maxdsiz_64bit<(4*1024*1024*1024*1024 - 1024*1024*1024)
*range maxdsiz_64bit>=256*1024
maxdsiz MAXDSIZ 0x10000000
maxdsiz_64bit MAXDSIZ_64BIT 0x0000000040000000

Appreciate if you can please let me know your inputs. Thanks!!

[RAC_1]

First where do you see, you have 2GB free memory?? Check glance and check for free memory.

Also the current kernel is /stand/system and not /usr/conf/....

The preocess is receving the signal 11, which is a segmentation violation. It cuold be maxtsiz, maxdsiz, maxssiz and shmmax.

If prgram is 64 bit program, it could be maxssiz_64bit etc. also how do you start the program. If from shell, look at the ulimit settings.

ulimit -Sa
ulimit -Ha

Anil

[Jaideep Krishnan]

Hi Anil, following is the result of ulimit -Ha and -Sa

ulimit -Ha:

core file size (blocks, -c) unlimited
data seg size (kbytes, -d) 1048576
file size (blocks, -f) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 2048
pipe size (512 bytes, -p) 16
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 76
virtual memory (kbytes, -v) unlimited


ulimit -Sa:

core file size (blocks, -c) 2097151
data seg size (kbytes, -d) 1048576
file size (blocks, -f) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 2048
pipe size (512 bytes, -p) 16
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 76
virtual memory (kbytes, -v) unlimited


Also, here is the extract from /stand/system

* Tunable parameters

STRMSGSZ 65535
dnlc_hash_locks 512
max_thread_proc 3000
maxdsiz 2063835136
maxdsiz_64bit 0X40000000
maxfiles 2048
maxfiles_lim 2048
maxssiz 0X800000
maxssiz_64bit 0X800000
maxswapchunks 2048
maxtsiz 0X4000000
maxtsiz_64bit 0X40000000
maxusers 512
ncallout 6000
nfile 22565
nkthread 6000
nproc 2048
nstrpty 60
shmmax 0X4000000

Any suggestion ??

[RAC_1]

Give the details for follows.

swpinfo -mat
glance -m, check for free memory
Is program 32/64 bit?? Do
file "prgram_name"

Anil

[Bill Hassell]

Your application is likely an old 32bit application and is subject to the severe limitations in addressing (compared to RAM sizes today). ENOMEM very seldom means that you've run out of RAM. Instead, it usually means that the program is asking for more memory than it is allowed use. maxdsiz and maxdsiz_64 are the global fences to keep programs from running away and stealing all the RAM. However, maxdsiz_64 *MUST* be larger than maxdsiz, so the first change is to increase maxdsiz_64 to 4Gb. Note that the 0x (hex) representation is just another way to show the limit. Change it to 4000000000 (4 billion). maxdsiz = 2Gb which is fine.

Now the signal 11 (SEGV) is likely due to the program not checking the return code from mmap and trying to use memory that was never assigned, a common programming error.

The mmap call seems to be referencing shared memory and similar limitations exist for 32bit programs using shared memory. The documentation for the programs should have indicated the size requirements for shared memory. But in case they left those out, change the shared memory parameters, specifically shmmax which is only 64megs. Most likely this is the problem. Unfortunately, the program did not provide a useful error message when mmap tried to report that it could not obtain the needed memory. Bump the kernel parameter shmmax up to 200-400 megs to allow the program to get what it needs.

[Jaideep Krishnan]

What's the PATH to 'swpinfo' and 'glance'?

$ file "prgram_name"
prgram_name: cannot open

$ getconf KERNEL_BITS
64

$ uname -srvm
HP-UX B.11.11 U 9000/800

Please suggest...
Thanks

[Bill Hassell]

/usr/sbin/swapinfo and /opt/perf/bin/glance. Note that swapinfo can only be run by root, and Glance is a separately priced product. There is a trial version of Glance on your Application CD/DVD.

[Jaideep Krishnan]

I set - maxdsiz_64=4000000000 and
shmmax=0X19000000

in /stand/system, rebooted the machine, and tried to start the process. The same error persits.

Here is the extract from /usr/sbin/swapinfo:

Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
dev 10485760 0 10485760 0% 0 - 1 /dev/vg00/lvol9
reserve - 82888 -82888
memory 3160672 9056 3151616 0%

..

[Jaideep Krishnan]

Also, here is the extract from /stand/system:

* Tunable parameters

STRMSGSZ 65535
dnlc_hash_locks 512
max_thread_proc 3000
maxdsiz 2063835136
maxdsiz_64bit 4000000000
maxfiles 2048
maxfiles_lim 2048
maxssiz 0X800000
maxssiz_64bit 0X800000
maxswapchunks 16384
maxtsiz 0X4000000
maxtsiz_64bit 0X40000000
maxusers 512
ncallout 6000
nfile 4097
nkthread 6000
nproc 2048
nstrpty 60
shmmax 0X19000000

Any thoughts?
Thanks.

[Jaideep Krishnan]

Hi, I increased shmmax to 0X19000000 and
maxdsiz_64bit to 4000000000 - its not really helping. Would I need to change certain other settings as well? Please suggest...

Also, I am able to bring up this product on certain other HP-UX machines that had similar settings (the original default settings). But the difference was, there weren't certain other products installed on them.

However as you can see from above, there is enough free memory, enough to start my product even if those other products were on the server.

I am wondering why it fails on certain machines and not failing on certain others - all machines mostly have similar settings.

I also ran gdb on the core file and it has to say the following:

#0 0xc1d29e3c in MemAllocPtr (pool=0x7b04e06a, size=2139039032,
flags=3243667608) at heap.c:991

Please suggest. Appreciate your suggestions. Thanks.