crt0,ERROR: mmap failed for dld (data) errno:000000012

[Kathy McGohan_1]

This system is a K400 running HP-UX 11.0.
3 GB of Physical memory
3 1/2 GB of swap
Database is Supra. Batch files run at night and it has for 5 years without any problems until this week. Cannot run the batch jobs now without getting lots of error messages. After about an hour after the batch jobs run, we run out of Physical memory.
When doing a swapinfo -tam, total memory used is 100%, although swap remains at 0% used.
We have to reboot the system to clear out memory, but then when the batch jobs run the next night, same thing happens.

The cron log has put out error messages when these jobs run, such as:

/usr/lib/dld.sl: Call to mmap() failed - BSS /usr/lib/libnsl.1
/usr/lib/dld.sl: Not enough space
sh[2]: 6035 Abort
/usr/lib/dld.sl: Call to mmap() failed - BSS /usr/lib/libnsl.1
/usr/lib/dld.sl: Not enough space
sh[2]: 6038 Abort
/usr/lib/dld.sl: Can't allocate memory Strings
/usr/lib/dld.sl: Not enough space
/usr/lib/dld.sl: Can't allocate memory /usr/lib/libc.2
/usr/lib/dld.sl: Not enough space
/usr/lib/dld.sl: Call to mmap() failed - ZEROES /usr/lib/libc.2
/usr/lib/dld.sl: Not enough space
/usr/lib/dld.sl: Call to mmap() failed - BSS /usr/lib/libnsl.1
/usr/lib/dld.sl: Not enough space
/usr/lib/dld.sl: Call to mmap() failed - DATA /usr/lib/libdld.2
/usr/lib/dld.sl: Not enough space
sh[2]: There is not enough memory available now.
sh[2]: 6783 Abort
fork: Not enough space

I looked at /usr/include/sys/errno.h and it says that errno12 is "Not Enough Core".

What does this mean?

***Also, in the messages file, it says "Can't get Kernel Namelist".

Could you also please tell me what this means?

Thanks in advance.

[A. Clay Stephenson · Accepted solution]

Hi Kathy:

At the very least you need to increase maxdsiz (or maxdsiz_64 if this is a 64-bit application.) What has happened, is that recently a memory-mapped file (I suspect a very heavily used table or the dictionary itself in the database) has finally grown large enough to exceed the process memory limit (maxdsiz). Increase it and generate a new kernel and that should get you going. You probably need to look at adding more memory to your system as well but maxdsiz is your immediate problem.

Clay

[James R. Ferguson]

Hi Kathy:

What's different - patches applied? new application code?

Your description genearlly fits a case where you need to increase swap or perhaps 'maxdsiz'. You might also want to monitor processes for memory leaks. You can do this by watching the system with:

# UNIX95= ps -e -o ruser,vsz,pid,args|sort -rnk2

...and seeing if a paricular process's virtual memory size (vsx) grows substantially over time.

...JRF...

[Kathy McGohan_1]

Clay and James,
This is a 32 bit system and no patches or application codes have been put on in a long time.
Thanks for responding so quickly. I will be presenting your answers to my co-workers and we will have to schedule a downtime to make this change with maxdsiz. What is your suggestion for maxdsiz to be set at? Right now it is set to the default. Maybe 256MB?
I will assign point later after we can make the change so that I can let you know how it goes.
Again, thanks.

[James R. Ferguson]

Hi Kathy:

'maxdsiz' (for 32-bit machines) has a minimum value of 0x400000 (4MB), a maximum of 0x7B03A000 (~ 2GB) and a default of 0x4000000 (64MB).

...JRF...

[A. Clay Stephenson]

Hi Kathy:

64MB is the default value so 256-512MB should do the trick. If you don't have it set already, I would also set swapmem_on=1. Setting this value consumes no additional resources, it is simply a 'fence' for processes. The only downside to setting it much above 512MB is that real memory hogs or memory leakers eat up more system-wide resources before they are killed.

Clay

[Kathy McGohan_1]

Guess what? For some reason, the batch jobs all ran fine last night. We will continue to monitor all of this, but for right now, we are not permitted to make any changes. I really appreciate your answers and I would like to go with increasing maxdsiz as you have suggested. Thanks again for your help.