pthread_create fails with errno 11

[Thu Nguyen_2]

My application calls pthread_create() and the call returns errno 11 (resource temporarily unavailable)
Here is output of swapinfo:
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 12 1012 1% 0 - 1 /dev/vg00/lvol2
dev 1000 12 988 1% 0 - 1 /dev/vg01/lvswap2
dev 1000 12 988 1% 0 - 1 /dev/vg01/lvswap3
reserve - 1209 -1209
memory 2205 1306 899 59%
total 5229 2551 2678 49% - 0

Here is the output of mem_test. The code I pulled from HP website (it calls pstat_dynamic, pstat_getproc to get output):

'mem_test' after my application restart:

page size is 4096 bytes
physical memory size is 786432 pages, -1073741824 bytes
total virtual memory allocated is 132293 pages, 541872128 bytes
active virtual memory is 110740 pages, 453591040 bytes
total real memory is 140071 pages, 573730816 bytes
active real memory is 113028 pages, 462962688 bytes
free memory is 216026 pages, 884842496 bytes
malloc arena memory is 28100 bytes
free malloc ordinary buffer memory is 11680 bytes
free malloc small block memory is 0 bytes
nlist: No such file or directory

This process: 8365
This process is using 404254720 bytes of RAM.
This process is using 507277312 bytes of VM.
This process is using 391376896 bytes of data.


'mem_test' output during memory errors:

page size is 4096 bytes
physical memory size is 786432 pages, -1073741824 bytes
total virtual memory allocated is 289366 pages, 1185243136 bytes
active virtual memory is 15624 pages, 63995904 bytes
total real memory is 283146 pages, 1159766016 bytes
active real memory is 15372 pages, 62963712 bytes
free memory is 4734 pages, 19390464 bytes
malloc arena memory is 28100 bytes
free malloc ordinary buffer memory is 11680 bytes
free malloc small block memory is 0 bytes
nlist: No such file or directory

This process: 8365
This process is using 897740800 bytes of RAM.
This process is using 1011187712 bytes of VM.
This process is using 884207616 bytes of data.

Here are some kernel parameters:
STRMSGSZ 65535
dbc_max_pct 30
eqmemsize 2000
max_thread_proc 512
maxdsiz 1174405120
maxfiles 2048
maxfiles_lim 2048
maxssiz 0X04FB3000
maxssiz_64bit 0X04FB3000
maxswapchunks 3200
maxuprc 2048
maxusers 1000
maxvgs 50
msgmax 65535
msgmnb 65535
msgmni 9600
msgseg 32767
msgssz 32
msgtql 1800
nfile 153600
nflocks 6400

Also my application is in SHARED_MAGIC format. The total threads created is under the max_thread_proc.

I still have plenty of swap space, why the system returns with no more memory. Please help. thanks

[Steven E. Protter]

You are looking at memory.

Look at process space.

nprocs

max_thread_proc 512


Bump those two up.

Check shared memory segments. shmmax and such. If those are not big enough and have enough message segments and queues, you could have the problem you are having.

SEP

[Murray Yutzy]

I wish I would have some insight, but can only share a similar situation.

We have a farm of servers(20x4 cell dome partitions) performing the same functionality. Periodically one server exhibits a slowdown condition seen from a few hours after startup to 43 hours after startup. We've observed similar VM related metrics. Page faults increase considerably, (normally in the 1000's, increase to millions). The Active VM shows a reduced size(normally 16gb to 5gb). Assume this then drives the page faults up. We have been struggling on getting the right data collection mechanisms in place to capture the events leading up to this condition.