HP-UX 11.31, maxdisz_64bit affecting malloc?

[jmgallag]

Hi,

We have a group of application servers that should be indentically configured. They run application server processes, one process per user. The servers are round robin load balanced, so over time, the loading over each server should be the same. One of the servers has consistently higher memory utilization than the others. We have been scratching our heads over this for a while, until this week, one of the admins noticed that maxdsiz_64bit is set to 32GB on the server with high utilization, while it is set to 16GB on the others (all of these servers have 64GB RAM.) None of us thought that should affect how the app consumes memory, but since we had no other ideas, we ran pmap against an application process on the 32GB maxdsiz server, and also against one of the 16GB maxdsiz servers. I was surprised to find that the smallest mmap'ed allocation on the 32GB maxdsiz server was 64KB, while the smallest on the other server was 8K.

Some quick math verified that this difference accounts for the additional memory consumption on the server in question. For example, from the 32GB maxdsiz server:

9fffffffb0380000    64K    64K PD       rwx [anonymous]
9fffffffb0390000    64K    64K PD       rwx [anonymous]
9fffffffb03a0000    64K    64K PD       rwx [anonymous]
9fffffffb03b0000    64K    64K PD       rwx [anonymous]
9fffffffb03c0000    64K    64K PD       rwx [anonymous]
9fffffffb03d0000    64K    64K PD       rwx [anonymous]
9fffffffb03e0000    64K    64K PD       rwx [anonymous]

From the 16GB maxdsiz server:

9fffffffbbe9c000     8K     8K PD       rwx [anonymous]
9fffffffbbe9e000     8K     8K PD       rwx [anonymous]
9fffffffbbea0000     8K     8K PD       rwx [anonymous]
9fffffffbbea2000     8K     8K PD       rwx [anonymous]
9fffffffbbea4000     8K     8K PD       rwx [anonymous]
9fffffffbbea6000     8K     8K PD       rwx [anonymous]
9fffffffbbea8000     8K     8K PD       rwx [anonymous]

Can anyone shed any additional light on this behavior? I have not found any mention of this in any of the docs or white papers I have referenced.

Thanks,
Jim

[Dennis Handly]

>the smallest mmap'ed allocation on the 32GB maxdsiz server was 64KB,

Since mmap(2), this is not likely to be used by malloc(3), which uses brk(2).

So who is calling mmap?  And if it is using a larger size, there just should be less of them, if filled with the same amount of data.

[jmgallag]

I'm quite sure these mmap'ed regions are resulting from calls to malloc.

I agree with you as to what should happen, but that is not what we are seeing.

[Dennis Handly]

>I'm quite sure these mmap'ed regions are resulting from calls to malloc.

While I've seen this happen with LARGE requests on Linux but I'm not sure how it can happen with the default malloc(3).

And I don't think mmap(2) is used for the _M_ARENA_OPTS or _M_CACHE_OPTS either.

[ranganath ramachandra]

The listings (apparently from pmap and pertaining to a single process each)  that you have posted seem to be simply extracts and not complete, so whether these allocations total to a bigger size on one machine than on the other cannot be determined from the listings.

Are the base page size settings on the two machines different?

/usr/sbin/kctune base_pagesize

Do the number of these mmap regions increase over time for a given process?