Unable to change the ulimit data and stack

[Ra Jose]

Hello friends,

I have a requirement from peoplesoft user to
change the ulimit. The need is to change the
data and stack of poeplesoft user as follows -
data(kbytes) 2097152
stack(kbytes) 524288

In order to change these values,I have to
change the system-wide kernel param maxdsiz
and maxssiz. The values we have on the sysem
(rx8640 running 11iv2 with 40GB memory) are -

root@f1e1pa07NEW:kctune|egrep -i "maxdsiz|maxssiz"|grep -v pa
maxdsiz 1073741824 Default Immed
maxdsiz_64bit 2147483648 2147483648 Immed
maxssiz 134217728 134217728 Immed
maxssiz_64bit 1073741824 1073741824 Immed

For the ulimit (data area), I changed maxdsiz
(doubled it), to 2147483648 which is 2 GB.
My ulimit (data area) should have picked this
but it shows strange output when you type ulimit -a. Here is o/p of ulimit -a

time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 4292870144
stack(kbytes) 262144
memory(kbytes) unlimited
coredump(blocks) 4194303

As you see, ulimit (data area) became almost
4 TB instead of 2 GB.

Has anyone noticed this situation? Is there anything I am doing wrong? This happens on
both 11iv1 and 11iv2.

For the ulimit (stack), I am able to double it.
Looks like I can't go beyond 393MB due to
restriction on maxssiz kernel param. The user
is requesting 512 MB. Is there anyway to increae
t this value?

Thank you all for your responses.

Ra Jose.

[Don Morris_1 · Accepted solution]

No, I've never seen that before. Try compiling the attached program (cc +DD32 -o data_inf data_inf.c) and replying with the output.

I suspect either your shell isn't handling values greater than an int32_t [signed 32-bit integer, which has a maximum positive value of 2Gb - 1] correctly or you've found a bug in the rlimit code I've never heard of. Calling getrlimit() directly should eliminate that question.

Sample output [on v3, which is all I have at the moment loaded up]:
# kctune maxdsiz=2147483648
==> Update the automatic 'backup' configuration first? n
* Future operations will ask whether to update the backup.
* The requested changes have been applied to the currently
running configuration.
Tunable Value Expression Changes
maxdsiz (before) 1073741824 Default Immed
(now) 2147483648 2147483648
# kctune maxdsiz maxdsiz_64bit
Tunable Value Expression Changes
maxdsiz 2147483648 2147483648 Immed
maxdsiz_64bit 4294967296 Default Immed
# ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 1048576
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048
# sh
# ulimia -a
sh: ulimia: not found.
# ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 2097152
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048
# cd /tmp
# cc +DD32 -o data_inf data_inf.c
# ./data_inf
My current Data hard/soft limits: 0x80000000 / 0x80000000 bytes.
Data soft limit now: unlimited
Data hard limit now: unlimited

# kctune maxdsiz=3*1024*1024*1024
==> Update the automatic 'backup' configuration first? n
* Future operations will ask whether to update the backup.
* The requested changes have been applied to the currently
running configuration.
Tunable Value Expression Changes
maxdsiz (before) 2147483648 2147483648 Immed
(now) 3221225472 3*1024*1024*1024
# ./data_inf
My current Data hard/soft limits: 0xc0000000 / 0xc0000000 bytes.
Data soft limit now: unlimited
Data hard limit now: unlimited
# sh
# ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 3145728
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048

Also note that you may get a failure to set Unlimited (this was a v3 enhancement, I don't believe v2 or v1 allow it). You also have to be superuser or otherwise appropriately privileged to do so (any raising of the limit above the system set rlim_max requires this privilege.... and Infinity is larger than max pretty much by definition).

As far as the stack size -- no, there's no way to override the kernel maximum. I'm sorry if the user thinks they need a half of a Gb for a stack -- but I think they actually need to revisit their programming practices. First, they should be aware that the virtual addresses are _reserved_ for the stack size -- so if you go with a 512Mb stack *everyone on the system by default is capped at 512Mb of data*. It takes playing some severe tricks with 32-bit process layout (text/data sharing, q3/q4 private or MPAS model) to get the data space back. Second -- malloc() and proper failures is always better than allocating large arrays on the stack and hoping you don't SIGSEGV and get killed.

[Don Morris_1]

And I'm so busy explaining things I don't attach the code... [no points on this followup, obviously].

[Ra Jose]

Thank you Don.

I got the program, compiled and ran it.

Here is the o/p -
My current Data hard/soft limits: 0x80000000 / 0x80000000 bytes.
setrlimit failure: Not owner

Looks like your explanation makes sense. May be ksh does not display it right. From another reseach, I came to know that this is not a problem for posix sh (/sbin/sh) or
csh. It is however a problem for ksh and I believe it is fixed by this ksh patch
for IA (PHCO_37284) and PA (PHCO_37283). I am currently looking into these patches and see if it works. I will keep posted.

Ra Jose.

[Ra Jose]

Yes I tested these patches and it worked.

I tested on PA machine and the ksh showed
correct ulimit values for data even when
> 2G.

Don, your input was valuable as you are right the ulimit has limitations when data
goes beyond 2G and also I am going to ask the peoplesoft why they want 512MB stack as your
explanation makes sense.

Thank you for your answer.

Ra Jose.

[Ra Jose]

The problem is resolved by patches.

[Dennis Handly]

>I have a requirement from peoplesoft user to change the ulimit. The need is to change the data and stack of peoplesoft user as follows -
data(kbytes) 2097152
stack(kbytes) 524288

You absolutely don't want to increase the stack to 500 Mb. That will remove that much from the heap for ordinary applications!

[Ra Jose]

Thank you Dennis.

The peoplesoft engineer insists on increasing
stack to 512 MB. He thinks that it only a
limit in case jdenet kernel process needs
it. They are having issues that thier kernel
process runs for a while (2 to 3 hrs) and
then suddeny dies. They have been able to
resolve the issue by increasing the stack.
His impression is that it only a limit and
does not make every single process consume
that much memory. Correct me if I am wrong,
if I increase the stack to 512 MB, then isn't
every process started on the system consumes
512 MB of stack?

Thanks again

Ra Jose.

[Don Morris_1]

First -- he can't be talking about a kernel process per se. A kernel thread uses an entirely different stack (UAREA) than the stack that maxssiz controls. Hence changing maxssiz wouldn't have any effect on a kernel process. I have to assume he meant "system" process instead of kernel in this case.

Second, please inform him that it isn't physical memory that gets consumed by the increased limit. While he is correct that this is a "fence line" as it were -- the way this works because the stack grows from low to high addresses on PA (and from high to low on IPF - but the RSE stack grows from low to high to meet it, so you get the same effect). Hence the stack always starts maxssiz [or more precisely the rlim_max[RLIMIT_STACK] before the end of the private data space, and all virtual addresses between the base of the stack and the end of the private data space are assumed to be intended for the stack.

To do otherwise would have a meaningless limit -- since the stack has to be virtually contiguous from a fixed point in user space, it _must_ be allowed the full range of virtual addresses from Base to Base+maxssiz if maxssiz is truly allowed to be the maximum size of the stack.

The problem here is that since nothing else can go between Base and Base+maxssiz -- you can't put Data, MMAP, or any other type of private object there. It is stack and nothing else. Since there is (by default) only 1Gb of total virtual address space for dynamic private data of 32-bit processes on HP-UX that means that maxssiz worth of that 1Gb is used for the stack and can not be used for anything else.

Hence my original statement that setting maxssiz to 512Mb mandates that Private data objects other than stack are by default limited to 512Mb as well (actually, I think it would be 512Mb - 4096 bytes [there's a single page that gets reserved as a Red Zone to distinguish between the stack and other objects]). The physical memory availability is unchanged -- but that's meaningless if you literally don't have the virtual range to map it into the process.

Hopefully this is sufficiently clear to drive the point home -- please feel free to ask again if more clarity is needed.

I'd also think that if a normally well behaved process (and I find myself rather curious what the "usual" stack size of the process is) suddenly requires 512Mb of stack that the core files generated should be checked for a recursion or other typical problem. This sounds very much like a class of fairly common and easy bugs, and bandaiding the situation by allowing the but to just eat a lot more assumes:

1) The bug won't reoccur and it won't need an even BIGGER stack..

2) The bug doesn't trigger or signal other incorrect code -- you may not be able to trust that the process is functioning correctly anyway.

3) It is worth it to throttle everything else on the system for this one adverse behavior.

[Ra Jose]

Thank you Don. You are right that the kernel process they mean is not the HP-UX kernel, but the peoplesoft jdenet processes. I do not have knowledge about the internals of peoplesoft application, here is how it appears when I do a ps -ef
psftb811 2424 2375 0 Mar 25 ? 0:26 jdenet_k 6013
psftb811 13707 2375 0 Mar 25 ? 2:37 jdenet_k 6013
psftb811 2963 2375 0 Mar 25 ? 13:15 jdenet_k 6013
psftb811 2502 2375 0 Mar 25 ? 0:26 jdenet_k 6013
psftb811 24501 2375 0 Mar 25 ? 0:22 jdenet_n 6013

I have shown few lines. There are about 36 such jdenet_k and 29 jdenet_n processes. I beleive the jdenet_k is the one what they call their kernel process.

You have given clear picture of how 32-bit processes are handled. This app runs on rx8640, 11iv2. What I understand is, there is a total of 1 GB space allocated to a 32-bit process of which if maxssiz is 512MB, then out of 1 GB, 512 MB is used by stack alone and nothing else. Whatever the process has such as data space and other objects have to fit in a 512 MB space. To state it in
other terms, if my stack is let's say 128MB, then the process has 1GB-128MB less 1 page for red zone available for data/user space.

Is it right?

So, one thing is clear. By increasing maxssiz
to 512MB is not taxing on the memory resource. It simply allocates more space to stack thereby limiting the data space.

Thank you,

Ra Jose.

[Don Morris_1]

Yes, that is an accurate summary for the default case. (And it is safe to assume that most binaries are compiled with the defaults).

I'm attaching a C program you could use to determine more details on these jdenet_k processes if you wish. Compile as either:

cc +DD64 -o object_dump object_dump.c

or

cc +DD32 -D_PSTAT64 -o object_dump object_dump.c

[Obviously rename as you wish].

If you already know the group id (I don't recognize exactly which column is which in the ps -ef output, sorry) you can run:

object_dump -v -g

which will give the per-process breakdown by type - you'll be wanting the "MAIN STACK" object for each. That will let you know exactly what these processes are using virtually (and physically) towards maxssiz. [The output is in pages, so you need to multiply by the reported "System page size" to compare directly with maxssiz].

If you need to figure out the group id, just

object_dump -v |grep GID|grep 2502

will find it, assuming that 2502 process is still there.

If you want to prove to them about the virtual address shift of the stack (and one would think they'd take your word for it given that you're asking the source), if you run object_dump with "-v -v" (i.e. verbosity 2 or higher) you get the virtual address and range of each object type:

Object 42: MAIN STACK at VA 0x7f800000 to VA 0x7fffffff.
VIRT: 2048 PHYS: 8 LOCKED: 0 SWAP: 8

You'll see the base shift lower on processes started after maxssiz changes (the tunable leaves running processes alone), but the end remains at the end of the process private data space.

[Ra Jose]

Thank you Don. I compiled and ran this.
I got lot of output. As you mentioned, I do
see the MAIN STACK how much virtual and
physical being used. It is taking little time
for me to digest. I am going to continue on
this tomorrow and will give you feedback.
This tool is awesome and gives breakings
of memory used by the process. Thank you
again.

Ra Jose.

[Dennis Handly]

>The peoplesoft engineer insists on increasing stack to 512 MB. He thinks that it only a limit in case jdenet kernel process needs it. They are having issues that their kernel process runs for a while (2 to 3 hrs) and then suddenly dies. They have been able to resolve the issue by increasing the stack.

It may just take longer to abort.
Here is where a stack trace of the abort will help.

>if I increase the stack to 512 MB, then isn't every process started on the system consumes 512 MB of stack?

And Don says, it consumes the virtual address range.

Of course if you really really need it for that one unmutual process, you can always use ulimit -s to reduce it for ordinary mutual users.