What dictates the maximum size of a process in HP-UX?

[Zinky]

Asking coz our developers are claiming there is a 600MB limit for a process (sz or vsz they point out). And they claim the solution is using the "chatr" comand:

chatr +q3p enable hpuxexe

Is'nt SZ/VSZ dictated by kernel parameters relating to stack, text and segment size limitations?

Are they correct in using the "chatr" approach?

[Patrick Wallek · Accepted solution]

Is the program that is running this 32-bit or 64-bit?

If it is 32-bit, it sounds as if they are bumping against the max available memory in Quadrant 2.

[Zinky]

32 bit app Patrick.

[Alessandro Pilati]

Nelson,
the approach seems correct coz you touch shared memory when you touch 4th quadrant.

Read this doc attached for further information.

Hope this helps.

Regards,
Alessandro

[TwoProc]

Kernel Parameters: maxdsiz, maxtsiz, maxssize.

Of these, maxdsiz is the one that generally needs the most "help". I usually have to bump up the stack size a little for custom developed apps. You can use glance to pull up their process and see what each piece's size is, and can actually watch it fail, and from that you can see which segment is getting too large. That is, if the memory consumption grows slowly enough.

[Zinky]

My kernel 32-bit text, stack and seg limits are 384M, 368M and 1024M respectively. The process of interest per measureware monitoring have not exceeded 600MB in size. Do you think we are hitting whatever limitations default quadrant has?

BTW, this executable uses oracle (compiled against Oracle libs - ProC?) and our developers are claiming there is an SQL-02103 error associated.

[A. Clay Stephenson]

Your problem is that maxssiz is much too large. Both the stack and dynamic data are allocated from the same 1GB quadrant in 32-bit applications. Let's suppose that you have maxdsiz set to 1GB and maxssiz set to 400MB, under the best of circumstances you could only dynamically allocate 600MB (and actually a little less) because 400MB (whether actually used or not) is reserved for the stack. Reduce your maxssiz to 32MB (and that is quite generous for well-written programs) and you instantly have >300MB of additional space. If you need to go beyond the 1GB quadrant boundary is when you need to compile with special options and use chatr.

[Zinky]

Thanks Clay.

So are you recommending reduction of maxssiz as a better alternative and we can forego the chatr approach?

[A. Clay Stephenson]

Yes (as long as you can live with about ~960 MB of dynamically allocated data). Then you won't have to add the chatr steps to your makefiles (and remove them later when you migrate to 64-bit land). The stack is used to store auto storage class data (essentially that's C speak for non-static variables local to a function) and to store the actual parameters of functions. Only extremely poorly written programs (even in 64-bit land) ever need stacks larger then 64MB and for most 32MB is more than adequate. About the minimum practical value (and the default setting by the way) is 8MB but 24-32MB is a very good all around setting.

There are a few exceptions that need large stacks but those are usually FORTRAN (or linked with FORTRAN libraries).

[TwoProc]

Good posting Clay. I didn't realize that just because a high setting of maxssiz was on the box, that it would actually be reserved - I just thought it was a max. I don't have anything set nearly that high though. Nelson, I agree with Clay, you really shouldn't need a huge stack size, unless someone is recursion crazy. Implement Clay's suggestion as a matter of course (even if you still can't make the program run afterwards, it is still too high).
Then you'll have to bump one or the other (I'd do dsiz first) independently until it runs.

I don't think you need to enable q3 - the man page (below) indicates that this allows you to get your data segment out past 1.9G. I don't think that this is your problem.
-------------
+q3p flag Control the flag bit setting to indicate how 32-bit
processes use the third quadrant as data space.

The enable flag sets the flag bit to indicate that 32-
bit processes use the third quadrant as a private data
space. By setting the bit, the private data space
increases from 1.9GB to 2.85GB for 32-bit processes.
----------------

So, I believe it is your kernel parameters that are locking you out.

Don't forget to examine your ulimits.
"ulimit -a"

[Bill Hassell]

As Clay mentioned, you cannot set maxssiz that large without affecting all of your 32bit programs. Only very poorly written programs will need hundreds of megs for a stack size (maxssiz) and your maxtsiz is outrageously high. The text size of a program is directly related to the number of instructions (not data) and 384megs would mean the source is millions of lines long and probably takes several hours to compile. maxtsiz needs only be slightly larger than the largest disk image of your executable. Most executables are less than 10 megs (hing: ll /usr/sbin|sort -rnk5). Set maxssiz to 8megs and maxtsiz to 64megs (normal settings).

Here are the basics:

Ordinary 32bit programs are limited to about 960 megs for local data or the value of maxdsiz, whichever is less.

Change the 32bit program to an EXEC_MAGIC program with the compiler options (I'm assuming a C program): cc -Ae -Wl -N -o myprog myprog.c and change maxdsiz to 1800 megs and now your program can reach about 1700+ megs.

+q3p option with chatr only works with 11.00 and 11.11 (latest patches) and will extend this limit to about 2700 megs (maxdsiz must be increased).

+q4p option will extend the limit to over 3500 megs (latest patches, does not work on 11.00) with some severe restrictions on shared memory.

So your 600Mb limit is coming from one of 4 limitations:

- maxdsiz is 600megs (you said maxdsiz is 1024megs) so that's not it,

- ulimit -d is set to 600000

- Not enough swap space for reservation

- maxdsiz_64 is 600megs - yeah, this is a zinger. If maxdsiz_64 is less than maxdsiz, then maxdsiz_64 limits the size of 32 and 64bit programs. Always make maxdsiz_64 several dozen Gb to avoid this hidden limitation.

To simplfiy everything, use the attached program code and compile/chatr it as shown in the comments. On a tiny A500 (rp2405) computer with 2Gb of RAM and 45Gb of swap space, I can get 900, 1700, 2700 and 3600 megs of local data (compiled/chatr'ed as documented) and 40Gb (!) of local data compiling as a 64bit process (maxdsiz_64=50Gb). You can get the same as long as you set maxdsiz accordingly and don't touch ulimit when you run the program.

32bit programs present quite complicated limitations when you start growing local and shared memory beyond a few hundred megs and it is the very reason that 64bit OS and programs exist.

[Zinky]

I have now a better understanding of the issue at hand.

I will hopefully now be better armed to respond to our applications folks.

Thanks y'all.. specially to A. Clay!