32 bit process terminates when data segement exceeds 64-bit data segement limit

[Jimmy Tom_2]

I am having issues with my apps and it seems to have to do with the kernel parameter.

The kernel parameter that limits process data size for 32-bit processes is called maxdsize and is set to 2GB.
For 64-bit processes it is called maxdsize_64bit it is set to 1GB.
Our 32-bit process dies when it hits the 1GB limit.
(process also dies at 512MB data size when maxdsize_64bit is set to 512MB).

Our server is an rp3440 64 bit machine running hpux 11i The process is 32 bit compiled with +DA2.0 and linked with -N option to overcome the 1GB data limit.
(On former K-class servers the process normally went well beyond 1GB)

Any patches or parameter I need to set?
Thx in advance.

[Patrick Wallek]

If the 64-bit parameter is set lower than the 32-bit parameter then the value of the 64-bit parameter will be used.

To solve this you should increase the maxdsiz_64 to be at least 2GB.

[A. Clay Stephenson]

If any of the maxXsiz_64bit values are lower than their maxXsiz 32-bit counterparts, the smaller of the two wins. All of your xxx_64bit values should be at least as large as their 32bit cousins.

One of the strange things that happens is that ulimit always uses the smaller of the two values.

[Don Morris_1]

Not that strange once you know why.

When a 64-bit process forks off a 32-bit process, you have to make a decision somehow. The rlimit standard states that children should inherit the limits of their parents -- but the 32-bit maximum may be much lower than the 64-bit limit (even if the parent didn't change the limit). So the most restrictive of the limits is used (parent limit vs. 32-bit system-wide limit).

What happens here is that shells are almost always 32-bit processes... and are spawned from a 64-bit process (init). So init starts with the 64-bit (lower) limit, forks 32-bit children... which inherit the lower limit of the 64-bit parent instead of the system-wide 32-bit limit. All processes spawned from the shell inherit the limit of their parent (the shell)... so it propagates from there.

[A. Clay Stephenson]

It's not that obvious to me but I'm dumb. This problem is when the 64-bit values are less than the 32-bit values and yet the 32-bit executables are limited by the smaller 64-bit values which should be ignored by a 32-bit executable --- including the shell.