ksh limits - Max # of Env Variables and Contents

[Alzhy]

After looking at the script, indeed it declares and exports variables in the thousands! More than 5000 in fact.

And it actually I htink abends on a routine when one variable exhausts its assignabe values (long strings of xml paths, etc..).

XmlVal="$XmlVal $str"

So if the limit is ~4MB for a variable -- they're likely exhausting with that "no space" error message spewed out.

They're alleging that on a different HP-UX environment(likely 11.11 or 11.23), they're able to have this application run flawlessly due to their environment's "ulimit -d" being set to a whopping 4TB! I did check on my own 11.11 system and indeed "ulimit -d" can be set to 4TB whereas on 11.31 (ksh) it can't.

Here's my ksh environment ulimit:

alzhy@unix11 > ulimit -a

time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 4292870144
stack(kbytes) 376832
memory(kbytes) unlimited
coredump(blocks) 4194303

[Dennis Handly]

>They're alleging that on a different HP-UX environment(likely 11.11 or 11.23), they're able to have this application run flawlessly due to their environment's "ulimit -d" being set to a whopping 4TB!

The shell can't use more than 1 Gb. Except if on PA where you can use chatr(1) to get a few more.

[Bill Hassell]

> And it actually I htink abends on a routine when one variable exhausts its assignabe values (long strings of xml paths, etc..).

> XmlVal="$XmlVal $str"

> So if the limit is ~4MB for a variable -- they're likely exhausting with that "no space" error message spewed out.

The variable can hold a lot more. It's just the that the total of all the assigned variables has exceeded the 32 bit limit for the shell.

Note that there are several limits in play here -- the ulimit value is an environment limit. It cannot be made larger than the kernel parameter maxdsiz, the 32 bit program limit for local memory. While maxdsiz can be set to a very large value, it is simply a fence and does not enable a process to obtain more RAM than it's program type will allow.

So a plain vanilla 32bit program is limited to about 950 MB. That's where ksh lives. By compiling the program for EXEC_MAGIC, two of the 4 program quadrants can be used to allow up to 1850 MB to be allocated locally. Similarly, additional program modifications can add quadrant 3 and 4 for a maximum of about 3800 MB.

For the results on other versions of HP-UX, check to see if the ksh program is the original or someone compiled a different version of ksh. You can recall a command line and type CTRL-v to see the version. Only 1988 is supplied as /usr/bin/ksh for HP-UX. There is a 1993 version called dtksh.

So if the developers insist on stretching the limits of ksh, get a copy of the source code from http://kornshell.com/ and compile it for much larger limits. Then adopt this as the interpreter (named something else like ksh-ultra) for all your HP-UX versions.

[Alzhy]

Then WHY I am seeing this on my HP-UX 11.11 Systems:

alzhy@unix11 > ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 4292870144
stack(kbytes) 376832
memory(kbytes) unlimited
coredump(blocks) 4194303

The above means 4TB right!?

[Dennis Handly]

>Then WHY I am seeing this on my 11.11 systems:
>data(kbytes) 4292870144

The shell is broken.

>The above means 4TB right?

Only if you are naive and believe in documentation fairies. :-)

You should continue this ulimit discussion in your other thread:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1433676

[Alzhy]

Okay I did a mini KSH script to test how BIG of a value/string I can allocate to a ksh variable:

Strval="%strval `cat file`

file is exactly 1024 bytes.

It seems I can only allocate 825x1024 bytes ~ 844800 bytes. Then it complains of "no space"

Is there any trick like we an to tcsh?


Thanks.

[Bill Hassell]

All of the shells, (ksh, csh, POSIX sh, etc) are 32 bit programs, so even if you force maxdiz to terabytes, it makes no difference. All 32 bit programs have 4ea 1GB quadrants. The shells in HP-UX have a 1 GB quadrant limit. tcsh (as well as bash) aren't supplied with a standard install of HP-UX so you can get the source code and set your own limits.

To recap, all 32 bit programs are limited to 4 quadrants of 1GB each. This limit cannot be changed. The fence that can set a maximum limit for 32 bit programs is MAXDSIZ. If it is set to a value larger than 4 GB, then there it is effectively disabled as a limit and now the maximum for a program is based on the way the program is compiled. ulimit is a local limit to the session and is usually set to a low value to prevent bad programs from grabbing a lot of memory. Set the ulimit value to a large value means that the program can grow to the maximum available within the quadrants.

As mentioned in your other thread, 4TB is an error (the limit is controlled by MAXDSIZ and the 32 bit program limits) and a patch exists to fix it. Even if ulimit is 4 TB, the shells can only grow to about 960 MB before encountering the 1 GB addressing limit for the local data quadrant.

FYI, a program compiled as a 64bit executable does not have these limitations -- MAXDSIZ_64 is the local memory limit and multi-TB local RAM is not a problem (except for available RAM and swap space). Because the data model for 64bit programs is not the same as a 32bit program, compiling ksh (or bash or tcsh, etc) may not be easy as these programs may have constructs internally that may need a rewrite.

[Alzhy]

So what limits the assignable size/length to a ksh environment variable?

Is it as well resticted by the number of declared environmental variables or variables within a korn shell scripty?

[Bill Hassell]

> So what limits the assignable size/length to a ksh environment variable?

As I mentioned above, I empirically discovered I could assign about 200 MB to a single variable.

> Is it as well resticted by the number of declared environmental variables or variables within a korn shell scripty?

Like any interpreter, all the local data such as variables are stored in a local data which is expanded to accommodate each additional element. So if I assigned VAR1=<200MB of data> and then did the same for VAR2, VAR3 and VAR4, the shell would need about 800 MB to hold these 4 variables. Any attempt to assign VAR5=<200MB of data> would fail with a no memory error. So whether you assign a few large variables or 5000 small variables, the total of all the assigned data has to fit into the data quadrant, about 900 MB.