ksh limits - Max # of Env Variables and Contents

[Alzhy]

I'm dealing with a humongous applications that need to declare ~ 5000 environment variables with a few environment variables set to more than "4096 character" values.

And the apps crew are complaining they're unable to at a certain point.

ulimit is set to:

opsuser@SRV001> ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 3145728
stack(kbytes) 376832
memory(kbytes) unlimited
coredump(blocks) 4194303

I am thinking of bumping of kernel param to the 32bit maxdsiz value of 4GB.

But is there a limit to the above? Accdg to our apps crew, this new environment is more than twice as big as the previous where they have around ~2000 environment variables declared @ maxdsiz of 3GB.


TIA!

[Kapil Jha]

Even if you consider 10000 variable or around 4096 characters
it would be around
(4096*10000)/1024/1024 ~40MB

and I suppose the current default maxdsiz value
1GB should be OK you have 3GB I guess.

Apps crew said new environment is twise as big as previous, I suppose it mean database is big or it can support more user or whatever apps for.

Did apps owner see any error while they faced the issue.

BR,
Kapil+

[Alzhy]

Then it is most likely they're declaring more or these environment variables are just assigned way too outrageously looooong values.

Is there a limit to the value of a korn shell variable?

Coz one of their issues is they're claiming there scripts are doing:

Envar="$Envar newstr"

(in some loop).

Thanks.

[Kapil Jha]

Ok I got smthing more

>>One-dimensional arrays are supported by the Korn shell. Arrays can have a maximum of
4096 elements

How can the variable more than 4096.
I am not sure if ksh93 also support more than that.

BR,
Kapil+

[Dennis Handly]

>declare ~ 5000 environment variables with a few environment variables set to more than "4096 character" values.

Time to use files or a database.
Or even shared memory?

At process creation the environ(5) is put on the stack.

[VK2COT]

Hello,

Dennis gave you a better option. DO
not use such large Shell-based arrays. Very inefficient, not portable and prone
to hit limits in various Shells.

Some Shells offer larger array supports and
some do not even have such features.
For example, HP's POSIX-compliant Shell does
not offer such feature.

Here are the ones I know or heard of:

Bash: unlimited

Solaris /usr/xpg4/bin/sh: 4096

Ksh88: 1024

Ksh93: 4096

For example, HP-UX 11.23 Ksh is Ksh88.

Cheers,

VK2COT

[VK2COT]

Hello,

One more surprise. I just tested it
on my Fedora 11 server at home.
According to the manual page, Ksh on this server supports maximum of 4194304 elements
in indexed (one-dimensional) array.
Looked too big, but test showed it was
actually correct:

Test script on Fedora 11:

#!/bin/ksh -x

set -A MyArr
MyArr[1023]="hola"
MyArr[4194303]="should-work-on Fedora-11"
MyArr[4194304]="should-fail-on Fedora-11"

Let's run it:

# myscript

+ set -A MyArr
+ MyArr[1023]=hola
+ MyArr[4194303]='should-work-on Fedora-11'
myscript: line 6: MyArr: subscript out of range

So, different Shells on different operating systems really have very different
maximum number of indices for arrays.

Cheers,

VK2COT

[Bill Hassell]

HP-UX continues to provide both ksh88 and ksh93 through 11.31. ksh93 is cleverly hidden at /usr/dt/bin/dtksh, so currently, ksh88 is the HP-UX ksh.

Arrays are very useful but as you have seen, there are too many variations on limits. I will use them for a few dozen items but would always switch to a file for larger values.

I do use variables to store large amounts of text (ie, VG=$(vgdisplay -v) which can then be reused as a memory-based resource with no further overhead. I have assigned 200 MBytes of text to a variable in the POSIX and ksh shell. Unfortunately, there does not appear to be any readable limits for each of the shells so you have to test big variables to see if it was successful. And of course, each variable's space is cumulative in the shell's work area, so you won't necessarily get 10 variables at 200 MB each.

> bumping of kernel param to the 32bit maxdsiz value of 4GB.

Won't do anything for any of the shells. They are compiled without the option to map additional quadrants. To expand heap memory will require changing the executable.

If the team really wants to use a shell with massive values (and does not care about portability), I would get Bash source code and compile it as a 64bit executable. Then 10 or 20 Gbytes can easily be allocated to the shell. But is it the appropriate tool? After all, shells (including Perl and even Java) are interpreters with a lot of overhead to run the scripts.

[James R. Ferguson]

Hi:

> Bill: After all, shells (including Perl and even Java) are interpreters with a lot of overhead to run the scripts.

As for Perl, it is actually both a compiler and an interpreter. Perl has no built-in size limits and allocates and deallocates memory automatically. It lies closer to C (upon which it is built) than any shell.

Regards!

...JRF...

[Dennis Handly]

>Bill: Then 10 or 20 Gbytes can easily be allocated to the shell.

Make sure you don't export these huge variables.