- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- ksh limits - Max # of Env Variables and Contents
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Forums
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2010 01:17 AM
06-04-2010 01:17 AM
ksh limits - Max # of Env Variables and Contents
And the apps crew are complaining they're unable to at a certain point.
ulimit is set to:
opsuser@SRV001> ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 3145728
stack(kbytes) 376832
memory(kbytes) unlimited
coredump(blocks) 4194303
I am thinking of bumping of kernel param to the 32bit maxdsiz value of 4GB.
But is there a limit to the above? Accdg to our apps crew, this new environment is more than twice as big as the previous where they have around ~2000 environment variables declared @ maxdsiz of 3GB.
TIA!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2010 01:33 AM
06-04-2010 01:33 AM
Re: ksh limits - Max # of Env Variables and Contents
it would be around
(4096*10000)/1024/1024 ~40MB
and I suppose the current default maxdsiz value
1GB should be OK you have 3GB I guess.
Apps crew said new environment is twise as big as previous, I suppose it mean database is big or it can support more user or whatever apps for.
Did apps owner see any error while they faced the issue.
BR,
Kapil+
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2010 01:49 AM
06-04-2010 01:49 AM
Re: ksh limits - Max # of Env Variables and Contents
Is there a limit to the value of a korn shell variable?
Coz one of their issues is they're claiming there scripts are doing:
Envar="$Envar newstr"
(in some loop).
Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2010 01:50 AM
06-04-2010 01:50 AM
Re: ksh limits - Max # of Env Variables and Contents
>>One-dimensional arrays are supported by the Korn shell. Arrays can have a maximum of
4096 elements
How can the variable more than 4096.
I am not sure if ksh93 also support more than that.
BR,
Kapil+
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2010 06:59 AM
06-04-2010 06:59 AM
Re: ksh limits - Max # of Env Variables and Contents
Time to use files or a database.
Or even shared memory?
At process creation the environ(5) is put on the stack.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-05-2010 04:59 AM
06-05-2010 04:59 AM
Re: ksh limits - Max # of Env Variables and Contents
Dennis gave you a better option. DO
not use such large Shell-based arrays. Very inefficient, not portable and prone
to hit limits in various Shells.
Some Shells offer larger array supports and
some do not even have such features.
For example, HP's POSIX-compliant Shell does
not offer such feature.
Here are the ones I know or heard of:
Bash: unlimited
Solaris /usr/xpg4/bin/sh: 4096
Ksh88: 1024
Ksh93: 4096
For example, HP-UX 11.23 Ksh is Ksh88.
Cheers,
VK2COT
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-05-2010 05:10 AM
06-05-2010 05:10 AM
Re: ksh limits - Max # of Env Variables and Contents
One more surprise. I just tested it
on my Fedora 11 server at home.
According to the manual page, Ksh on this server supports maximum of 4194304 elements
in indexed (one-dimensional) array.
Looked too big, but test showed it was
actually correct:
Test script on Fedora 11:
#!/bin/ksh -x
set -A MyArr
MyArr[1023]="hola"
MyArr[4194303]="should-work-on Fedora-11"
MyArr[4194304]="should-fail-on Fedora-11"
Let's run it:
# myscript
+ set -A MyArr
+ MyArr[1023]=hola
+ MyArr[4194303]='should-work-on Fedora-11'
myscript: line 6: MyArr: subscript out of range
So, different Shells on different operating systems really have very different
maximum number of indices for arrays.
Cheers,
VK2COT
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-05-2010 07:37 AM
06-05-2010 07:37 AM
Re: ksh limits - Max # of Env Variables and Contents
Arrays are very useful but as you have seen, there are too many variations on limits. I will use them for a few dozen items but would always switch to a file for larger values.
I do use variables to store large amounts of text (ie, VG=$(vgdisplay -v) which can then be reused as a memory-based resource with no further overhead. I have assigned 200 MBytes of text to a variable in the POSIX and ksh shell. Unfortunately, there does not appear to be any readable limits for each of the shells so you have to test big variables to see if it was successful. And of course, each variable's space is cumulative in the shell's work area, so you won't necessarily get 10 variables at 200 MB each.
> bumping of kernel param to the 32bit maxdsiz value of 4GB.
Won't do anything for any of the shells. They are compiled without the option to map additional quadrants. To expand heap memory will require changing the executable.
If the team really wants to use a shell with massive values (and does not care about portability), I would get Bash source code and compile it as a 64bit executable. Then 10 or 20 Gbytes can easily be allocated to the shell. But is it the appropriate tool? After all, shells (including Perl and even Java) are interpreters with a lot of overhead to run the scripts.
Bill Hassell, sysadmin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-05-2010 09:32 AM
06-05-2010 09:32 AM
Re: ksh limits - Max # of Env Variables and Contents
> Bill: After all, shells (including Perl and even Java) are interpreters with a lot of overhead to run the scripts.
As for Perl, it is actually both a compiler and an interpreter. Perl has no built-in size limits and allocates and deallocates memory automatically. It lies closer to C (upon which it is built) than any shell.
Regards!
...JRF...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-05-2010 01:59 PM
06-05-2010 01:59 PM
Re: ksh limits - Max # of Env Variables and Contents
Make sure you don't export these huge variables.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-06-2010 09:24 AM
06-06-2010 09:24 AM
Re: ksh limits - Max # of Env Variables and Contents
And it actually I htink abends on a routine when one variable exhausts its assignabe values (long strings of xml paths, etc..).
XmlVal="$XmlVal $str"
So if the limit is ~4MB for a variable -- they're likely exhausting with that "no space" error message spewed out.
They're alleging that on a different HP-UX environment(likely 11.11 or 11.23), they're able to have this application run flawlessly due to their environment's "ulimit -d" being set to a whopping 4TB! I did check on my own 11.11 system and indeed "ulimit -d" can be set to 4TB whereas on 11.31 (ksh) it can't.
Here's my ksh environment ulimit:
alzhy@unix11 > ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 4292870144
stack(kbytes) 376832
memory(kbytes) unlimited
coredump(blocks) 4194303
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-06-2010 10:50 AM
06-06-2010 10:50 AM
Re: ksh limits - Max # of Env Variables and Contents
The shell can't use more than 1 Gb. Except if on PA where you can use chatr(1) to get a few more.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-06-2010 03:59 PM
06-06-2010 03:59 PM
Re: ksh limits - Max # of Env Variables and Contents
> XmlVal="$XmlVal $str"
> So if the limit is ~4MB for a variable -- they're likely exhausting with that "no space" error message spewed out.
The variable can hold a lot more. It's just the that the total of all the assigned variables has exceeded the 32 bit limit for the shell.
Note that there are several limits in play here -- the ulimit value is an environment limit. It cannot be made larger than the kernel parameter maxdsiz, the 32 bit program limit for local memory. While maxdsiz can be set to a very large value, it is simply a fence and does not enable a process to obtain more RAM than it's program type will allow.
So a plain vanilla 32bit program is limited to about 950 MB. That's where ksh lives. By compiling the program for EXEC_MAGIC, two of the 4 program quadrants can be used to allow up to 1850 MB to be allocated locally. Similarly, additional program modifications can add quadrant 3 and 4 for a maximum of about 3800 MB.
For the results on other versions of HP-UX, check to see if the ksh program is the original or someone compiled a different version of ksh. You can recall a command line and type CTRL-v to see the version. Only 1988 is supplied as /usr/bin/ksh for HP-UX. There is a 1993 version called dtksh.
So if the developers insist on stretching the limits of ksh, get a copy of the source code from http://kornshell.com/ and compile it for much larger limits. Then adopt this as the interpreter (named something else like ksh-ultra) for all your HP-UX versions.
Bill Hassell, sysadmin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-06-2010 05:49 PM
06-06-2010 05:49 PM
Re: ksh limits - Max # of Env Variables and Contents
alzhy@unix11 > ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 4292870144
stack(kbytes) 376832
memory(kbytes) unlimited
coredump(blocks) 4194303
The above means 4TB right!?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-06-2010 09:23 PM
06-06-2010 09:23 PM
Re: ksh limits - Max # of Env Variables and Contents
>data(kbytes) 4292870144
The shell is broken.
>The above means 4TB right?
Only if you are naive and believe in documentation fairies. :-)
You should continue this ulimit discussion in your other thread:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1433676
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-06-2010 10:03 PM
06-06-2010 10:03 PM
Re: ksh limits - Max # of Env Variables and Contents
Strval="%strval `cat file`
file is exactly 1024 bytes.
It seems I can only allocate 825x1024 bytes ~ 844800 bytes. Then it complains of "no space"
Is there any trick like we an to tcsh?
Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2010 03:10 AM
06-07-2010 03:10 AM
Re: ksh limits - Max # of Env Variables and Contents
To recap, all 32 bit programs are limited to 4 quadrants of 1GB each. This limit cannot be changed. The fence that can set a maximum limit for 32 bit programs is MAXDSIZ. If it is set to a value larger than 4 GB, then there it is effectively disabled as a limit and now the maximum for a program is based on the way the program is compiled. ulimit is a local limit to the session and is usually set to a low value to prevent bad programs from grabbing a lot of memory. Set the ulimit value to a large value means that the program can grow to the maximum available within the quadrants.
As mentioned in your other thread, 4TB is an error (the limit is controlled by MAXDSIZ and the 32 bit program limits) and a patch exists to fix it. Even if ulimit is 4 TB, the shells can only grow to about 960 MB before encountering the 1 GB addressing limit for the local data quadrant.
FYI, a program compiled as a 64bit executable does not have these limitations -- MAXDSIZ_64 is the local memory limit and multi-TB local RAM is not a problem (except for available RAM and swap space). Because the data model for 64bit programs is not the same as a 32bit program, compiling ksh (or bash or tcsh, etc) may not be easy as these programs may have constructs internally that may need a rewrite.
Bill Hassell, sysadmin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2010 07:04 AM
06-07-2010 07:04 AM
Re: ksh limits - Max # of Env Variables and Contents
Is it as well resticted by the number of declared environmental variables or variables within a korn shell scripty?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2010 07:48 AM
06-07-2010 07:48 AM
Re: ksh limits - Max # of Env Variables and Contents
As I mentioned above, I empirically discovered I could assign about 200 MB to a single variable.
> Is it as well resticted by the number of declared environmental variables or variables within a korn shell scripty?
Like any interpreter, all the local data such as variables are stored in a local data which is expanded to accommodate each additional element. So if I assigned VAR1=<200MB of data> and then did the same for VAR2, VAR3 and VAR4, the shell would need about 800 MB to hold these 4 variables. Any attempt to assign VAR5=<200MB of data> would fail with a no memory error. So whether you assign a few large variables or 5000 small variables, the total of all the assigned data has to fit into the data quadrant, about 900 MB.
Bill Hassell, sysadmin