topic Re: Error: 'no space' in Ksh88 in Operating System - HP-UX

Error: 'no space' in Ksh88

Victor Akamine — Tue, 19 Oct 2010 16:23:46 GMT

Hi,

I need a help because I've a script (.ksh). This script merge many archives (xml) and concatenates all in one.

But the problem happens when I need to concatenates a lot of archives and the error 'no space' ocurrs.

I try to increase the filesystem's capacity.

----------------------------
Number of Archives: 104036
Total Archives size: 26 Gb
FileSystem size: 74 Gb
----------------------------

Even so, the error is still 'no space'.

I use ksh88, have any problem about it?

This is the error line in sysout:
/rating/diario/scripts//xml_aggregate_to_sdl.ksh[110]: no space

Have any idea to solve my problem?

Thanks in advance

Re: Error: 'no space' in Ksh88

James R. Ferguson — Tue, 19 Oct 2010 16:39:13 GMT

Hi:

More information would be helpful. Is this a merge as opposed to a sort-and-merge?

The error "no space" could refer to filesystem space or an out-of-memory state. Seeing the 'xml_aggregate_to_sdl.ksh' script might be useful.

While you report the total filesystem size, you don't offer the actual free space. What is it?

How far does the process get before the error?

Can you break up the process into intermediate "merges"?

Regards!

...JRF...

Re: Error: 'no space' in Ksh88

Steven E. Protter — Tue, 19 Oct 2010 16:47:59 GMT

Shalom,

Look at the script, see where it stores temporary files. Make sure that file system has enough space for this job or redirect the mount point to a file system with sufficient space.

You could post the script for us to look at as well.

SEP

Re: Error: 'no space' in Ksh88

Dennis Handly — Tue, 19 Oct 2010 17:45:02 GMT

>/rating/diario/scripts//xml_aggregate_to_sdl.ksh[110]: no space

What's on line 110 of that script?
You may be running out of memory in the shell because your strings are too long?

Re: Error: 'no space' in Ksh88

Victor Akamine — Tue, 19 Oct 2010 17:51:27 GMT

More information would be helpful. Is this a merge as opposed to a sort-and-merge?

- It is a normal merge, just put the xml file concatenates with the next.

The error "no space" could refer to filesystem space or an out-of-memory state. Seeing the script might be useful.

- I don't know this part, but I attached the script

While you report the total filesystem size, you don't offer the actual free space. What is it?

- So, this is the filesystem's free space (76Gb)

How far does the process get before the error?

- I don't know

Can you break up the process into intermediate "merges"?

- I can't simulate the error in my enviroment because this only happens in client enviroment.

Thanks

Script is attached

Re: Error: 'no space' in Ksh88

Dennis Handly — Sun, 02 Oct 2011 02:26:35 GMT

>I attached the script
>for NEXTFILE in `echo ${INPUT_PATH}/*.tmp | xargs ls`

If I counted lines correctly, you have too many files in your directory. Going out of your way to use xargs is useless in this format.

You need to switch to something like:
ls ${INPUT_PATH} | fgrep .tmp | while read NEXTFILE ; do
rm -f ${INPUT_PATH}/${NEXTFILE}
...
done

Your old echo commands had: ${INPUT_PATH}/${NEXTFILE}
This wasn't valid in your current script but will be needed with my changes.

>I can't simulate the error in my environment because this only happens in client environment.

Sure you can, you need lots of files. :-)

Re: Error: 'no space' in Ksh88

Victor Akamine — Tue, 19 Oct 2010 18:22:39 GMT

You need to switch to something like:
ls ${INPUT_PATH} | fgrep .temp | while NEXTFILE read; do
rm -f ${INPUT_PATH}/${NEXTFILE}

- I've used this command (xargs) just because before I've used the command 'ls', but this command haven't worked, because I had a big quantity of archives and when I've passed all in the parameter, this not able to accept this number of archives and the error 'arg list too long' is launched.

Re: Error: 'no space' in Ksh88

Dennis Handly — Tue, 19 Oct 2010 22:21:14 GMT

>I've used this command (xargs) just because before I've used the command 'ls', but this command haven't worked

Yes but you're using it wrong. echo will have too many arguments. And if no error there, you'll get too many for the output of the command in ``. And that will have too many for the "for".

Re: Error: 'no space' in Ksh88

Victor Akamine — Thu, 21 Oct 2010 15:56:52 GMT

> Yes but you're using it wrong. echo will have too many arguments. And if no error there, you'll get too many for the output of the command in ``. And that will have too many for the "for".

I still have some doubts. First, you was said to use ls, but after you're saying to use xargs with other parameters.
I tried to use ls, but i couldn't have good results.
Can you explain to me how is the correct form to use the command xargs?

Re: Error: 'no space' in Ksh88

Steven E. Protter — Thu, 21 Oct 2010 16:00:56 GMT

Shalom,

A way around too many files issue.

Modify listing plan.

# create a list of files to process
ls -1 > list | while read -r filename
do
echo "file being processed it ${filename}"
#.... insert more processing here
done

Bottom line is the script is in need of modification.

This analysis is based on Dennis's input.

SEP

Re: Error: 'no space' in Ksh88

Alzhy — Thu, 21 Oct 2010 16:37:25 GMT

ulimit

What is your ulimit?

"no space" usually is a result of a KSH variable or placeholder exceeding its maximum storage limit.

it could also mean your data segment allowance for the shell is being breached.

Try doing a trace on your script and see where it abends. Whilst doing so too -- you may want to put it under tusc or trusss.

HTH.

Re: Error: 'no space' in Ksh88

Victor Akamine — Thu, 21 Oct 2010 16:48:48 GMT

I've searched in web and I've found this:

The default shell of AIX is /bin/ksh (ksh88) and it uses an inefficient stack system. When you code the form "for i in " then ksh attempts to put on the stack. You're running out of memory. Splat.

However, ksh93 uses a different stack library that is more efficient and should work fine in this case. Other shells (like "bash") are more adept at handling such a large data set.

Wondering why ksh88 is not updated? It's the AIX default shell. All the system utilities, SMIT functions, etc. all use ksh. Changing the behavior even a little can have dramatic consequences.

Do you think this is relationed with my problem? Is it possible?!

Re: Error: 'no space' in Ksh88

Alzhy — Thu, 21 Oct 2010 17:01:05 GMT

Absolutely as I have mentioned.

But first try increasing your stack/data size in ulimit sir.

I had a similar issue not long ago where the issue was with KSH limits. A script tring to concatenate thousands of variables' contents into one.

The solutionm ultimately was a reweite of the script to essetially piecemeal the concat and use file I/O for the same.

Re: Error: 'no space' in Ksh88

Victor Akamine — Thu, 21 Oct 2010 17:11:04 GMT

>But first try increasing your stack/data size in ulimit sir.

Sorry, but how can i do this? I've tried to increase de filesystem size, but no the stack size.

>The solutionm ultimately was a reweite of the script to essetially piecemeal the concat and use file I/O for the same.

I can't understand this part my friend, can you be more specific?

thanks

Re: Error: 'no space' in Ksh88

James R. Ferguson — Thu, 21 Oct 2010 18:47:43 GMT

Hi (again):

Victor, as Dennis said, the problem is with:

> for NEXTFILE in `echo ${INPUT_PATH}/*.temp | xargs ls`

The shell expands the '*' to create a list of all filenames in the path. If this number is too large to fit all of them in an argument array (run: 'getconf ARG_MAX'). As Dennis noted, this occurs even before you get a chance to funnel the list on to 'xargs'. This is you "out-of-space" (memory) error.

Dennis showed you how to rewrite the line above.

> SEP ls -1 > list | while read -r filename

No. The 'while' loop isn't going to see anything. You want(ed):

# ls -l | while read filename

As for the comments about 'ksh' and AIX, yes, the Korn88 shell is the default one on AIX, but this is HP-UX. With HP-UX you should use the POSIX shell ('/usr/bin/sh'). This shell is the HP-UX default and is a "super-set" of 'ksh'.

Regards!

...JRF...

Re: Error: 'no space' in Ksh88

Dennis Handly — Fri, 22 Oct 2010 03:25:34 GMT

>you was said to use ls, but after you're saying to use xargs with other parameters.

I said to use ls(1) on the directory and then use fgrep to filter it. This still produces a massive amount of data but it is in a pipe.

>I tried to use ls, but i couldn't have good results.

What were your results?

>Can you explain to me how is the correct form to use the command xargs?

In this case, you don't use xargs.
If you really really wanted to use xargs, you would invoke a script to do the rm and other your checking. Since this now requires two files, it is better to switch to a while loop and forget about xargs.

Re: Error: 'no space' in Ksh88

Victor Akamine — Fri, 22 Oct 2010 17:59:39 GMT

I've tried to use your solution, but this error happens..

Usage: basename string [suffix]

I think, i need more informations, if u can help me..

Thanks

Re: Error: 'no space' in Ksh88

Dennis Handly — Sun, 02 Oct 2011 02:22:48 GMT

>I've tried to use your solution, but this error happens. Usage: basename string [suffix]

What's your new script look like?
I see these places:
FILE_NAME=`basename ${NEXTFILE}`

So $NEXTFILE must be empty. Not sure how that happens with my pipeline.

Re: Error: 'no space' in Ksh88

James R. Ferguson — Fri, 22 Oct 2010 19:51:13 GMT

Hi (again):

> I've tried to use your [Dennis'] solution, but this error happens.

Dennis wrote:

ls ${INPUT_PATH} | fgrep .temp | while NEXTFILE read; do

...where he meant to write:

ls ${INPUT_PATH} | fgrep .temp | while read NEXTFILE; do

...

Make the adjustment in your script.

Regards!

...JRF...

Re: Error: 'no space' in Ksh88

Dennis Handly — Sun, 02 Oct 2011 02:27:53 GMT

>JRF: ls ${INPUT_PATH} | fgrep .tmp | while read NEXTFILE; do

Oops, did that recently elsewhere. Post retconned.