Error: 'no space' in Ksh88

[Victor Akamine]

Hi,

I need a help because I've a script (.ksh). This script merge many archives (xml) and concatenates all in one.

But the problem happens when I need to concatenates a lot of archives and the error 'no space' ocurrs.

I try to increase the filesystem's capacity.

----------------------------
Number of Archives: 104036
Total Archives size: 26 Gb
FileSystem size: 74 Gb
----------------------------

Even so, the error is still 'no space'.


I use ksh88, have any problem about it?

This is the error line in sysout:
/rating/diario/scripts//xml_aggregate_to_sdl.ksh[110]: no space


Have any idea to solve my problem?

Thanks in advance

[James R. Ferguson]

Hi:

More information would be helpful. Is this a merge as opposed to a sort-and-merge?

The error "no space" could refer to filesystem space or an out-of-memory state. Seeing the 'xml_aggregate_to_sdl.ksh' script might be useful.

While you report the total filesystem size, you don't offer the actual free space. What is it?

How far does the process get before the error?

Can you break up the process into intermediate "merges"?

Regards!

...JRF...

[Steven E. Protter]

Shalom,

Look at the script, see where it stores temporary files. Make sure that file system has enough space for this job or redirect the mount point to a file system with sufficient space.

You could post the script for us to look at as well.

SEP

[Dennis Handly]

>/rating/diario/scripts//xml_aggregate_to_sdl.ksh[110]: no space

What's on line 110 of that script?
You may be running out of memory in the shell because your strings are too long?

[Victor Akamine]

More information would be helpful. Is this a merge as opposed to a sort-and-merge?

- It is a normal merge, just put the xml file concatenates with the next.

The error "no space" could refer to filesystem space or an out-of-memory state. Seeing the script might be useful.

- I don't know this part, but I attached the script


While you report the total filesystem size, you don't offer the actual free space. What is it?

- So, this is the filesystem's free space (76Gb)

How far does the process get before the error?

- I don't know

Can you break up the process into intermediate "merges"?

- I can't simulate the error in my enviroment because this only happens in client enviroment.


Thanks

Script is attached

[Dennis Handly]

>I attached the script
>for NEXTFILE in `echo ${INPUT_PATH}/*.tmp | xargs ls`

If I counted lines correctly, you have too many files in your directory. Going out of your way to use xargs is useless in this format.

You need to switch to something like:
ls ${INPUT_PATH} | fgrep .tmp | while read NEXTFILE ; do
    rm -f ${INPUT_PATH}/${NEXTFILE}
   ...
done

Your old echo commands had: ${INPUT_PATH}/${NEXTFILE}
This wasn't valid in your current script but will be needed with my changes.

>I can't simulate the error in my environment because this only happens in client environment.

Sure you can, you need lots of files. :-)

[Victor Akamine]

You need to switch to something like:
ls ${INPUT_PATH} | fgrep .temp | while NEXTFILE read; do
rm -f ${INPUT_PATH}/${NEXTFILE}

- I've used this command (xargs) just because before I've used the command 'ls', but this command haven't worked, because I had a big quantity of archives and when I've passed all in the parameter, this not able to accept this number of archives and the error 'arg list too long' is launched.

[Dennis Handly]

>I've used this command (xargs) just because before I've used the command 'ls', but this command haven't worked

Yes but you're using it wrong. echo will have too many arguments. And if no error there, you'll get too many for the output of the command in ``. And that will have too many for the "for".

[Victor Akamine]

> Yes but you're using it wrong. echo will have too many arguments. And if no error there, you'll get too many for the output of the command in ``. And that will have too many for the "for".

I still have some doubts. First, you was said to use ls, but after you're saying to use xargs with other parameters.
I tried to use ls, but i couldn't have good results.
Can you explain to me how is the correct form to use the command xargs?

[Steven E. Protter]

Shalom,

A way around too many files issue.

Modify listing plan.

# create a list of files to process
ls -1 > list | while read -r filename
do
echo "file being processed it ${filename}"
#.... insert more processing here
done


Bottom line is the script is in need of modification.

This analysis is based on Dennis's input.

SEP

[Alzhy]

ulimit

What is your ulimit?

"no space" usually is a result of a KSH variable or placeholder exceeding its maximum storage limit.

it could also mean your data segment allowance for the shell is being breached.

Try doing a trace on your script and see where it abends. Whilst doing so too -- you may want to put it under tusc or trusss.

HTH.

[Victor Akamine]

I've searched in web and I've found this:

The default shell of AIX is /bin/ksh (ksh88) and it uses an inefficient stack system. When you code the form "for i in " then ksh attempts to put on the stack. You're running out of memory. Splat.

However, ksh93 uses a different stack library that is more efficient and should work fine in this case. Other shells (like "bash") are more adept at handling such a large data set.

Wondering why ksh88 is not updated? It's the AIX default shell. All the system utilities, SMIT functions, etc. all use ksh. Changing the behavior even a little can have dramatic consequences.


Do you think this is relationed with my problem? Is it possible?!

[Alzhy]

Absolutely as I have mentioned.

But first try increasing your stack/data size in ulimit sir.

I had a similar issue not long ago where the issue was with KSH limits. A script tring to concatenate thousands of variables' contents into one.

The solutionm ultimately was a reweite of the script to essetially piecemeal the concat and use file I/O for the same.

[Victor Akamine]

>But first try increasing your stack/data size in ulimit sir.

Sorry, but how can i do this? I've tried to increase de filesystem size, but no the stack size.

>The solutionm ultimately was a reweite of the script to essetially piecemeal the concat and use file I/O for the same.

I can't understand this part my friend, can you be more specific?

thanks

[James R. Ferguson]

Hi (again):

Victor, as Dennis said, the problem is with:

> for NEXTFILE in `echo ${INPUT_PATH}/*.temp | xargs ls`

The shell expands the '*' to create a list of all filenames in the path. If this number is too large to fit all of them in an argument array (run: 'getconf ARG_MAX'). As Dennis noted, this occurs even before you get a chance to funnel the list on to 'xargs'. This is you "out-of-space" (memory) error.

Dennis showed you how to rewrite the line above.

> SEP ls -1 > list | while read -r filename

No. The 'while' loop isn't going to see anything. You want(ed):

# ls -l | while read filename

As for the comments about 'ksh' and AIX, yes, the Korn88 shell is the default one on AIX, but this is HP-UX. With HP-UX you should use the POSIX shell ('/usr/bin/sh'). This shell is the HP-UX default and is a "super-set" of 'ksh'.

Regards!

...JRF...

[Dennis Handly]

>you was said to use ls, but after you're saying to use xargs with other parameters.

I said to use ls(1) on the directory and then use fgrep to filter it. This still produces a massive amount of data but it is in a pipe.

>I tried to use ls, but i couldn't have good results.

What were your results?

>Can you explain to me how is the correct form to use the command xargs?

In this case, you don't use xargs.
If you really really wanted to use xargs, you would invoke a script to do the rm and other your checking. Since this now requires two files, it is better to switch to a while loop and forget about xargs.

[Victor Akamine]

I've tried to use your solution, but this error happens..

Usage: basename string [suffix]

I think, i need more informations, if u can help me..

Thanks

[Dennis Handly]

>I've tried to use your solution, but this error happens.  Usage: basename string [suffix]

What's your new script look like?
I see these places:
FILE_NAME=`basename ${NEXTFILE}`

So $NEXTFILE must be empty. Not sure how that happens with my pipeline.

[James R. Ferguson]

Hi (again):

> I've tried to use your [Dennis'] solution, but this error happens.

Dennis wrote:

ls ${INPUT_PATH} | fgrep .temp | while NEXTFILE read; do

...where he meant to write:

ls ${INPUT_PATH} | fgrep .temp | while read NEXTFILE; do

...

Make the adjustment in your script.

Regards!

...JRF...

[Dennis Handly]

>JRF: ls ${INPUT_PATH} | fgrep .tmp | while read NEXTFILE; do

Oops, did that recently elsewhere. Post retconned.

[Victor Akamine]

Hi!

I can change the script! But i have one doubt yet.

In the line:

for NEXTFILE in `echo ${INPUT_PATH}/*.tmp | xargs ls`

I can change the code, but in the another line is need to change too, doesn't it?
The line is this:

for NEXTFILE in `echo ${INPUT_PATH}/*${LIST_PID[$CONT_PID_AUX]}_*.tmp | xargs ls`

Must I change this too?!

Thanks!

[James R. Ferguson]

Hi (again):

> I can change the code, but in the another line is need to change too, doesn't it?

Once again, whether or not you need to modify this line will depend on the number of runtime arguments that are created. You could fashion the same solution as Dennis suggested for the first case.

By the way, both Dennis and I wrote ".temp" when it appears that should have been ".tmp".

Regards!

...JRF...