System Administration
Showing results for 
Search instead for 
Do you mean 

replacing ls from part of script

SOLVED
Go to Solution
Frequent Advisor

replacing ls from part of script

We have a backup script that tars up some files to a filesystem which then gets backed up to tape daily.
The piece of script is :

cd $PROPATHFILES
rc=0
if [ "`ls MB* X* OC* JO* CV* 2>/dev/null | grep -E -v '^XX|^XM|^XOX|^[0-9]'`" ]
then

#
# Create a file list.
# Done this way to ignore XX and XM files but include X* files
#

ls MB* X* OC* JO* CV* 2>/dev/null | grep -E -v '^XX|^XM|^XOX|^[0-9]' > $TMP/
CISAM_check_file_list
echo "Backing up CISAM files"
tar cvf - `cat $TMP/CISAM_check_file_list` | gzip -3 -c > $BACKUP_DIR/backup
/CISAM_backup.tar.gz
rc=$?
[ $rc -ne 0 ] && echo "Error $rc from 'gzip' - $TMP/CISAM_check_file_list"

else
echo "No CISAM files to backup in $PROPATHFILES"
fi

The problem being that the number of files the ls command finds is now too big and we get the following error:

sh: /usr/bin/ls: The parameter list is too long.

Any suggestions on a solution?
8 REPLIES
Honored Contributor Honored Contributor

Re: replacing ls from part of script

Use another grep on the ls output to select the files first.

example-
cd $PROPPATHFILES
rc=0
ls | grep -e "^MB" -e "^X" -e "^OC" -e "^JO" -e "^CV" >/tmp/holdout2 | grep -E -v '^XX|^XM|^XOX|^[0-9]' >$TMP/CISAM_check_file_list
if [ -s $TMP/CISAM_check_file_list ] ; then
echo "Backing up CISAM files"
tar ...

HTH

-- Rod Hills
There be dragons...
Advisor

Re: replacing ls from part of script

try "echo" instead of ls. As in:
echo MB* X* OC* JO* CV* | grep ...
Honored Contributor Honored Contributor

Re: replacing ls from part of script

Side note-
The "tar" command you are using may also get the error "parameter list is too long" if their are too many files.

You may want to look at "pax" or even "cpio", since both of them can take a list of files names for back up.

example-
cat $TMP/CISAM_check_file | pax -w -f - | gzip -3 -c >$BACKUP_DIR/backup/CISAM_backup.pax.gz

My 2 cents...

-- Rod Hills
There be dragons...
Honored Contributor Honored Contributor

Re: replacing ls from part of script

On my first reply, I accidentely left a ">/tmp/holdout" on the line, which should not be there.

Correct line-
ls | grep -e "^MB" -e "^X" -e "^OC" -e "^JO" -e "^CV" | grep -E -v '^XX|^XM|^XOX|^[0-9]' >$TMP/CISAM_check_file_list

-- Rod Hills
There be dragons...
Honored Contributor Honored Contributor

Re: replacing ls from part of script

The other way to go would be to split out your ls into multiple lines.

ls MB* > /tmp/filelist
ls OC* >> /tmp/filelist
ls JO* >> /tmp/filelist
ls CV* >> /tmp/filelist
ls X* | grep -E -v '^XX|^XM|^XOX|^[0-9] >> /tmp/filelist

Then do your checks on that file.

Its not as "clean" but it is straightforward.

Best regards,

Kent M. Ostby
"Well, actually, she is a rocket scientist" -- Steve Martin in "Roxanne"
Frequent Advisor Frequent Advisor

Re: replacing ls from part of script


Hi !
use find command

example :
# clear backup file :
>$BACKUP_DIR/backup/CISAM_backup.tar.gz
# find my pattern and tar :
for in MB OC JO CV pattern pattern ....
do
find $PROPATHFILES -name "$i*" -exec tar -xvf $BACKUP_DIR/backup/CISAM_backup.tar {} \ ;
-print 1>/tmp/my_list_of_files 2>/tmp/my_errors

done
Acclaimed Contributor Acclaimed Contributor

Re: replacing ls from part of script

Hi Matthew:

In lieu of using 'ls' to collect the files meeting your criteria, you could use 'find':

# cd yourdir
# find . -path "./*" -prune -name "MB* X* OC* JO* CV*" -type f > $TMP/CISAM_check_file_list

This 'find' will not descend subdirectories so it will mimic a simple 'ls'.

Regards!

...JRF...
Highlighted
Frequent Advisor Frequent Advisor

Re: replacing ls from part of script

Hi !

The tar with -r

example :
# clear backup file :
>$BACKUP_DIR/backup/CISAM_backup.tar.gz
# find my pattern and tar :
for in MB OC JO CV pattern pattern ....
do
find $PROPATHFILES -name "$i*" -exec tar -rvf $BACKUP_DIR/backup/CISAM_backup.tar {} \ ;
-print 1>/tmp/my_list_of_files 2>/tmp/my_errors

done