Operating System - HP-UX
1748288 Members
3499 Online
108761 Solutions
New Discussion юеВ

Re: replacing ls from part of script

 
Go to solution
Matthew Pegge_1
Frequent Advisor

replacing ls from part of script

We have a backup script that tars up some files to a filesystem which then gets backed up to tape daily.
The piece of script is :

cd $PROPATHFILES
rc=0
if [ "`ls MB* X* OC* JO* CV* 2>/dev/null | grep -E -v '^XX|^XM|^XOX|^[0-9]'`" ]
then

#
# Create a file list.
# Done this way to ignore XX and XM files but include X* files
#

ls MB* X* OC* JO* CV* 2>/dev/null | grep -E -v '^XX|^XM|^XOX|^[0-9]' > $TMP/
CISAM_check_file_list
echo "Backing up CISAM files"
tar cvf - `cat $TMP/CISAM_check_file_list` | gzip -3 -c > $BACKUP_DIR/backup
/CISAM_backup.tar.gz
rc=$?
[ $rc -ne 0 ] && echo "Error $rc from 'gzip' - $TMP/CISAM_check_file_list"

else
echo "No CISAM files to backup in $PROPATHFILES"
fi

The problem being that the number of files the ls command finds is now too big and we get the following error:

sh: /usr/bin/ls: The parameter list is too long.

Any suggestions on a solution?
8 REPLIES 8
Rodney Hills
Honored Contributor
Solution

Re: replacing ls from part of script

Use another grep on the ls output to select the files first.

example-
cd $PROPPATHFILES
rc=0
ls | grep -e "^MB" -e "^X" -e "^OC" -e "^JO" -e "^CV" >/tmp/holdout2 | grep -E -v '^XX|^XM|^XOX|^[0-9]' >$TMP/CISAM_check_file_list
if [ -s $TMP/CISAM_check_file_list ] ; then
echo "Backing up CISAM files"
tar ...

HTH

-- Rod Hills
There be dragons...
T Dockery
Advisor

Re: replacing ls from part of script

try "echo" instead of ls. As in:
echo MB* X* OC* JO* CV* | grep ...
Rodney Hills
Honored Contributor

Re: replacing ls from part of script

Side note-
The "tar" command you are using may also get the error "parameter list is too long" if their are too many files.

You may want to look at "pax" or even "cpio", since both of them can take a list of files names for back up.

example-
cat $TMP/CISAM_check_file | pax -w -f - | gzip -3 -c >$BACKUP_DIR/backup/CISAM_backup.pax.gz

My 2 cents...

-- Rod Hills
There be dragons...
Rodney Hills
Honored Contributor

Re: replacing ls from part of script

On my first reply, I accidentely left a ">/tmp/holdout" on the line, which should not be there.

Correct line-
ls | grep -e "^MB" -e "^X" -e "^OC" -e "^JO" -e "^CV" | grep -E -v '^XX|^XM|^XOX|^[0-9]' >$TMP/CISAM_check_file_list

-- Rod Hills
There be dragons...
Kent Ostby
Honored Contributor

Re: replacing ls from part of script

The other way to go would be to split out your ls into multiple lines.

ls MB* > /tmp/filelist
ls OC* >> /tmp/filelist
ls JO* >> /tmp/filelist
ls CV* >> /tmp/filelist
ls X* | grep -E -v '^XX|^XM|^XOX|^[0-9] >> /tmp/filelist

Then do your checks on that file.

Its not as "clean" but it is straightforward.

Best regards,

Kent M. Ostby
"Well, actually, she is a rocket scientist" -- Steve Martin in "Roxanne"
TOMAS BERNABEU
Frequent Advisor

Re: replacing ls from part of script


Hi !
use find command

example :
# clear backup file :
>$BACKUP_DIR/backup/CISAM_backup.tar.gz
# find my pattern and tar :
for in MB OC JO CV pattern pattern ....
do
find $PROPATHFILES -name "$i*" -exec tar -xvf $BACKUP_DIR/backup/CISAM_backup.tar {} \ ;
-print 1>/tmp/my_list_of_files 2>/tmp/my_errors

done
James R. Ferguson
Acclaimed Contributor

Re: replacing ls from part of script

Hi Matthew:

In lieu of using 'ls' to collect the files meeting your criteria, you could use 'find':

# cd yourdir
# find . -path "./*" -prune -name "MB* X* OC* JO* CV*" -type f > $TMP/CISAM_check_file_list

This 'find' will not descend subdirectories so it will mimic a simple 'ls'.

Regards!

...JRF...
TOMAS BERNABEU
Frequent Advisor

Re: replacing ls from part of script

Hi !

The tar with -r

example :
# clear backup file :
>$BACKUP_DIR/backup/CISAM_backup.tar.gz
# find my pattern and tar :
for in MB OC JO CV pattern pattern ....
do
find $PROPATHFILES -name "$i*" -exec tar -rvf $BACKUP_DIR/backup/CISAM_backup.tar {} \ ;
-print 1>/tmp/my_list_of_files 2>/tmp/my_errors

done