need help with this script

[Ben Prusinski]

Hi unix script gurus,

I am working on a script and it doesnt quite do the trick right yet. Basically, the script will run from a cron job and function to remove old files of a specific type. Once this one is working, the process will probably be expanded to other file types and/or source directories.

File selection criteria:
1) File location: /ifas/admin/datagc (remove only files from this directory)

2) Because IFAS still has roots from its MPE days, each file has an MPE header record which tells us what program created the file. The files we want to remove were created by the IFAS Ad-Hoc report writer.There are 200 files that need to have this field stripped away:

Example:
The first record for /ifas/admin/datagc/BACTBSG is:
MPE 1028 196 64 50000 0 -1 0 0 -1 0

The string from the above that we need to select on is : " 0 0 -1 0" (there is a space before the first 0). This is the file code for the output from the Ad-Hoc Report Writer.
3) The third criteria is to remove only those files with a last modified date of ?? number of days or more (probably 30 days).


Process
1) The selected files should be moved (mv) from /ifas/admin/datagc to /ifas/admin/trash (new directory). The file should be appended with the directory name that the file came from. Example: /ifas/admin/datagc/BACTBSG is moved to /ifas/admin/trash/BACTBSG.datagc. Be sure to keep the same file permissions and modified date. We haven't yet decided how often to run this script, perhaps weekly.

2) A separate script/cron job (not yet written) will then do the actual delete from ./trash for files last modified ?? number of days or more (probably 90 days). This allows the users a last chance to keep the file before it is deleted.

Here is what I have thus far:

#!/usr/bin/ksh
#
SOURCE_DIR=/ifas/admin/datagc
TARGETA_DIR=/ifas/admin/datagc
#
find $SOURCE_DIR -type f -mtime +30|while read FILE # find files>30 days old
do
FIRST_RECORD=`head -1 $FILE` # read first record of file
echo $FIRST_RECORD
FILE_FOUND=`grep " 0 0 -1 0" $FILE|wc -l` # set flag to 1 if record matches pattern
if [ $FILE_FOUND -eq 1 ]
then
cd $SOURCE_DIR
FILE_NAME=`echo $FILE|awk -F"/"'{print $NF}'`
# get rid of directory path from file to show just the file name
#
find . -name "$FILE_NAME"|-mivd>archive.cpio # create cpio archive of file includeparent dir
mv archive.cpio $TARGET_DIR # move archive to target directory
cd $TARGET_DIR
cpio -mivd fi
done
#
# end of script
However I am still stuck. It doesnt move the cleaned up files with the field stripped to the new directory. I am trying to cut this MPE field out and move the modified file to the directory. Thanks for your help

Ben Prusinski

[Patrick Wallek]

Two things:

1) You reference $TARGET_DIR in your script, but you set the variable TARGETA_DIR at the top of the script.

2) Your TARTETA_DIR is the same directory as your SOURCE_DIR so that will cause you problems too.

[Ben Prusinski]

Thanks for the input. I updated the script above fixing the errors:

the script now reads:

#!/usr/bin/ksh
#
SOURCE_DIR=/ifas/admin/datagc
TARGET_DIR=/ifas/admin/trash
#
find $SOURCE_DIR -type f -mtime +30|while read FILE # find files>30 days old
do
FIRST_RECORD=`head -1 $FILE`
# read first record of file
echo $FIRST_RECORD
FILE_FOUND=`grep " 0 0 -1 0" $FILE|wc -l`
# set flag to 1 if record matches pattern
if [ $FILE_FOUND -eq 1 ]
then
cd $SOURCE_DIR
FILE_NAME=`echo $FILE|awk -F"/"'{print $NF}'`
# get rid of directory path from file to show # just the file name
#
find . -name "$FILE_NAME"|
cpio -mivd>archive.cpio
# create cpio archive of old file include
# parent directory
#
mv archive.cpio $TARGET_DIR # move archive to target directory
cd $TARGET_DIR
cpio -mivd# unarchive cpio archive and create other dirs # if needed
fi
done
#
# end of script

OUTPUT from script:

# ./ifas2
MPE 5 80 16 1023 0 0 10 0 -1 0
pwd
MPE 5 80 16 1023 0 0 10 0 -1 0
MPE 5 80 16 1023 0 0 10 0 -1 0
MPE 5 80 16 1023 0 0 10 0 -1 0
MPE 5 80 16 1023 0 0 10 0 -1 0
MPE 5 80 16 1023 0 0 10 0 -1 0
MPE 1028 80 16 1023 0 -1 10 0 -1 0
End of volume
errno: 25, Can't read input
End of volume
errno: 25, Can't read input
MPE 1028 80 16 1023 0 -1 10 0 -1 0
MPE 1028 80 16 1023 0 -1 10 0 -1 0
MPE 1028 196 64 50000 0 -1 0 0 -1 0
End of volume
errno: 25, Can't read input

# cd /ifas/admin/trash; pwd
/ifas/admin/trash
# ll
total 0
-rw-rw-r-- 1 root bsi 0 Mar 10 09:46 archive.cpio

However I still have that annoying MPE field in the updated BACTBSG file that I am trying to get rid of. I want to keep everything in this file but delete the legacy old MPE field above and output the modified file to the /ifas/admin/datagc directory while sending the archived old file to /ifas/admin/trash
What am I doing wrong here in my script? Thanks!

Ben Prusinski

[David_246]

Hi Ben,

for i in `find /ifas/admin/datagc -mtime +30`
do
tst=`head -1 $i | grep " 0 -1 0 0 -1 0 "$`
if [ -n "$tst" ]
then
move, copy or whatever
echo "message \n">>file1
fi

Ben, don't forget to give points, also your previous questions!

Is this a school project or so ??

Regs David

[Chris Vail]

I would approach this completely differently:

for FILE in `find $SOURCE_DIR -type f -mtime +30`
do
FIRST_RECORD=`head -1 $FILE`
FILE_FOUND=`grep " 0 0 -1 0" $FILE
if test "$FILE_FOUND"
then
cat $FILE|grep -v FIRST_RECORD>/tmp/TFILE
mv /tmp/TFILE $FILE
fi
tar cvf - $FILE|(cd $TARGET_DIR;tar xvfp -)
done

The test in there checks to see if FILE_FOUND exists. If the grep command doesn't work, it won't exist. I've attached a script that does something similar in filtering out certain lines out of system log files. Its fully commented, so you should be able to make sense of it.


Good luck
Chris

[Jose Mosquera]

Hi Ben,

I am not sure of having understood you, in any case try this:


#!/usr/bin/ksh
#
SOURCE_DIR=/ifas/admin/datagc
TARGET_DIR=/ifas/admin/trash
#
cd $SOURCE_DIR
FILES=`find . -path "./*" -prune -type f -mtime +30|cut -c3-`
for FILE in $FILES
do
FIRST_RECORD=`head -1 $FILE` # read first record of file
FILE_FOUND=`grep " 0 0 -1 0" $FILE|wc -l` # set flag to 1 if record matches pattern
if [ $FILE_FOUND -ne 0 ]
then
find . -name $FILE|-mivd > archive.cpio # create cpio archive of file includeparent dir
mv archive.cpio $TARGET_DIR # move archive to target directory
cd $TARGET_DIR
cpio -mivd < archive.cpio # unarchive cpio archive and create other dirs if needed
fi
done
#
# end of script


Rgds.

[john korterman]

Hi Ben,
maybe it is the copy/paste, but you need a space in between the second double-qoute and the first single-qoute in the awk statement, e.g.: awk -F"/" '{print}'
You also need the cpio command itself in the line:
find . -name "$FILE_NAME"|-mivd>archive.cpio
and the parameters for that cpio should be for "out", e.g.:
find . -name "$FILE_NAME" -print |cpio -movd >archive.cpio

regards,
John K.

[Ben Prusinski]

Thanks for your help thus far. I still for some strange reason cannot get the files to move over with the changes to the new /ifas/admin/trash directory. Here is the updated script:

#!/usr/bin/ksh
#
#
# Program: The script will search each file in the /ifas/admin/datagc
# directory, then it will trim the initial record which is a legacy
# MPE field. After this record is trimmed from each file, the file
# will be moved to the /ifas/admin/trash directory with the directory
# name appended to its name as an extension.
# Example: /ifas/admin/datagc/BACTBSG will be moved to
# /ifas/admin/trash/BACTBSG.datagc
#
# Define directory and file names
#
SOURCE_DIR=/ifas/admin/datagc
TARGET_DIR=/ifas/admin/trash
#FILE1=$SOURCE_DIR/BACTBSG
#
# Search for files and MPE record string
#
for FILE in `find $SOURCE_DIR -type f -mtime +30`
do
FIRST_RECORD=`head -1 $FILE` # read first record of file
# echo $FIRST_RECORD
FILE_FOUND=`grep " 0 0 -1 0" $FILE`
#
#
if test "$FILE_FOUND"
then
# cat the file and grab the first record that contains the legacy MPE string
cat $FILE|grep -v FIRST_RECORD>/ifas/admin/trash/BACTBSG.datagc
#
# now lets move these files over to where we want to store them.
# move (mv) each file, in this case BACTBSG to the /ifas/admin/trash
#
mv /ifas/admin/datagc/BACTBSG /ifas/admin/trash/BACTBSG.datagc
# make copy of file and send to temp directory
#
# #
fi
done
#
# end of script program

[john korterman]

Hi again,
you probably mean "$FIRST_RECORD" in the line:
cat $FILE|grep -v FIRST_RECORD>/ifas/admin/trash/BACTBSG.datagc
I guess the idea is that all lines except "$FIRST_RECORD" should go into:
/ifas/admin/trash/BACTBSG.datagc
However, the following line:
mv /ifas/admin/datagc/BACTBSG /ifas/admin/trash/BACTBSG.datagc
will overwrite the file just created, which is probably not what you want.

regards,
John K.

[Ben Prusinski]

John
You are correct. How can I move these changed files to the new /ifas/admin/trash directory without overwriting the original files and appending the filename extension .datagc to each new file? Thanks
Ben

[john korterman]

Hi again,
Hmm, if I understand you correctly, try these changes:

TEMP_DIR=/tmp
TEMP_FILENAME=noheader
# The above is needed for temporary storing each file with the header stripped off.


# Change the "if test" construction to this in order to achieve temporary storage of each file without the header and then moving this file to destination, appending to the filename:

if test "$FILE_FOUND"
then
# create a temporary file..
#
cat $FILE|grep -v "$FIRST_RECORD">${TEMP_DIR}/${TEMP_FILENAME}
#
# Move temporary file to destination
# Prepend name, append datagc
#
FILENAME=$( basename $FILE )
mv ${TEMP_DIR}/${TEMP_FILENAME} $TARGET_DIR/BACTBSG${FILENAME}.datagc
fi

Hope it helps,
John K.

[Ben Prusinski]

Hi John,

Thanks. Actually sorry to confuse you guys more but I just spoke with the user and he wants the following change to the script:

1. move all files under the /ifas/admin/datagc
directories that CONTAIN the MPE string " 0 0 -1" to the /ifas/admin/trash directory. There will be no trimming or changes made to the file.

2. Append the .datagc to each filename that is moved to the new /ifas/admin/trash directory without changing the contents of the file.

Once again I really appreciate your help on this script. I used to be a programmer but since I have not used my coding chops in a while I am rusty on unix programming.

3. Create another script, delete the files from the /ifas/admin/trash directory that are older than 90 days. Run this script from cron.

;-)
Ben

[john korterman · Accepted solution]

Hi,
as for 1. and 2., you can try this:

#!/usr/bin/sh

FULL_SOURCE_PATH=/ifas/admin/datagc
DEST_PATH=/ifas/admin/trash
HEADER=" 0 0 -1"

cd $FULL_SOURCE_PATH
if [ "$?" = "0" ]
then
find . -type f -print | while read line
do
grep -q "$HEADER" $line
if [ "$?" = "0" ]
then
cp $line $DEST_PATH/${line}.datagc
fi
done
else
echo cd to $FULL_SOURCE_PATH failed, stop
exit 1
fi

You can make a script like this for deleting the old files:
#!/usr/bin/sh
DEL_DIR=/ifas/admin/trash

cd $DEL_DIR
if [ "$?" = "0" ]
then
find . -type f -name "*datgc" -mtime +90 -exec ls {} \;
else
echo cd to $DEL_DIR failed
fi

When you are very sure of everything, then change the "ls" in the above to "rm".

regards,
John K.

[Ben Prusinski]

Thanks a million John and to everyone for your help. This looks great I will test it and let you all know the results. Now to get better at scripting once again I will have to devote more time to programming.

~ Ben Prusinski