topic Re: Scripting issue in Operating System - HP-UX

Scripting issue

Dadski — Tue, 27 Nov 2012 02:55:16 GMT

Hi All

I am attempting to create a quick and simple skulker script. This is what I intended.

A text file containing some thing similar to
cat HOUSEKEEP_FILES
-------------------------------------------------------------
FileSys Strin ACTION DAYS OLD
-------------------------------------------------------------
/tmp/martin/1 file1001 REMOVE 2
/tmp/martin/1 file1002 REMOVE 2
/tmp/martin/1 file1003 REMOVE 2
/tmp/martin/2 file1004 REMOVE 3
/tmp/martin/2 file1005 REMOVE 3
/tmp/martin/2 file1006 REMOVE 3

I wanted to use a sed and awk command, lets say to fin files older than 2 or 3 ($4) in the directory ($1) the actual file ($2). I dont think I'm to far away bu reading that but the command blurps. By the way I am just trying to list initially. The command I am using is,

sed -n '/REMOVE/p' HOUSEKEEP_FILES | awk '{system(find $1, " " -name $2," " -print)}'

is there any awk gurus who can simplify this?

thanks very much

Martin

P.S. this thread has been moved from HP-UX > System Administration to HP-UX > languages - HP Forums Moderator

Re: Scripting issue

Autocross.US — Thu, 08 Jan 2009 15:48:22 GMT

Here's a solution that doesn't use sed/awk:

egrep "REMOVE" HOUSEKEEP_FILES | while read DIR FILE ACTION DAYS_OLD ; do
[[ $DAYS_OLD -gt 2 ]] && find $DIR -name "$FILE" -print
done

Re: Scripting issue

Matti_Kurkela — Thu, 08 Jan 2009 16:17:42 GMT

Looks a bit dangerous. For each line, you're about to remove potentially several files.

For example, when receiving the first non-header line of HOUSEKEEP_FILES, your script would not remove only:
/tmp/martin/1/file1001

but also anything matching:
/tmp/martin/1/*/file1001
/tmp/martin/1/*/*/file1001
/tmp/martin/1/*/*/*/file1001
...

(When creating automated file removers, you *must* be as specific and predictable as possible. The hard way of learning this lesson involves restoring files from backups and restarting long-running processing jobs while thinking about how to explain this to an angry customer :-)

If the directory structure is predictable, it would be much safer to directly assemble the name of the file to be removed instead of fishing around with the find command.

If the file paths are of the form "/FileSys/Strin", then you could do something like this:

#!/bin/sh
grep '^/.*REMOVE' HOUSEKEEP_FILES | \
while read FileSys Strin junk; do
echo "rm \"$FileSys/$Strin\""
# rm "$FileSys/$Strin" # uncomment after testing
done

The first grep command passes through only the lines beginning with a '/' character, so we get rid of all the header lines. After that it's just a matter of reading each line, collecting the columns to variables and assembling the necessary command line.

Of course, the above approach is slow and causes excessive forking of new processes.
So, here's version 2:

#!/bin/sh
grep '^/.*REMOVE' HOUSEKEEP_FILES | \
awk '{ print $1 "/" $2 };' | \
xargs echo rm
# Replace "xargs echo rm" with "xargs rm" if OK

Note that there is no loop at all, just a single pipeline. You could even write it as a one-liner, but I think it's easier to understand when written on multiple lines.

MK

Re: Scripting issue

OldSchool — Thu, 08 Jan 2009 16:27:31 GMT

be aware that the above solutions do not address the "age" of the file. And you've not defined what you mean by age. There are access times and modification times, but "creation" time is not available (unless the file is absolutely never touched after creation)

Re: Scripting issue

VK2COT — Thu, 08 Jan 2009 16:31:40 GMT

Hello,

Hmm, lot of possibilities.

I was even thinking about generalizing the
script. What if HOUSEKEEP_FILES could contain
different keywords in the third column (not only "REMOVE"). For example:

-------------------------------------------------------------
FileSys Strin ACTION DAYS OLD
-------------------------------------------------------------
/tmp/martin/1 file1001 REMOVE 2
/tmp/martin/1 file1002 REMOVE 2
/tmp/martin/1 file1003 REMOVE 2
/tmp/martin/2 file1004 REMOVE 3
/tmp/martin/2 file1005 KEEP 3
/tmp/martin/2 file1006 REMOVE 3

Here is my little script:

#!/bin/sh

awk '! /---|DAYS OLD/ {print}' HOUSEKEEP_FILES | while read SEARCHDIR FILE ACTION HOWOLD
do
case $ACTION in
REMOVE|remove) ACTION="rm -f" ;;
*) ACTION="ll" ;;
esac
find $SEARCHDIR -name $FILE -type f -mtime $HOWOLD -exec $ACTION {} \;
done

And here is what it would run:

find /tmp/martin/1 -name file1001 -type f -mtime 2 -exec rm -f {} \;
find /tmp/martin/1 -name file1002 -type f -mtime 2 -exec rm -f {} \;
find /tmp/martin/1 -name file1003 -type f -mtime 2 -exec rm -f {} \;
find /tmp/martin/2 -name file1004 -type f -mtime 3 -exec rm -f {} \;
find /tmp/martin/2 -name file1005 -type f -mtime 3 -exec ll {} \;
find /tmp/martin/2 -name file1006 -type f -mtime 3 -exec rm -f {} \;

I am sure there are many more ways to
do it :)

Cheers,

VK2COT

Re: Scripting issue

Peter Nikitka — Thu, 08 Jan 2009 16:36:09 GMT

Hi,

it is not clear, in which way you want to use the file name.
I suggest, that if there is no value of 'fil' found in awk, use any filespec found in the list. Else use only the specified filespec.

Wrap my awk in a script of your choice;
start with something like this in the commandline to test your expectations:

#filespec=file1002 # may be set
awk -v age=3 -v dir=/tmp/martin/1 -v fil="$filespec" 'NR<4 {next}
($3 == "REMOVE") && ($4 >= age) && ($1 == dir) { if(fil && $2 != fil) next
sys_str="find "dir" -type f -name "$2" -mtime +"age" -print"
print sys_str
system(sys_str)}' HOUSEKEEP_FILES

You could implement the handling of directory names the same way as I did with the filespec.

mfG Peter

Re: Scripting issue

Dadski — Fri, 09 Jan 2009 07:38:23 GMT

Thanks all for replying VK2COT you've nailed it, thats exactly what I was looking for. thanks to all.

Re: Scripting issue

Peter Nikitka — Fri, 09 Jan 2009 11:12:36 GMT

Hi,

nice, that we could help you!
Because you are new to this forum - Welcome! - I want to put yout attention to its unique point system:

http://forums11.itrc.hp.com/service/forums/helptips.do?#33

mfG Peter

Re: Scripting issue

Mark McDonald_2 — Fri, 09 Jan 2009 11:41:48 GMT

Just a note - we have a similar script.

We use ZIP as an option too. So 2 day old logs get zipped, then deleted after 7 days.

So you would have:
/tmp/martin/1 file1001 ZIP 2
/tmp/martin/1 file1001 REMOVE 7