calculate days between two dates

[Bala_8]

Hi,

I have some logfiles being generated thru various batch processes. The file names include the date,time of generation of the logfile. A sample listing is shown below :
-------------------------------------------
fuji# ll
total 836
-rw-r----- 1 wladm services 398079 Dec 23 2002 BIWPSLB001fBatchLog
-rw-r----- 1 wladm services 1696 Dec 30 2002 biwpsnss005m.iwps.int30Dec2002134438.log
-rw-r----- 1 wladm services 1696 Dec 30 2002 biwpsnss005m.iwps.int30Dec2002153119.log
-rw-r--r-- 1 wladm services 409 Jan 17 2003 BIWPSQO002mBatchLog
-rw-r--r-- 1 wladm services 907 Jan 9 2003 biwpssp004d.iwps.ext09Jan2003150024.log
-rw-r--r-- 1 wladm services 907 Jan 9 2003 biwpssp004d.iwps.ext09Jan2003150040.log
-rw-r--r-- 1 wladm services 887 Jan 9 2003 biwpswa007d.iwps.int09Jan2003141625.log
-rw-r--r-- 1 wladm services 887 Jan 9 2003 biwpswa007d.iwps.int09Jan2003142218.log
-rw-r--r-- 1 wladm services 887 Jan 9 2003 biwpswa007d.iwps.int09Jan2003142341.log
-rw-r--r-- 1 wladm services 1242 Jan 9 2003 biwpswa007d.iwps.int09Jan2003150420.log
-rw-r--r-- 1 wladm services 1242 Jan 9 2003 biwpswa007d.iwps.int09Jan2003151043.log
-rw-r--r-- 1 wladm services 5679 Jan 9 2003 biwpswa007d.iwps.int09Jan2003151433.log
-rw-r--r-- 1 wladm services 1347 Jan 10 2003 ftp_recovery10Jan2003200423.log
-rw-r--r-- 1 wladm services 1343 Jan 10 2003 ftp_recovery10Jan2003201529.log
-rw-r--r-- 1 wladm services 321 Jan 13 2003 ftp_recovery13Jan2003190819.log
-rw-r--r-- 1 wladm services 321 Jan 13 2003 ftp_recovery13Jan2003190951.log
-rw-r--r-- 1 wladm services 318 Jan 13 2003 ftp_recovery13Jan2003191048.log
-rw-r--r-- 1 wladm services 318 Jan 13 2003 ftp_recovery13Jan2003191507.log
-rw-r--r-- 1 wladm services 321 Jan 13 2003 ftp_recovery13Jan2003192100.log
-------------------------------------------

I need help in writing a script which compares the date string within the file names, of all files in /abc/log directory, with the current system date. If the difference is 14 days or more, i need to delete that file from this directory. Since i've backup of these logfiles in tape, i need to clear from the system to free up space. This script will be scheduled to run daily.

Looking forward for all your valuable feedbacks / suggestions.

cheers
-bala-

[Armin Feller]

Hi,

please have a look at this:

http://bizforums.itrc.hp.com/cm/QuestionAnswer/1,,0x2b9f85079106d71190050090279cd0f9,00.html

Hope that helps.

Regards ...
Armin

[Chris Wilshaw]

in that directory

find . -name "*.log" -mtime +14 -exec rm {} \;


Any log file over 14 days old will then be deleted.

[Bala_8]

But Chris,

I want to use the time stamp within the file name, due to some reasons.

thanks
-bala-

[Tom Jackson]

Hi Bala:

I've done something like this in C. You may want to do some printf statements to show the results. Sorry about the formatting.

Tom

/* includes */
#include
#include
#include
#include

#define RETENTION 10

main( argc, argv, envp )
int argc;
char *argv[];
char *envp[];
{
int cur_date;
int file_date;
char today[BUFSIZ];
char file_date_ascii[BUFSIZ];
char string[BUFSIZ];
DIR *dir;
time_t t;
struct stat file_status;
struct dirent *ent;
struct tm *sys_date;
struct tm *file_date_st;


t = time(NULL);
sys_date = localtime(&t);
cur_date = sys_date->tm_yday;
cur_date = sys_date->tm_yday;
strcpy(today, asctime(sys_date));
today[strlen(today)-1] = '\0';

chdir("/some/dir/somewhere");

dir = opendir("/some/dir/somewhere");
while ((ent = readdir(dir)) != NULL)
{
sprintf(string, "%s/%s", ent->d_name, "someFile.*");
if (!stat(string, &file_status))
{
file_date_st = localtime(&file_status.st_atime);
file_date = file_date_st->tm_yday;
if (file_date > cur_date)
{
if (file_date < cur_date+365-RETENTION)
{
do something ...
}
else
{
do something else
}
}
}
}
}

[john korterman · Accepted solution]

Hi Bala,
please try the following script, but first configure RETURN_SECS_SCRIPT to point at the attached script.
LOGDIR must point to the directory where your logfiles are located.
FILE_AGE is currently set to 1209600, which is 14 days expressed in seconds - can be changed for test purposes. the rm command has been commented out.

#!/usr/bin/sh

RETURN_SECS_SCRIPT=""
FILE_AGE="1209600" # no. of seconds
LOGDIR="" # location of logfiles
FILE_NAME=""
typeset -i LENGTH=0

# Secs since 1970
CURRENT_TIME=`$RETURN_SECS_SCRIPT`

ll ${LOGDIR} | while read line
do
FILE_NAME=`echo "$line"| awk '{print $NF}'| grep ".log$"`
# Length of filename
LENGTH="${#FILE_NAME}"
if [ "$LENGTH" -gt 0 ]
then
# Extract timespecs from filename
FILE_DAY=`echo $FILE_NAME | cut -c $(( $LENGTH - 18 ))-$(( $LENG
TH - 17 ))`
FILE_MONTH=`echo $FILE_NAME | cut -c $(( $LENGTH - 16 ))-$(( $LE
NGTH - 14))`
FILE_YEAR=`echo $FILE_NAME | cut -c $(( $LENGTH - 13 ))-$(( $LEN
GTH - 10))`
FILE_HOUR=`echo $FILE_NAME | cut -c $(( $LENGTH - 9 ))-$(( $LENG
TH - 8 ))`
FILE_MINUTE=`echo $FILE_NAME | cut -c $(( $LENGTH - 7 ))-$(( $LE
NGTH - 6 ))`
FILE_SECOND=`echo $FILE_NAME | cut -c $(( $LENGTH - 5 ))-$(( $LE
NGTH - 4 ))`
# How old is file..
FILE_TIME_STAMP=`$RETURN_SECS_SCRIPT 1970 $FILE_YEAR $FILE_MONTH
$FILE_DAY $FILE_HOUR $FILE_MINUTE $FILE_SECOND`
# How old is file in secs since now?
DIFFERENCE=$(( $CURRENT_TIME - $FILE_TIME_STAMP ))
# Is that above limit?
if [ "$DIFFERENCE" -gt "$FILE_AGE" ]
then
echo $FILE_NAME is to be removed
# rm $FILE_NAME
else
echo $FILE_NAME is not old enough
fi
fi
done

# End of script
Sorry about the missing indentions.

regards,
John K.

[Bala_8]

Thanks to Armin, Chris, Tom & John .....

I've tested with John's script and it's good ... but John, my file names are not fixed in length .... the file name format is *Date.log .....

see the output below :
fuji# ll |grep ".log$" |awk '{print $NF}'
biwpsnss005m.iwps.int30Dec2002134438.log
biwpsnss005m.iwps.int30Dec2002153119.log
biwpssp004d.iwps.ext09Jan2003150024.log
biwpssp004d.iwps.ext09Jan2003150040.log
biwpswa007d.iwps.int09Jan2003141625.log
biwpswa007d.iwps.int09Jan2003142218.log
biwpswa007d.iwps.int09Jan2003142341.log
biwpswa007d.iwps.int09Jan2003150420.log
biwpswa007d.iwps.int09Jan2003151043.log
biwpswa007d.iwps.int09Jan2003151433.log
ftp_recovery10Jan2003200423.log
ftp_recovery10Jan2003201529.log
ftp_recovery13Jan2003190819.log
ftp_recovery13Jan2003190951.log
ftp_recovery13Jan2003191048.log
ftp_recovery13Jan2003191507.log
ftp_recovery13Jan2003192100.log
fuji#

I think i need to change John's script to fetch time details by subtracting from the end ....

cheers
-bala-

[Bala_8]

Hi John,

ignore the previous contents ...

I get error in retun_seconds.sh while doing (09 - 1) .. ie ., the date is 09 or month is 09 ....

i suspect it occurs while doing

# Adjust entered time specs acc. to
# match date conventions.
#
# Subtract last month
#
MONTH=$(( $MONTH - 1 ));
#
# Subtract one day
#
DAY=$(( $DAY - 1 ))
#

in return_seconds.sh .....

The error message i receive is as below:

biwpsnss005m.iwps.int30Dec2002134438.log is not old enough
biwpsnss005m.iwps.int30Dec2002153119.log is not old enough
/tmp/script/return_seconds.sh[215]: 09 - 1 : The specified number is not valid for this command.
./log_retention.sh[30]: 7989910 - : Expression is not complete; more tokens expected.

How to resolve this ???
Thanks in advance...

-bala-

[john korterman]

Hi again Bala,
I have tried to reproduce the error on my system, using these file names:
BIWPSLB001fBatchLog
biwpsnss005m.iwps.int30Dec2002134438.log
BIWPSQO002mBatchLog
biwpssp004d.iwps.ext09Jan2003150024.log
biwpssp004d.iwps.ext09Jan2003150040.log
biwpswa007d.iwps.int09Jan2003141625.log
biwpswa007d.iwps.int09Jan2003142218.log
biwpswa007d.iwps.int09Jan2003142341.log
biwpswa007d.iwps.int09Jan2003150420.log
biwpswa007d.iwps.int09Jan2003151043.log
biwpswa007d.iwps.int09Jan2003151433.log
ftp_recovery10Jan2003200423.log
ftp_recovery10Jan2003201529.log
ftp_recovery13Jan2003190951.log
ftp_recovery13Jan2003191048.log
ftp_recovery13Jan2003191507.log
ftp_recovery13Jan2003192100.log

and I get this result:

biwpsnss005m.iwps.int30Dec2002134438.log is to be removed
biwpssp004d.iwps.ext09Jan2003150024.log is to be removed
biwpssp004d.iwps.ext09Jan2003150040.log is to be removed
biwpswa007d.iwps.int09Jan2003141625.log is to be removed
biwpswa007d.iwps.int09Jan2003142218.log is to be removed
biwpswa007d.iwps.int09Jan2003142341.log is to be removed
biwpswa007d.iwps.int09Jan2003150420.log is to be removed
biwpswa007d.iwps.int09Jan2003151043.log is to be removed
biwpswa007d.iwps.int09Jan2003151433.log is to be removed
ftp_recovery10Jan2003200423.log is to be removed
ftp_recovery10Jan2003201529.log is to be removed
ftp_recovery13Jan2003190951.log is not old enough
ftp_recovery13Jan2003191048.log is not old enough
ftp_recovery13Jan2003191507.log is not old enough
ftp_recovery13Jan2003192100.log is not old enough

i.e. "we could not reproduce the error on our system" - you have probably heard that before. But try running the return_seconds.sh on the command line, e.g.:
#
./return_seconds.sh 1970 2003 09 09 09 09 09
On my system it returns
1063098549

I cannot quite figure out what has caused the error, apart from the obvious that the script did not like the input. The return_seconds.sh does not check on any input as it is slow enough already. However, it only supports month names in English and Danish, e.g. for month no. 9 it accecps only the string "Sep" or "09" or "9", e.g.:
Sep|9|09) echo 9;;
Could that be the problem? How does month no. 9 appear in the file name causing the problem? Could you post the file name that causes the error?
I do not know when you have time for reading this, but in 22 hours from now I shall be on my way to Norway, skiing all of next week. It might therefore take a while before I can look into this again.


regards,
John K.

[Bala_8]

Hi John,

It occurs in date field ....
Exactly on the files created on *09Jan2003* ...

regards
-bala-

[Bala_8]

Hi John,

i edited your script to replace REFERENCE_YEAR to 2002 instead of 1970 .... i also made the same change in the main script to set the BASE YEAR as 2002.

The first two files contains 2002 timestamp, so they passed the test... but the third file onwards the timestamp is 2003 ....

For your info,
For some reasons our system date is set back to March 2002 and the current system date is 03/April/2002 .....

Will it cause any problems ???

cheers
-bala-

[john korterman]

Hi Bala,
changing the REFERENCE_YEAR should not make a big difference; the parameter only specifies to which year return_seconds.sh should count back; it should actually increase speed.
When executed without parameters, return_seconds.sh will then count back from April 3rd 2002 till midnight 1/1-2002. I had actually assumed that your system's date was past the date expressed in the file names, which it apparently is not! Your system's timesetting will cause a negative value when subtracting the number of seconds specified in the file names, from the number of seconds of your system's curent time. However, I do not think that the problem is there. Try executing return_seconds.sh with the parameters similar to those it gets from the file names causing the problem, in order to see which causes the crash. It should be something like:
# return_seconds.sh 1970 2003 Jan 09 15 14 33
As mentioned, I am not able to reproduce the error here. Can you reproduce it on another machine?

regards,
John K.

[john korterman]

Hi Bala,
I think that I know what is wrong, but as I am just about leaving, I do not have time to search for the full solution. Anyway, I am convinced that the problem is that on your system the number "09" inside the $(( )) is regarded by the shell to be a hexadecimal number.
If your system is a 11.00 and perhaps you should have patch PHCO_26789 installed; there may be other patches for this problem as well. If that is not the case, then search the forum for solutions to the "shell believes 09 inside $(()) is hex" problem.

regards,
John K.

[john korterman]

Hi Bala,
sorry, I am already in holiday mode: I of course mean that the shell belives "09" to be an octal number.
Alternatively, try to "typeset -i" the following variables in return_seconds.sh, e.g. like this:
typeset -i DAYS=0
typeset -i MONTH=0
typeset -i YEAR_TO_TEST=0
typeset -i HOUR=0
typeset -i SECS=0

regards,
John K.

[john korterman]

Sorry Bala,
please ignore the alternative solution suggested in my previous posting. It will not work, as the input for MONTH can be either numeric, e.g. "09" or string, e.g. "Sep". I suggest you look for a patch solution for making the shell accept that numbers inside $(()) preceeded by zero, e.g. "09", are not regarded as octal values.

regards and regrets,
John K.

[A. Clay Stephenson]

Thisd is actually very easy. I assume that you can use awk or cut to extract the Month, Day, and Year values. Caljd.sh can easily recoginze your month names directly and output the number of days since 4713BCE.

e.g

DT1=$(caljd.sh -i Dec 23 2002)
-- The value is 2452632
Now
DT2=$(caljd.sh -i Dec 25 2002)
-- The value is 2452634

Now
DAYS_DIFF=$((${DT2} - ${DT1}))

echo "Diff = ${DAYS_DIFF}"

Invoke as caljd.sh -u for usage and many examples.

[Bala_8]

Yes John,

You are correct .... it's treating $((09)) as hex .... i reproduced the error below :

fuji#
fuji# ./return_seconds.sh 2002 2002 Sep 09 10 10 10
./return_seconds.sh[215]: 09 - 1 : The specified number is not valid for this command.
fuji# ./return_seconds.sh 2002 2002 Sep 08 10 10 10
./return_seconds.sh[215]: 08 - 1 : The specified number is not valid for this command.
fuji# ./return_seconds.sh 2002 2002 Sep 07 10 10 10
21550210
fuji#
fuji#
fuji# ./return_seconds.sh 2002 2002 Sep 07 10 10 09
./return_seconds.sh[274]: 09 : The specified number is not valid for this command.
fuji#
fuji# ./return_seconds.sh 2002 2002 Sep 07 10 09 10
./return_seconds.sh[269]: 09 * 60: The specified number is not valid for this command.
fuji#
fuji# ./return_seconds.sh 2002 2002 Sep 07 09 10 10
./return_seconds.sh[264]: 09 * 60 * 60: The specified number is not valid for this command.
fuji#
fuji# ./return_seconds.sh 2002 2002 Sep 07 10 10 07
21550207
fuji#

I'm running HP-UX 11.00.
Since John is on vacation, I would like to request others to help me out. Will you guys ??

cheers
-bala-
-bala-

[A. Clay Stephenson]

Since you seemed to be stuck, I spent about 3 minutes whipping together a shell script which creates an awk script that it turns calls caljd.sh to do the calculations. Make sure that you grab caljd.sh from my earlier posting (and use the latest version 2.1). This should do the trick for you.

[A. Clay Stephenson]

I should add that it expects to use your directory listing (or file) as stdin. If your above input listing were 'myfiles.txt' then

deletethem.sh < myfiles.txt

You might also want to comment out the system() function that invokes rm until you feel safe.

[Bala_8]

Thanks Clay and thanks to John too, for your valuable time ...

Being a DBA it helped me a lot in UNIX scripting .....

cheers
-bala-