file re-created? How to know?

[yyghp]

Since we can not find the file creation time in HP-UX,
( See this:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=203044
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=832810
)
So how can we determine whether the file has been re-created or not?
I need to find out whether someone or process remove the file and creat it again, instead of just updating it. ( The case is that the log files keep being modified by the application, which is normal I don't need to care, but our support people may remove that log file manually, then the application will generate a new file with same name, but I need to find out this file is a NEW file, then I can reset my flag in my script file, which scan such long file every 10 minutes. )

I tried "inode number", but it didn't work for me, because when I tried to remove a file and "touch" a new file with same name, it kept the same inode number, which I cannot use to determine whether it has been removed and new created.

Please give me suggestion. Thanks a lot!

[James R. Ferguson]

Hi:

A "good" log file would have some indication of the time of an event beyond just the ordinal occurance of an event. '/var/adm/syslog/syslog.log' would be one example of this.

Why not retain the knowledge of the last record's timestamp and then report from that point forward when you begin a new process cycle?

Regards!

...JRF...

[Prashant Zanwar_4]

New file cksum will be all different than old one..

Thanks
Prashant

[Prashant Zanwar_4]

New file cksum will be all different than old one..

$ cksum pz
1864731933 4 pz
$ rm pz
$
$ touch pz
$ echo 1 > pz
$ cksum pz
4219530715 2 pz

Thanks
Prashant

[Rodney Hills]

If you want to track file operations, then could try HP's HIDS applications.

See-
http://h20293.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPUX-HIDS

HTH

-- Rod Hills

[A. Clay Stephenson]

In your particular case, the ctime is sufficient because it would be highly unlikely the someone would chown or chmod this file --- if so, it would appear as a new file to you.

The attached Perl script, fileage.pl, will probably be of use.

I would store the last time in a file (if this file is not found then it is considered t new file).

#!/usr/bin/sh

typeset -i LASTTIME=0
typeset -i THISTIME=0
typeset TDIR=${TMPDIR:-/var/tmp}
typeset TSFILE=${TDIR}/logtimestamp
typeset LFILE=/xxx/yyy/mylog
itypeset -i STAT=0

if [[ -r ${TSFILE} ]]
then
LASTTIME=$(cat ${TSFILE})
else
LASTTIME=-1
fi
if [[ -r ${LFILE} ]]
then
THISTIME=$(fileage.pl -c -e)
if [[ ${THISTIME} -gt ${LASTIME} ]]
then
echo "New file"
echo "${THISTIME}" > ${LFILE}
else
echo "Old file"
fi
else
echo "No logfile, ${LFILE}, found." >&2
STAT=2
fi
exit ${STAT}

------------------------------------
You would need to make sure that the Perl script is executable and is in your PATH (as well as the perl executable itself).

Invoke as fileage.pl -u for full usage.

Cksum would be a rather pointless metric in this case be presumably a logfile would be undergoing nearly constant modification.

[yyghp]

Hi James,
The logs are generated by third party application, which is out of our control and we can't manually modify them.

Hi Prashant,
I can't use "cksum", because the logs are being modified by the application, while I just want to know whether it has been new created or not. "cksum" cannot distinguish "re-created" and "modified".

Hi Rodney,
My script is to monitor and filter critical errors in the application logs, then notify our operators. I don't want HIDS to be involved because of such feature.

Hi A.Clay,
Is there any command to easily get "ctime" of the file?

Thanks a lot!

[A. Clay Stephenson]

I sometimes wonder why I bother. Did you even try the Perl script? If you had, you would have found that "fileage.pl -c -e filename" would output exactly what you need although not in a form that you would easily recognize. It outputs in epoch seconds such as 1143257372. However, these epoch seconds make the comparisons in the script very easy. If you want to to see the timestamp in a more meaningful form then
perl -e 'print scalar localtime(1143257732)'. You can also do an ls -lc myfile to display the change time of a file but that format is not easy to use for calculations.

[A. Clay Stephenson]

Ooops, this line:
echo "${THISTIME}" > ${LFILE}
should be changed to:
echo "${THISTIME}" > ${TSFILE}

because this is where we store the timestamp when a newly detected file is found.

[James R. Ferguson]

Hi :

If all you want is to capture the 'ctime' of a file for tracking, do:

# perl -le 'print 0+(stat($ARGV[0]))[10]' filename

If the file doesn't exist, zero (0) will be returned.

In a shell script:

# ct=`perl -le 'print 0+(stat($ARGV[0]))[10]'`

Regards!

...JRF...

[yyghp]

Hi A.Clay,
I haven't tried your Perl script, I am sure it can find the "ctime". However, I wonder whether I could use only simple command to get "ctime", instead of using a long perl.
I tried this:

# ls -lc abc
-rw-r--r-- 1 root sys 14 Mar 30 16:06 abc

# echo "abc" >> abc

# ls -lc abc
-rw-r--r-- 1 root sys 18 Mar 30 16:07 abc

Why did the "ctime" changed? no matter I used ">>" or "vi" to modify the file "abc".

Thanks!

[yyghp]

Hi James,

I tried your "# perl -le 'print 0+(stat($ARGV[0]))[10]' filename", but it returned different number if I modified the file, but I did NOT re-create it, why?

# perl -le 'print 0+(stat($ARGV[0]))[10]' abc
1143752821

# echo "abc" >> abc

# perl -le 'print 0+(stat($ARGV[0]))[10]' abc
1143753200

[A. Clay Stephenson]

Well, I'll be horn-swaggled by a horny-toad. Much to my surprise I found this in the man pages of the write() system call:

"Upon successful completion, where nbyte is greater than 0, write() will mark for update the st_ctime and st_mtime fields of the file."

This sure ain't the way my AT&T UNIX manuals describe write() so somewhere along the line, write() has done got itself "improved".

This means than anytime a write() is done both mtime (expected) and ctime(unexpected) are updated.

Thus ctime is pretty useless for your purposes under HP-UX. I would fuss at HP about this but they have done gone and documented the behavior so it's my own fault for thinking no one would "improve" write()'s behavior.

[yyghp]

Thanks a lot A. Clay !
So, any suggestion to determine whether the file has been recently created?
Thanks again!

[A. Clay Stephenson]

Well now that I know that the write system call always updates ctime and mtime there really is no way --- and even my previous assumption assumed that no one would chmod or chown the file after it was created. The only observation I am left with is this:

The logfile will grow over time so if current size of the file < stored size of the file then it is a new file. This is not fool proof. You can get the size of a file using ls -l or if you change the "[10]" in James' Perl one-liner to "[7]" that is the size of the file in bytes.

Without knowing what the file looks likes it's diifcult to know but generally a timestamp is written into the first few lines of a logfile and you could use that when comapred against your last time stamp to determine if this is a new log.

[A. Clay Stephenson · Accepted solution]

Here is another way. Suppose that you check to see if the setuid (04000) mode bit is set on this file. If not, do a chmod setting this bit and that will be your flag that this is a new file. The setuid bit has no significance to a data file so it should not interfere with the normal function of your application. The setgid (02000) bit implies mandatory file locking and the sticky-bit (01000) is used as a transition link flag so those could have unintended consequences when applied to a data file but the setuid bit should have no unforeseen side effects.

[yyghp]

Thanks A.Clay!

Since the new created file may be bigger or smaller than the orignial file, so I don't think we can use file size to determine if it is a new log.

About changing the mode bit, that's a good tip! However, is there any file "attribute" that I can use to be a better "flag"? I am looking for a flag which I can modify and read the value of it, but won't be questioned by other people.

Thanks!

[A. Clay Stephenson]

If there had been one, I would have suggested it. The setuid bit will have zero impact. You could set one or more of the execute bits; they too have no meaning for data files but the setuid bit is the least evil.

[yyghp]

Thanks A.Clay!
Sorry, maybe I have misunderstood what you said about "setuid (04000) mode bit is set on this file".
I use command:
# chmod 04000 abc
then when I used "ll":
# ll
---S------ 1 root sys 12 Mar 30 20:53 abc
But I have to keep it like "-rw-r--r--".
I think I was wrong, but please tell me how to "setuid".
Thanks!

BTW, can I use the "acl" file attribute as a flag?

[A. Clay Stephenson]

I said set the setuid bit; not blast the remaining mode bits. In your case that would be chmod 4644. Whatever you do, don't also set any execute bits on this file since it is owned by root. It's at least potentially possible to coerce this file into a real setuid program owned by root. It would be better still if this were a file not owned by root.

[JASH_2]

yyqhp,

When I had to set a script to check for errors and report by email if there were any, I copied the log file I was checking out to a file with a date stamp after it:-

#cp -p logfile logfile'date'#with syntax for date/time required.
#cat /dev/null > logfile

When there was an error in it. This means that if there was an error a new file was always created, so I was never looking at an old error.

Simple but effective!

Regards,

JASH

[JASH_2]

May not have made it clear, but only copied file out if there was an error!

Regards,

JASH

[yyghp]

Hi JASH,

But the error log file of the application will be added more NEW error into the file, and every time I scan this error log file, I don't want to report those errors I have sent to the operator. I have to use another file to remember which line I scan last time.
So, have a copied file won't help much.
Thanks!

[James R. Ferguson]

Hi (again):

You don't say how you scan your log. I would assume that you retain some state information about where you last left off and begin a scan from that point forward.

If that's true, you could simply conclude that the absence of a restart point means that a new log has been created and that you should report any events from the beginning.

Regards!

...JRF...

[yyghp]

Hi James,

Thanks! But how can I conclude that the absence of a restart point when a new log has been created?
Because I schedule to check the log file every 5 minutes for example, first I need to know whether or not it's a new log, if yes, I reset the pointer to line 1, if not, I continue to scan the log file from the last line ( I record this number in another status file. ) I scan last time.
Now, I can use A.Clay's suggestion to set the permission mode: everytime I scan the log, check whether it has "S" or not, if yes, it means old log, then keep scan from last line as record, if no, then set "S", and reset pointer to line #1...
Any other suggestion about marking a better "flag" for such purpose is very welcome.
Thanks guys!