topic Re: file re-created? How to know? in Operating System - HP-UX

file re-created? How to know?

yyghp — Thu, 30 Mar 2006 14:41:21 GMT

Since we can not find the file creation time in HP-UX,
( See this:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=203044
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=832810
)
So how can we determine whether the file has been re-created or not?
I need to find out whether someone or process remove the file and creat it again, instead of just updating it. ( The case is that the log files keep being modified by the application, which is normal I don't need to care, but our support people may remove that log file manually, then the application will generate a new file with same name, but I need to find out this file is a NEW file, then I can reset my flag in my script file, which scan such long file every 10 minutes. )

I tried "inode number", but it didn't work for me, because when I tried to remove a file and "touch" a new file with same name, it kept the same inode number, which I cannot use to determine whether it has been removed and new created.

Please give me suggestion. Thanks a lot!

Re: file re-created? How to know?

James R. Ferguson — Thu, 30 Mar 2006 14:50:50 GMT

Hi:

A "good" log file would have some indication of the time of an event beyond just the ordinal occurance of an event. '/var/adm/syslog/syslog.log' would be one example of this.

Why not retain the knowledge of the last record's timestamp and then report from that point forward when you begin a new process cycle?

Regards!

...JRF...

Re: file re-created? How to know?

Prashant Zanwar_4 — Thu, 30 Mar 2006 14:55:21 GMT

New file cksum will be all different than old one..

Thanks
Prashant

Re: file re-created? How to know?

Prashant Zanwar_4 — Thu, 30 Mar 2006 14:56:30 GMT

New file cksum will be all different than old one..

$ cksum pz
1864731933 4 pz
$ rm pz
$
$ touch pz
$ echo 1 > pz
$ cksum pz
4219530715 2 pz

Thanks
Prashant

Re: file re-created? How to know?

Rodney Hills — Thu, 30 Mar 2006 14:56:52 GMT

If you want to track file operations, then could try HP's HIDS applications.

See-
http://h20293.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPUX-HIDS

HTH

-- Rod Hills

Re: file re-created? How to know?

A. Clay Stephenson — Thu, 30 Mar 2006 15:19:24 GMT

In your particular case, the ctime is sufficient because it would be highly unlikely the someone would chown or chmod this file --- if so, it would appear as a new file to you.

The attached Perl script, fileage.pl, will probably be of use.

I would store the last time in a file (if this file is not found then it is considered t new file).

#!/usr/bin/sh

typeset -i LASTTIME=0
typeset -i THISTIME=0
typeset TDIR=${TMPDIR:-/var/tmp}
typeset TSFILE=${TDIR}/logtimestamp
typeset LFILE=/xxx/yyy/mylog
itypeset -i STAT=0

if [[ -r ${TSFILE} ]]
then
LASTTIME=$(cat ${TSFILE})
else
LASTTIME=-1
fi
if [[ -r ${LFILE} ]]
then
THISTIME=$(fileage.pl -c -e)
if [[ ${THISTIME} -gt ${LASTIME} ]]
then
echo "New file"
echo "${THISTIME}" > ${LFILE}
else
echo "Old file"
fi
else
echo "No logfile, ${LFILE}, found." >&2
STAT=2
fi
exit ${STAT}

------------------------------------
You would need to make sure that the Perl script is executable and is in your PATH (as well as the perl executable itself).

Invoke as fileage.pl -u for full usage.

Cksum would be a rather pointless metric in this case be presumably a logfile would be undergoing nearly constant modification.

Re: file re-created? How to know?

yyghp — Thu, 30 Mar 2006 15:41:10 GMT

Hi James,
The logs are generated by third party application, which is out of our control and we can't manually modify them.

Hi Prashant,
I can't use "cksum", because the logs are being modified by the application, while I just want to know whether it has been new created or not. "cksum" cannot distinguish "re-created" and "modified".

Hi Rodney,
My script is to monitor and filter critical errors in the application logs, then notify our operators. I don't want HIDS to be involved because of such feature.

Hi A.Clay,
Is there any command to easily get "ctime" of the file?

Thanks a lot!

Re: file re-created? How to know?

A. Clay Stephenson — Thu, 30 Mar 2006 15:48:47 GMT

I sometimes wonder why I bother. Did you even try the Perl script? If you had, you would have found that "fileage.pl -c -e filename" would output exactly what you need although not in a form that you would easily recognize. It outputs in epoch seconds such as 1143257372. However, these epoch seconds make the comparisons in the script very easy. If you want to to see the timestamp in a more meaningful form then
perl -e 'print scalar localtime(1143257732)'. You can also do an ls -lc myfile to display the change time of a file but that format is not easy to use for calculations.

Re: file re-created? How to know?

A. Clay Stephenson — Thu, 30 Mar 2006 15:51:01 GMT

Ooops, this line:
echo "${THISTIME}" > ${LFILE}
should be changed to:
echo "${THISTIME}" > ${TSFILE}

because this is where we store the timestamp when a newly detected file is found.

Re: file re-created? How to know?

James R. Ferguson — Thu, 30 Mar 2006 15:55:39 GMT

Hi :

If all you want is to capture the 'ctime' of a file for tracking, do:

# perl -le 'print 0+(stat($ARGV[0]))[10]' filename

If the file doesn't exist, zero (0) will be returned.

In a shell script:

# ct=`perl -le 'print 0+(stat($ARGV[0]))[10]'`

Regards!

...JRF...

Re: file re-created? How to know?

yyghp — Thu, 30 Mar 2006 16:10:39 GMT

Hi A.Clay,
I haven't tried your Perl script, I am sure it can find the "ctime". However, I wonder whether I could use only simple command to get "ctime", instead of using a long perl.
I tried this:

# ls -lc abc
-rw-r--r-- 1 root sys 14 Mar 30 16:06 abc

# echo "abc" >> abc

# ls -lc abc
-rw-r--r-- 1 root sys 18 Mar 30 16:07 abc

Why did the "ctime" changed? no matter I used ">>" or "vi" to modify the file "abc".

Thanks!

Re: file re-created? How to know?

yyghp — Thu, 30 Mar 2006 16:18:57 GMT

Hi James,

I tried your "# perl -le 'print 0+(stat($ARGV[0]))[10]' filename", but it returned different number if I modified the file, but I did NOT re-create it, why?

# perl -le 'print 0+(stat($ARGV[0]))[10]' abc
1143752821

# echo "abc" >> abc

# perl -le 'print 0+(stat($ARGV[0]))[10]' abc
1143753200

Re: file re-created? How to know?

A. Clay Stephenson — Thu, 30 Mar 2006 17:09:45 GMT

Well, I'll be horn-swaggled by a horny-toad. Much to my surprise I found this in the man pages of the write() system call:

"Upon successful completion, where nbyte is greater than 0, write() will mark for update the st_ctime and st_mtime fields of the file."

This sure ain't the way my AT&T UNIX manuals describe write() so somewhere along the line, write() has done got itself "improved".

This means than anytime a write() is done both mtime (expected) and ctime(unexpected) are updated.

Thus ctime is pretty useless for your purposes under HP-UX. I would fuss at HP about this but they have done gone and documented the behavior so it's my own fault for thinking no one would "improve" write()'s behavior.

Re: file re-created? How to know?

yyghp — Thu, 30 Mar 2006 17:20:06 GMT

Thanks a lot A. Clay !
So, any suggestion to determine whether the file has been recently created?
Thanks again!

Re: file re-created? How to know?

A. Clay Stephenson — Thu, 30 Mar 2006 17:35:11 GMT

Well now that I know that the write system call always updates ctime and mtime there really is no way --- and even my previous assumption assumed that no one would chmod or chown the file after it was created. The only observation I am left with is this:

The logfile will grow over time so if current size of the file < stored size of the file then it is a new file. This is not fool proof. You can get the size of a file using ls -l or if you change the "[10]" in James' Perl one-liner to "[7]" that is the size of the file in bytes.

Without knowing what the file looks likes it's diifcult to know but generally a timestamp is written into the first few lines of a logfile and you could use that when comapred against your last time stamp to determine if this is a new log.

Re: file re-created? How to know?

A. Clay Stephenson — Thu, 30 Mar 2006 17:57:46 GMT

Here is another way. Suppose that you check to see if the setuid (04000) mode bit is set on this file. If not, do a chmod setting this bit and that will be your flag that this is a new file. The setuid bit has no significance to a data file so it should not interfere with the normal function of your application. The setgid (02000) bit implies mandatory file locking and the sticky-bit (01000) is used as a transition link flag so those could have unintended consequences when applied to a data file but the setuid bit should have no unforeseen side effects.

Re: file re-created? How to know?

yyghp — Thu, 30 Mar 2006 19:07:51 GMT

Thanks A.Clay!

Since the new created file may be bigger or smaller than the orignial file, so I don't think we can use file size to determine if it is a new log.

About changing the mode bit, that's a good tip! However, is there any file "attribute" that I can use to be a better "flag"? I am looking for a flag which I can modify and read the value of it, but won't be questioned by other people.

Thanks!

Re: file re-created? How to know?

A. Clay Stephenson — Thu, 30 Mar 2006 20:31:15 GMT

If there had been one, I would have suggested it. The setuid bit will have zero impact. You could set one or more of the execute bits; they too have no meaning for data files but the setuid bit is the least evil.

Re: file re-created? How to know?

yyghp — Thu, 30 Mar 2006 21:25:20 GMT

Thanks A.Clay!
Sorry, maybe I have misunderstood what you said about "setuid (04000) mode bit is set on this file".
I use command:
# chmod 04000 abc
then when I used "ll":
# ll
---S------ 1 root sys 12 Mar 30 20:53 abc
But I have to keep it like "-rw-r--r--".
I think I was wrong, but please tell me how to "setuid".
Thanks!

BTW, can I use the "acl" file attribute as a flag?

Re: file re-created? How to know?

A. Clay Stephenson — Thu, 30 Mar 2006 23:13:11 GMT

I said set the setuid bit; not blast the remaining mode bits. In your case that would be chmod 4644. Whatever you do, don't also set any execute bits on this file since it is owned by root. It's at least potentially possible to coerce this file into a real setuid program owned by root. It would be better still if this were a file not owned by root.