how to list all changed/added files under file systems?

[Hanry Zhou]

I have several file systems, and wanted to list all chnaged/added files under these fs's in last 24 hours. Please help me with commands. Thanks,

[TwoProc]

Well, to do this properly you should be using something like TripWire.

Short of that, go to each file system and run:

cd /filesystema
find . -xdev -depth -mtime 0 -exec ls -al {} \; > /tmp/changed_file_list.log

If you want to do the whole server and are willing to wait...

cd /
find . -depth -mtime 0 -exec ls -al {} \; > /tmp/changed_file_list.log

[James R. Ferguson]

Hi Hanry:

If by "changed" you mean 'modified' do:

# find /path -xdev -mtime -1

This finds all files in /path that have been modified in the last 24-housrs where each increment equals 24-hours.

If you mean an "chage" in permissions, ownership or modification, use:

# find /path -xdev -ctime -1

The 'ctime' tracks changes in a file's inode.

See the manpages for 'find' for more information!

...JRF...

[TwoProc]

In re-reading my prev post, I realized you may not understand how "-mtime 0" gives you "the last 24 hours". This is because that mtime argument is number of days. Day 0 is the last 24 hours (it considers 24 hours from now a "day" - not to be confused with calendar days).

0 = last 24 hours
1 = 24 to 48 hours ago
+1 = more than 48 hours ago
+0 = more than 24 hours ago

etc...

[Hanry Zhou]

can I use "rsync" command to achieve the same? Because it'd be better for me to use rsync.

[TwoProc]

I thought you wanted just to list the files that have changed over the last 24 hours.

Rsync is a tool to sync two file systems together,
but ...

You "can" use rsync in a "dry-run" mode to see the files that need to change since the last rsync, IF that was 24 hours ago, AND the source machine that you're rsyncing from *hasn't* changed in the last 24 hours. It can get a little funny over file deletes though...

man rsync and look for the "dry-run" option...
"--dry-run" would be the command to merely review the changes that rsync would make.

[A. Clay Stephenson]

Rsync is normally used to sync one set of files to another; you could use the -n option to show what files would be copied but in order for that to work you would have to have a reference filetree with which to compare. This is really not a task for rsync. Find will do what you asked; however; (more or less) What about files that have been removed? There is really no easy answer for those unless you do a find each day and list the files and the timestamps and compare that to the current list.

[James R. Ferguson]

Hi (again) Hanry:

You're original query suggested that all you wanted was to list files that have been changed or added during the last 24-hours.

The 'find' command used as I showed accomodates this quite easily. You can redirect your output to a file if you want!

Remote file synchronization/distribution is designed to tranfer from one server to another files that have been changed. You don't need that layer simply to see files modified/changed!

Regards!

...JRF...

[Hanry Zhou]

I tried both following commands as suggested, but they all also pulled out files not in last 24 hours. I believe, the reason is because they listed these files undernead directories that have been modified/new created in last 24 hours. How can we list files, and not old files under the new directories? I need -exec ls -lt, because I need to have the full names, and therefore I can caclulate the total space of being occupied by these files.


find . -xdev -depth -mtime 0 -exec ls -al {} \; > /tmp/changed_file_list.log

# find /path -xdev -mtime -1 -exec ls -at {} \;

[James R. Ferguson · Accepted solution]

Hi Hanry:

I'm sorry, I should have restricted things to *files* only:

# find /path -xdev -type f -mtime -1
# find /path -xdev -type f -ctime -1

Also, far kinder on your system's performance is the use of 'xargs' instead of '-exec':

# find /path -xdev -type f -mtime -1|xargs ls -l

...The above does the same thing as:

# find /path -xdev -type f -mtime -1 -exec ls -l {} \;

...but the 'xargs' filter bundles *multiple* filenames and runs then as "one" instatantiation of 'ls'.

Please see the 'xargs' manpages for more information.

Regards!

...JRF...

[A. Clay Stephenson]

This would really be better and more efficiently done with Perl's File::Find module because you have everything you need including the stat() function so that you wouldn't need an external ls -l command. In any event, your find command needs to only do regular files because when you do an ls -l on a directory the files underneath are listed as well.

add "-type f" to your find command so that only regular files with match.

[A. Clay Stephenson]

Okay, this should do what you want. It will list each regular file and its size that has been modified within the last 24 hours.

Use it like this:

find24.pl /aaa /bbb/ccc /xxx

[Hanry Zhou]

It works now.

I am sorry, but my next follow-up question is how can we sum up the 5th field (amount of space) of each one of files together? I wanted to calcuate how much these new created/modified files will occupy. Thanks,

[James R. Ferguson]

Hi Hanry:

...and if you wanted to do this with Perl, as Clay suggests:

# perl -MFile::Find -wle 'File::Find::find(sub{print $_," ",scalar localtime ((stat(_))[9]),"\n" if -f && -M _ <1},@ARGV)' /path

...would give you the filenames and modification (mtime) times for all files below "/path".

Change "[9]" to "[10]" if you want the 'ctime' instead of the 'mtime'.

Regards!

...JRF...

[James R. Ferguson]

Hi (again) Hanry:

If you want the total size summation per your last question, just do this:

# perl -MFile::Find -wle 'File::Find::find(sub{$size+=((stat(_))[7]) if -f && -M _ <1},@ARGV);END{print $size}' /path

Regards!

...JRF...

[Hanry Zhou]

James,

If you can, it'd be better if you can use ksh, since we don't know perl here.

[A. Clay Stephenson]

Then you should learn Perl because the Perl solution is going to execute ~50x as fast as a shell equivalent and since the code is already written, you don't need to know any Perl.

[James R. Ferguson]

Hi Hanry:

OK, if you won't use Perl, even though you have it installed, here's a *pure* shell approach to totalling the file sizes in bytes for all files in "/path" that have been modified during the last 24-hours.

#!/usr/bin/sh
typeset -i TOTSIZE;
find /tmp -xdev -type f -mtime -1 -exec ls -l {} \; | \
while read X X X X SIZE D1 D2 D3 NAME
do
echo "${NAME} ${SIZE} ${D1} ${D2} ${D3}"
TOTSIZE=$(( ${TOTSIZE} + ${SIZE} ))
done
echo "\nTotal Size = $TOTSIZE"
exit 0

Regards!

...JRF...

[James R. Ferguson]

Hi Hanry:

OK, if you won't use Perl, even though you have it installed, here's a *pure* shell approach to totalling the file sizes in bytes for all files in "/path" that have been modified during the last 24-hours.

#!/usr/bin/sh
typeset -i TOTSIZE;
find /tmp -xdev -type f -mtime -1 -exec ls -l {} \; | \
while read X X X X SIZE D1 D2 D3 NAME
do
echo "${NAME} ${SIZE} ${D1} ${D2} ${D3}"
TOTSIZE=$(( ${TOTSIZE} + ${SIZE} ))
done
echo "\nTotal Size = $TOTSIZE"
exit 0

Regards!

...JRF...

[A. Clay Stephenson]

In addition to Perl's much superior performance there is still another reason
that you shouldn't use the shell for this: possible integer overflow. If enough files are changed within the timeframe to overflow the shell's 32-bit signed integers then your results are bogus. This can be avoided by using bc to do the may=th but at a cost of even poorer performance.

Here's a refined Perl script that will do everything;

Use it like this:

finder.pl -g -s 86400 /dir1 /dir2 ...

This will descend each directory listed and give a total for each for all regular files that have been modified within the last 86400 seconds (1 day) and (-g) list a grand total. Because I intentionally did not do integer arithmetic, the totals will not overflow.

Invoke as finder.pl -u for full usage.

[Hanry Zhou]

well, what I really need is not only list all new/modifiled files, also be able to copy them over to the remote site. So what I am really looking for is rsync tool. can anybody please let me know the option of rsync being able to do that?

Sorry for confusing, and appeciate your messages.

[James R. Ferguson]

Hi Hanry:

Hanry, this would be easier if you more clearly stated the objective you want met. That said, however, have a look at 'rdist' and its manpages. Examples are included therein:

http://www.docs.hp.com/en/B2355-60127/rdist.1.html

Regards!

...JRF...

[A. Clay Stephenson]

Life would have been much easier had you just indicated what you were trying to do. You can get rsync from here:
http://hpux.cs.utah.edu/hppd/hpux/Networking/Admin/rsync-2.6.8/

One very nice feature of rsync over rdist it the ability to transfer just the parts of a file that have changed -- and this becomes very important over busy or limited-bandwidth connections. The downside to the partial transmissions is that more computation has to be done so you will need to try it both ways to find out which is better for your application. The -n option while actully list whay rsync would do without actually doing the transfer. You really shouldn't need anymore than the rsync man pages as there are several examples.