What happens with files being created while I'm moving all files in the directory?

[Eric Antunes]

I'm defining a script to run at night with the following commands:

mydir="/.../admin/arch"
mv mydir/* /NFS_MOUNT_POINT/ARCH_PROD

What happens if a file is being created in mydir while files get moved? Is the file moved as it is, without ending his creation??

Best Regards,

Eric Antunes

[Pete Randall]

Eric,

Your regular expression is going to be expanded when the shell processes the "mydir/*", resulting in a list of files present at that partiular moment in time. Files added to the directory after that will be ignored.


Pete

[Bill Hassell · Accepted solution]

mv is a combination of copy and remove, so the file continues the copy process until it is complete and then it is removed. If a single file starts the copy process, it is open and some other process that removes or moves that same file will continue too except the remove step marks the file as removed but the contents are still there and available to the first copy. Once the file is closed by all processes, then it will be truly removed.

The steps to removal of a file all relate to the directory, not the contents. A file can disappear (be removed) from the directory but if it is still open, the program(s) will still be able to access the file. When a file is first opened, all the information needed to access the file is stored in the program's file descriptor, so the directory is no longer needed. Once the file is completely closed, then the inodes pointing to the file are marked as unused in the directory.

[Ivan Ferreira]

I tested in this way:

Session 1:

mkdir /var/tmp/test
cd /var/tmp/test
dd if=dev/zero of=bigfile


Session 2:

cd /var/tmp
mv test test.bak -----> sucess
ls test.bak
bigfile

Session 1:

pwd
permission denied


Conclusion: Because the movement is on the same file system, the directory is moved and the file keeps growing (writing data). After that, the path is invalid and you will get permission denied. The data is available on the new path.

[Ivan Ferreira]

Test two, moving between file system:

I tested in this way:

Session 1:

mkdir /var/tmp/test
cd /var/tmp/test
dd if=dev/zero of=bigfile


Session 2:

cd /var/tmp
mv test /usr/tmp -----> Waiting for the write command to finish

Session 1:

CTRL+C ------------> Session 2 finished the move of the directory
pwd
permission denied

Conclusion: Because the movement is on the differents file systems, the directory is moved after the open file is closed. After that, the path is invalid and you will get permission denied. The data is available on the new path.

[Pete Randall]

My apologies, Eric. I'm afraid I misinterpreted your question. Bill's response seems to better address what you really wanted to know.


Pete

[Eric Antunes]

Thank you all!

That was the behaviour I was expecting but since I have the database in archive mode and a different behaviour could crash it, I wanted to be sure. :)

Best Regards,

Eric Antunes

[Eric Antunes]

What would be the command within a find to move just those files with more than 14 days? Something like this (?):

find $mydir -name "*.arc" -ctime +14 -exec mv /NFS_MOUNT_POINT/ARCH_PROD

Best Regards,

Eric Antunes

[James R. Ferguson]

Hi Eric:

Be careful here.

Consider that a 'mv' can be a simple ;rename(2)' or a copy and remove ('unlnk(2)'). A rename occurs only if file remains in the *same* filesystem.

If you 'mv' within a filesystem, then the data being added by your process to its file continues to be written into the renamed file (same inode). That is, your process creates a file with the same data that it would have had the file not been renamed.

If, however, your 'mv' occurs accross filesystems, then the final file contents will only reflect what you had at the time the copy occured. New data created by your process after the moment of the copy will be lost.

Regards!

...JRF...

[Pete Randall]

Eric,

I would use

"find $mydir -name "*.arc" -mtime +14 -exec mv /NFS_MOUNT_POINT/ARCH_PROD {} \;"

-or-

"find $mydir -name "*.arc" -mtime +14 |xargs mv /NFS_MOUNT_POINT/ARCH_PROD "


Pete

[Eric Antunes]

James,

"If, however, your 'mv' occurs accross filesystems, then the final file contents will only reflect what you had at the time the copy occured. New data created by your process after the moment of the copy will be lost."

That's the situation: I want to move the archivelog files from a server to another (via NFS).

You are telling me that:

mv $mydir/* /NSF_MOUNT_POINT/ARCHPROD

is different than:

find $mydir -name "*.arc" -ctime +14 -exec mv {} \;

Those 2 expressions acts differently??

Thanks,

Eric Antunes

[Eric Antunes]

In my last post instead of:

find $mydir -name "*.arc" -ctime +14 -exec mv {} \;

I wanted to write:

find $mydir -name "*.arc" -ctime +14 -exec mv /NFS_MOUNT_POINT/ARCH_PROD {} \;

[James R. Ferguson]

Hi (again) Eric:

Yes, I am saying that if you are crossing mountpoints, then executing a 'mv' command becomes a copy and not a 'rename' and what I said previously applies.

If you have an active process that is writing to a file, you do not want to move/copy the file until you are finished writing to it.

One way to handle this kind of situation is to preface the move/copy with a test that attempts to see if any process is using the file. I have used 'fuser' to look for processes actively associated with the file in situations like this. If the file is inuse, wait until its not and then copy it, or skip it entirely.

Regards!

...JRF...

[Eric Antunes]

Ok,

Since, I'm sure that no process will be writing to files with more than 1 day since the files are just created and never changed (they are archive logs), I will use the find first and move (copy and delete) just files older than 14 days:

find $mydir -name "*.arc" -ctime +14 -exec mv /NFS_MOUNT_POINT/ARCH_PROD {} \;

Thanks to all,

Eric Antunes