Avoid file appending while performing a mv

[Dave La Mar]

O.K. I guess I'm too stupid to be able to find reference to this using the search, tried, but alas, no luck.
Here's the scenario.
1. Third party product outputs data at unknown intervals (depends on outside sources sending data in.
2. The third party data output is appended to a file.
3. Another process (owned by us) check to see if the file is no longer being appended to (2 minute intervals).
4. If test for "appending stopped" passes, we have another process the renames the file.

Problem:
With no indication of when additional data may come in, we have found the file appended to with partial data of another record while the move is being performed.
Solution search:
Is there a way to lock the file at the point of "appending stopped"?
How does one truly insure a file has completed the append process in order to rename, copy, etc.?
Currently, we are checking the file size for number of lines, two minutes later checking again; if line count did not change, next process acts on file. But, on occasion another partial record sqeaks through.

Appreciate any and all input.

Regards,

dl

[Mark Grant]

Firstly, moving the file will not necessarily stop a process writing to it. If the process still has the file open when it is moved, then it will still write to it.

You are always going to have a potential race condition within that two minutes and therefore you are always potentially going to get bits of data that you weren't expecting.

If you really can just lock the file without breaking your third party application, just remove write access from it while you work with it

The real solution to this though is to make your third party data file a named pipe and have a process reading from that pipe that can analyse the data coming through it, make proper decisions and write out to an appropriate output file.

[Dave La Mar]

Never mind forumers. After brainstorming with a couple of guys I have a number of ways to accomplish this, from the complicated to the "too" simple.
Sorry for the spam of this post.
Consider this closed.

dl

[Dave La Mar]

Thanks Mark. Named pipes was one of the options we discussed.
Here's my intent at this point.
Since doing a line count is being used to determine when the file is no longer being appended to, I intend to use the value to sed out those lines to a new file and proceed from there.
Though not fool proof, it is highly unlikely that a record will not finish writing out in that two minute window. (records are small)
Thanks again.

[Volker Borowski]

Hi,

"fuser" should be the one to handle this.
And your "mv" should not go directly to a diffrent filesystem, because this could take some time, in which the file coud be re-opened.
So do a rename first (changes only the inode) and the "mv" to diffrent filesystem later.

"man-page"
...
fuser returns a non-zero return code if none of the specified files is accessed or in case of a fatal error. If at least one access has been found, fuser returns zero.
...


Best chance for this aproach

if fuser ....
then
echo Someone on it, try later .
else
mv file file.safe
touch file .....in case you need an existing one
mv file.safe /difrent_filesystem/file
fi

[Dave La Mar]

Thanks Volker. That was an option I did not consider.
Presently, we mv the file when the line count remains the same over a two minute period. The developer created some of the problem by having a second process run that was not as fast as a move after determining appending had stopped.

We'll let him run with doing the move immediately upon "appending stopped".

If that doesn't work out, I'll move on with the posted suggestions.

Thanks again.