fseek and ftell on stdout (i.e. SYS$OUTPUT)

[John McL]

We would like to use C functions ftell() and fseek() to extract records from a log file during execution of a detached process. (This is web access stuff and during development we'd like to post back, via the intranet, the records from the log file.)

Attempts to use C's ftell() on stdout - i.e. SYS$OUTPUT, the log file for the detached process - fail with status EOF when accessing the log file that was defined in the call to SYS$CREPRC but after I use 'freopen(logfile_name, "w+", stdout, "rop=wbh", "shr=get" , "fop=dfw")' to create a new log file the functions ftell() and fseek() work fine on that new file.

Is it possible to do what we want to do with just the original log file or must we use two log files (original plus the one with certain attributes)?

[John Gillings]

John,

Figuring out exactly what "work fine" means in this context can be complex! You don't necessarily know when/if you're seeing the latest data in your file, as it can be in several buffers upstream from the disk.

It all depends on the sharing options you specify on the file for both reader and writer, how the file is flushed, and how the EOF is updated. For some combinations, the EOF is NEVER updated until the file is closed by the writer. Bottom line is, if the default SYS$OUTPUT doesn't give you the right sharing options to do what you want, you'll need to open it yourself.

I do something that sounds much like what you're doing. A detached process which tracks the log files of several other detached processes. I scan the contents for interesting looking strings, and, for some files, send every record to a remote system - kind of file/record level "shadowing". We can do monitoring, analysis and display on the remote system without interfering with production processing.

From the READER, I open the files with:

LogFAB: $FAB FAC=GET SHR= XAB=LogFHC NAM=LogNAM FOP=NAM

Note the "SHR=", I'm using RFAs to keep track of the high water mark, and I OPEN and CLOSE the file on each scan (I'm typically tracking several hundred files, so it's not reasonable to keep all the files open all the time). I'm not sure if it's even possible to have RMS update the EOF of a file open for read access, depending on updates from another process, so the OPEN and CLOSE in the READER may be the key to fix your issue?

Since I don't have source for any of my writer processes, I'm not entirely sure how the files are opened. Since you have control of the source of the writer, you're in better shape.

A few things to test... Does TYPE display the contents of the file? What does this do:

$ OPEN/APPEND/SHARE=WRITE LOG your-log-file
$ CLOSE LOG
$ TYPE your-log-file

(the request to append the file forces RMS to update the EOF).


FWIW, here are some stats from one process, monitoring 130 files, scan interval is 0.25 seconds, with a maximum block of 200 records from a single scan. That means the downstream side will see worst case lag of about 30 seconds for an update in a log file.

ELAPSED: 1 13:53:56.26 CPU: 0:06:10.86 BUFIO: 8346195 DIRIO: 761163 FAULTS: 228

RMS stats: $PARSE:32364, $SEARCH:32364, $OPEN:706328, $CONNECT:706328, $FIND:697914, $GET:11128952

So, don't be scared off by the cost of the OPEN. 4 seconds CPU per day is a tiny price to pay (and that includes a whole lot of pattern matching and message dispatches)

Of course, this would be significantly reduced if RMS had a (supported and documented) mechanism to tell me when a particular file has been updated, so I don't have to poll!

Since you have control of your writers... you could take a whole different approach to the general problem. Instead of writing your log files to disk, send SYS$OUTPUT to a "logfile daemon" process, possibly through a mailbox. Have your logging routine on the writer side add an ID (& timestamp?) to each message. What you do with it once the daemon has it is up to you.

[John McL]

Thanks John, I'll do some investigations as well as wait for further comments.

Just one thing at the moment because it might be relevant to those comments ... we have the line "fsync(fileno(stdout));" just before the freopen() call, so I believe that all data is flushed to disk before we attempt to access it.

[John McL]

I got a chance to test it sooner than expected.

"$ TYPE logfile" works fine.

"$ OPEN/APPEND/SHARE=WRITE LOG logfile" fails with message "RMS-E-FLK, file currently locked by another user" (as does any attempt to edit the file, even /READONLY)

When trying to open a new channel to the file with "r" (read) and "shr=get,put", the open works okay, but then the ftell on stdout fails.

I'm tempted to try opening the file "a+" to test if ftell will work on the new channel, but I'll wait for further responses in case they say "Won't work".

[John Gillings]

John,

>"$ TYPE logfile" works fine.

Aha! I think that's your clue. Don't be too concerned about the writer.

I'm fairly sure your reader process won't ever see an EOF beyond the point it was at when the file was opened, regardless of what you do in the writer. The only way to see an updated EOF is for the reader to close and reopen the file.

[John McL]

John,

As I understand it, the ftell() function should return the byte offset from the start of the file. The value EOF (which I think is -1) is returned when any error occurs and is not an indication that the file is at EOF. Presumably if the file was positioned at the start the returned value would be zero, so any error flags do need to be negative.

Unfortunately it seems typical of C RTL function that different types of errors are not separately flagged, so I'll just have to experiment with various alternatives.

[WW304289]

> Unfortunately it seems typical of C RTL function that different types of errors are not separately flagged [...]

The C RTL functions differentiate the errors using errno.

-Boris

x.c
---
#include

int main() {
if ( ftell(stdout) == -1 )
perror("ftell");
}

$ run x
ftell: illegal seek
$

[John McL]

WW304289, thank you but it just demonstrates my point.

I'd like something like
- SS$_NORMAL (or equivalent)
- Warning - position is past EOF
- Warning - position reset to start of current record
- Error - attempt to access before start of file
- Error - operation not permitted on this file type

etc.

[John McL]

Progress report. (Earlier attempt when blank when I hit preview so I guess it's lost.)

I've managed to extract the log file records by the following method.

(a) Set-up
1 - Open new channel (for Read) on log file, positioning at EOF
2 - use ftell() and save offset for EOF
3 - close channel

(b) to extract
4 - Open new channel (for Read) on Log file
5 - use fseek() to set position saved in step 2
6 - Read forward to EOF (i.e. get extract)

I'd rather open the second channel just once, but it seems the EOF is recorded when I open the file for reading and later I can't read any new records written past that old EOF point.

John G, does RMS give me the freedom to control (or worki around) this? I'd prefer to use RFA's to position the file pointer anyway, but then how do I get the RFA of the initial EOF point?

[John Gillings · Accepted solution]

John,

As much as we would like to reduce the number of opens, I don't think it's possible.

I remember going into this for a customer case some years ago. Ultimately the answer was, "you need to reopen the file to get the new EOF". I couldn't find anything, even undocumented/unsupported, to get around that. See source code for TYPE/TAIL/CONTINUOUS as an example.

From my stats, opens aren't the performance bogeyman that one might expect, at least for this type of application.

To find the RFA of the last record before EOF you either need to scan the file with $FIND, or open the file with ROP=EOF and try to work your way backwards (see source of TYPE/TAIL for some heuristics in locating records backwards... it ain't pretty!).

Given the position of the EOF, I believe you could fabricate an RFA for a record written after that position. A few experiments should reveal the secrets. Obviously not supported, but also not something engineering would dare change!

(My log file tracker always reads from the beginning of the file. Even when recovering, I just read to EOF and keep track of the RFA. Even though I'm following numerous files, some of which are quite large, timing is fast enough that I didn't think it worth the effort to try to do anything more heroic)