Error when appending files on VMS

[RobCox]

I have encountered an issue when appending some files as follows:

$ append/log/new apps:*gsm3*.dat t.t

This works fine. But if I add dates such as

$ append/log/new apps:*gsm3*.dat/sin="02-jul-2010" t.t


I get the following error:

%APPEND-W-NOTCMPLT, APPS:*GSM3*.DAT; not completely copied

I don't understand why adding a date range would cause an issue with the file 'not completely copied'
Any ideas would be appreciated.

Thanks.

[RBrown_1]

I don't see this behaviour with VMS 7.1 on Alpha, but here is a guess.

Does the /SINCE exclude any files of GSM3*.DAT? Do you still get the same message if the /SINCE *DOES NOT* exclude any files?

[Hein van den Heuvel]

Rob,

Are you not getting much more error messages?
Perhaps the specific file, and for example an RMS-F-RSZ message?

By using the /NEW you are trusting the FIRST file selected to define the characteristics for the output. That causes uncontrolled, random behaviour.

The first, old file may have say 80 byte records.
The next, more recent file file might have a small maximum record size, say 50.
And the next also recent file might have records of 60 bytes.

Without the /SINCE, the MRS for the output will be 80 (or more) and all other files will happily play along.

With the /SINCE the MRS might be that 50 bytes, and the 3rd file would fail to deliver its longed records: Partially copied.

My preferred solution would be to control the randomness by using

$ CREATE t.t
$ append/log/NOnew/since apps:*gsm3*.dat t.t

Cheers,
Hein


$ copy tmp.exe tmp.tmp
$ app/log %.tmp tmp.tmp
%APPEND-W-INCOMPAT, C.TMP;1 (input) and TMP.TMP;1 (output) have in
compatible attributes
%APPEND-E-WRITEERR, error writing TMP.TMP;1
-RMS-F-RSZ, invalid record size
%APPEND-W-NOTCMPLT, C.TMP;1 not completely copied
%APPEND-W-NOTCMPLT, %.TMP; not completely copied

[RobCox]

RBrown,
you are correct. The problem does not manifest itself on the Alpha and we have V 8.3 I have just confirmed that.
However we have just commissioned some new Integrity servers and this is where the problem is occurring. I can't see why the same script will error out on one server but not the other, when processing the same files.

I believe the files may have different attributes but are all sequential text files.

Hein,
this is the only error message displayed - no further information. I just tried your suggestion of $create t.t and got the same error.

[Hein van den Heuvel]

It is difficult to see you would get only the final message.
What about the /LOG data lines?

It this a typed command, or a spawn or scripts?

What about the resulting file?
Did it copy all records or partial as warned?
What's missing?

What files are selected, which not? (again /LOG)
If you change the /SINCE to /SINCE=1-jan-1900, that is to include all files, then does it work again?

I'd be tempted to grab CMKRNL privs and add a SET WATCH FILE/CLASS=MAJOR just before the APPEND. Check the read and write stats per file. IF you decide to share that output, then please append as a text file, and toss in some $ DIRECTORY /DATE=(CRE, MOD) output for the relevant files.

( to stop SET WATC FILE/CLASS=NONE )

btw, my example failure was produced on OpenVMS Alpha 8.3.

hth,
Hein.

[RobCox]

Ok please find the full text below:

Node MNID03 is an Integrity server
MNID03 >set ver
MNID03 >@t
$ set def traf_anal_extract:[000000]
$ append/log/new application_traf_anal_data:*gsm3*.dat/sin="02-jul-2010" -
/before="03-jul-2010" t.t
%APPEND-W-NOTCMPLT, APP_TRAF_DAT:[TRAF_ANAL_DATA1]*GSM3*.DAT; not completely copied
$ exit
MNID03 >




Here's the same script, executed on the Alpha
MNID01
MNID01 >@t
$ set def traf_anal_extract:[000000]
$ append/log/new application_traf_anal_data:*gsm3*.dat/sin="02-jul-2010" -
/before="03-jul-2010" t.t
%APPEND-W-INCOMPAT, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123089.DAT;1 (input) and TRAF_ANAL_EXTRACT:[000000]T.T;3 (out
put) have incompatible attributes
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123089.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (615
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123090.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (661
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123091.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (714
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123092.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (732
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123093.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (627
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123094.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (519
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123095.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (412
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123096.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (60
records)
$ exit
MNID01 >

My issue is, why does it work on the Alpha, but not the Integrity?

[Hein van den Heuvel]

And that's a cluster, so you are looking at the very same files huh?

I'm reluctantly starting to think this may be an interesting problem after all. I'll give it a second whirl in my copious spare time (later, ,much later)

You may want to move forward with SET WATCH FILE.
In you place I would also want to know what the reported attribute mismatch is.
Maybe the Itanium code reacts differently (wrongly) to that.
Check with DIR/FULL on T.T; and on the first file to report a mismatch.
Also, remove T.T and copy that first file as T.T to see whether that changes anything.

Later,
Hein
NEL - URU : 3 - 2. Oranje Boven!

[RobCox]

Ok I have done a couple things here:

1) here's what a dir/full of t.t looks like
MNID01 >dir/full t.t;3

Directory TRAF_ANAL_EXTRACT:[000000]

T.T;3 File ID: (269149,2489,0)
Size: 4001/4356 Owner: [COX_R]
Created: 6-JUL-2010 13:31:02.23
Revised: 6-JUL-2010 16:45:11.40 (3)
Expires:
Backup:
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 4356, Extend: 0, Global buffer count: 0, No version limit
Record format: Variable length, maximum 0 bytes, longest 233 bytes
Record attributes: Carriage return carriage control
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RE, World:
Access Cntrl List: None
Client attributes: None

Total of 1 file, 4001/4356 blocks.
MNID01 >

Secondly, I did as you suggested and copied the first file as t.t. When run on the Alpha, no warnings about compatible attributes were displayed. However on the Integrity, I get the same error as before. Nothing changed there.

Thirdly, yes these machines are in a cluster. The Alpha will be phased out shortly.

Can you explain the 'set watch file' command. I've never used that.

[Hein van den Heuvel]

>> Can you explain the 'set watch file' command. I've never used that.

See my earlier reply, or GOOGLE: +"set watch" +openvms

Looks like the incompatible attributes are benin in your case. So they can be ignored. For yourwon sake I;d check the attributes on the new t.t, but is is probably stream-lf or VFC versus VAR. No biggie

Hein

[Hoff]

"OpenVMS Tips: Troubleshooting File System Access Errors"

http://labs.hoffmanlabs.com/node/217

Includes information on SET WATCH, as well as using security mechanisms.

And Google can be a good friend here, too. You'll find a variety of discussions of this topic and related topics with the search string /OpenVMS set watch/ or analogous.

[RobCox]

Ok here it is.
I have modified the script to include a set priv command and the set watch file command as well. This is attached in the text file Output.txt.

[Jon Pinkley]

Rob,

It appears that someone broke append in I64. I wonder if it is now broken in 8.4 on the Alpha? (I would expect both to use the same source code)

This test is on Alpha 8.3, and as Rob stated, it works correctly.

Suggestion, create a simple reproducer that does not rely on anything (create the files, modify the creation dates with

$ set file/attribute=(credate=1-jan) a.1 ! etc

If you can provide a simple reproducer that demonstrates the issue, you will be much more likely to get a timely fix.

See attachment for log file.

Jon

[RobCox]

Guys
I have been fiddling with this and I have observations:

The command $ append/log/new application_traf_anal_data:*gsm3*.dat/sin="02-jul-2010"/before="03-jul-2010" t.t
definitely does not work on the original files. I get the "%APPEND-W-NOTCMPLT" error.
If I remove the date parameters from the command, it works fine. But this is NOT what I want since the directory contains thousands of files outside the date range that is required.

Therefore as a work around, I have to copy the required files to the current working directory, append them there and then delete. That works ok but is a bit cumbersome. I can't change the creation dates of the original files because that is critical for selecting the files for the downstream system.

So there seems to be a problem with dates. Jon, I did as you suggested and changed the dates on some other files, and they were appended without a problem. So I'm a little confused. Is it really a problem with the append command or is there a problem with the creation dates on the files? How can I know?

[Andy Bustamante]

Just to see what happens, can you use

$copy *gsm3*.dat/sin="02-jul-2010" t.t temp.dat

[RobCox]

Andy,
the copy command works fine, no problems there. In fact this is what I used as part of my work around to get the required files in the current working directory before appending.

[Cass Witkowski]

Why do you have double quotes around the dates?

[RobCox]

Cass,
the copy command is a line extracted from a script which allows the user to insert the date range of the files to be extracted and appended before passing to the downstream system. The user can enter a date like 02-jul-2010 or 02-jul-2010 12:35, hence the quotes are needed.

[Hein van den Heuvel]

Guys
>> I have been fiddling with this and I have observations:

Me too, withing limited time available.

1) They did not 'break' append on I64 as suggest earlier. It works in general, and it works (on my itanium ) for a much similar command using my files.

2) Append is actually done by 'copy'. I check the code and the NOTCMPLT is a catch-call error AFTER an other error happened... but that other error should have been signalled, and it is not.

3) You SET WATCH log suggest trouble accessing (opening) the first file. No read are done. It is a mystery why no error was signalled. I suspect it is something very specific in your environment but for now I am at a loss as to what. Maybe the directory file is is odd?
3A) is APPS: some sour of search list or is there a bound volume set in play?
3B) create a fresh directory and rename all files there? then give the new directory the old name?
3C) analyze / disk [ /repair ]



>> Therefore as a work around, I have to copy the required files to the current working directory, append them there and then delete. That works ok but is a bit cumbersome.

So use RENAME to simplify this some, and minimize overhead on the box and minimize failure opportunities.

Use $ RENA/SINCE ... APPS:*GSM3*.DAT *.DAX
Then $ append *.DAX
And put it back $ RENA *.DAX *.DAT

I deliberately suggest to use the same main directory, and DAX as alternative for DAT to avoid/minimize the directory entry movements.

Good luck!
Hein

[RobCox]

Hein,
thanks for your input here. The logical name "APPS" is actually a search list of 4 directories on 4 disks. I don't know if it significant, but all the files identified by the date criteria are actually in the first directory of the search list.
I also like your suggestion of doing a rename rather than copying the files. I tried it and it works fine.
However, I'm still bothered as to why the original script works fine on the Alpha but throws up an error on the Integrity, when both are in the same cluster.
I'm not going to put any further time into this. I have a work around that is satisfactory for my purposes. Thanks to all the experts who contributed.

[The Brit]

Hi Rob,
Note that RENAME cannot be used when moving files between devices. (You mentioned that your files were on 4 different devices).

If all of your files are on a single device, and that is the same as your work directory, then the RENAME will work (as you seemed to indicate), however if any of the files are on other devices, then that part of the rename will fail.

(Note rename does not move any data, is simply modifies the INDEXF.SYS file (as I understand it))

Dave.

[Hein van den Heuvel · Accepted solution]

Rob,
Can you do a quick test where you define APPS just pointing to that first directory alone, not the searchlist?

Dave wrote>> Hi Rob,
Note that RENAME cannot be used when moving files between devices. (You mentioned that your files were on 4 different devices).

Good point Dave. We only learned about the searchlist after suggesting the rename.
The rename will fail if the file is not found in the first directory in the list.

Dave>> (Note rename does not move any data, is simply modifies the INDEXF.SYS file (as I understand it))

Close. Rename does even less work. It removes the old directory entry and inserts a new one... in the first directory in the searchlist if a searchlist is in play

The target file header in INDEXF.SYS is only updated if the directory backlink needs to be update.

Hein.

[RobCox]

Hein,
I redefined the logical to point to one disk only and got the same error result - "%APPEND-W-NOTCMPLT"

$ set ver
$ set def traf_anal_extract:[000000]
$ define/proc application_traf_anal_data APP_TRAF_DAT:[TRAF_ANAL_DATA1]
$ append/log/new application_traf_anal_data:*gsm3*.dat/sin="02-jul-2010" -
/before="03-jul-2010" t.t

But as I said, if I remove both the "/sin" and "/bef" switches, the append command works. Really strange.

[John McL]

Rob, I've seen other problems with wildcard lookups on search lists. IIRC it was a double listing of certain files (or maybe just the last file).

I put it down to having a wildcard context on the search list and a wildcard context on the filename. I have a vague suspicion that it might be a problem of access sequence (e.g. files in directories further along the search list having names that are alphabetically earlier than file names found further back up the search list).

I've seen the problem on Alpha under VMS v8.3 and someone reported it to HP for me.

I can't be sure that it's the same kind of problem but this path might be worth exploring.

In the meantime, have you tried the COPY command with wildcard input file names and a single output file (without ;) ?

[RobCox]

John
The copy was a good suggestion. I tried it and it worked, or so I thought. Even though the copy produced the resultant file, the very last system message indicates that 'some file' was not completely copied. It is a similar error to the Append error. See the attachment.

[John McL]

Rob,

1. How many files should have been copied in the test that you include just above? I suggest a check with $ DIR/CREATED/MODIFIED (and your time qualifiers) to get a list of files that *should* be processed.

2. Are all the files okay, with no relevant errors from ANALYZE/DISK and ANALYZE/RMS ?

3. The output from HELP/MESSAGE NOTCMPLT says

Facility: Shared by several facilities

Explanation: A copy operation began but did not finish.

User Action: Take action based on the accompanying OpenVMS RMS file system message. Use the DCL command TYPE or DUMP to verify the contents of the output file, and delete the output file before reentering the command.

So where's the accompanying message, HP?

4. I notice that the last file in your example copied just 272 records. Was this the correct size of the last file? It wasn't open by some other process at the same time was it?

5. Do you get a different result if you fully specify the time (e.g. /SINCE="1-Jul-2010 00:00"/BEFORE="2-Jul-2010 00:00" ?

There's got to be a reason for this odd behaviour ...