topic Re: Error when appending files on VMS in Operating System - OpenVMS

Error when appending files on VMS

RobCox — Tue, 06 Jul 2010 15:15:50 GMT

I have encountered an issue when appending some files as follows:

$ append/log/new apps:*gsm3*.dat t.t

This works fine. But if I add dates such as

$ append/log/new apps:*gsm3*.dat/sin="02-jul-2010" t.t

I get the following error:

%APPEND-W-NOTCMPLT, APPS:*GSM3*.DAT; not completely copied

I don't understand why adding a date range would cause an issue with the file 'not completely copied'
Any ideas would be appreciated.

Thanks.

Re: Error when appending files on VMS

RBrown_1 — Tue, 06 Jul 2010 15:51:27 GMT

I don't see this behaviour with VMS 7.1 on Alpha, but here is a guess.

Does the /SINCE exclude any files of GSM3*.DAT? Do you still get the same message if the /SINCE *DOES NOT* exclude any files?

Re: Error when appending files on VMS

Hein van den Heuvel — Tue, 06 Jul 2010 16:14:40 GMT

Rob,

Are you not getting much more error messages?
Perhaps the specific file, and for example an RMS-F-RSZ message?

By using the /NEW you are trusting the FIRST file selected to define the characteristics for the output. That causes uncontrolled, random behaviour.

The first, old file may have say 80 byte records.
The next, more recent file file might have a small maximum record size, say 50.
And the next also recent file might have records of 60 bytes.

Without the /SINCE, the MRS for the output will be 80 (or more) and all other files will happily play along.

With the /SINCE the MRS might be that 50 bytes, and the 3rd file would fail to deliver its longed records: Partially copied.

My preferred solution would be to control the randomness by using

$ CREATE t.t
$ append/log/NOnew/since apps:*gsm3*.dat t.t

Cheers,
Hein

$ copy tmp.exe tmp.tmp
$ app/log %.tmp tmp.tmp
%APPEND-W-INCOMPAT, C.TMP;1 (input) and TMP.TMP;1 (output) have in
compatible attributes
%APPEND-E-WRITEERR, error writing TMP.TMP;1
-RMS-F-RSZ, invalid record size
%APPEND-W-NOTCMPLT, C.TMP;1 not completely copied
%APPEND-W-NOTCMPLT, %.TMP; not completely copied

Re: Error when appending files on VMS

RobCox — Tue, 06 Jul 2010 16:34:21 GMT

RBrown,
you are correct. The problem does not manifest itself on the Alpha and we have V 8.3 I have just confirmed that.
However we have just commissioned some new Integrity servers and this is where the problem is occurring. I can't see why the same script will error out on one server but not the other, when processing the same files.

I believe the files may have different attributes but are all sequential text files.

Hein,
this is the only error message displayed - no further information. I just tried your suggestion of $create t.t and got the same error.

Re: Error when appending files on VMS

Hein van den Heuvel — Tue, 06 Jul 2010 17:22:02 GMT

It is difficult to see you would get only the final message.
What about the /LOG data lines?

It this a typed command, or a spawn or scripts?

What about the resulting file?
Did it copy all records or partial as warned?
What's missing?

What files are selected, which not? (again /LOG)
If you change the /SINCE to /SINCE=1-jan-1900, that is to include all files, then does it work again?

I'd be tempted to grab CMKRNL privs and add a SET WATCH FILE/CLASS=MAJOR just before the APPEND. Check the read and write stats per file. IF you decide to share that output, then please append as a text file, and toss in some $ DIRECTORY /DATE=(CRE, MOD) output for the relevant files.

( to stop SET WATC FILE/CLASS=NONE )

btw, my example failure was produced on OpenVMS Alpha 8.3.

hth,
Hein.

Re: Error when appending files on VMS

RobCox — Tue, 06 Jul 2010 19:47:58 GMT

Ok please find the full text below:

Node MNID03 is an Integrity server
MNID03 >set ver
MNID03 >@t
$ set def traf_anal_extract:[000000]
$ append/log/new application_traf_anal_data:*gsm3*.dat/sin="02-jul-2010" -
/before="03-jul-2010" t.t
%APPEND-W-NOTCMPLT, APP_TRAF_DAT:[TRAF_ANAL_DATA1]*GSM3*.DAT; not completely copied
$ exit
MNID03 >

Here's the same script, executed on the Alpha
MNID01
MNID01 >@t
$ set def traf_anal_extract:[000000]
$ append/log/new application_traf_anal_data:*gsm3*.dat/sin="02-jul-2010" -
/before="03-jul-2010" t.t
%APPEND-W-INCOMPAT, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123089.DAT;1 (input) and TRAF_ANAL_EXTRACT:[000000]T.T;3 (out
put) have incompatible attributes
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123089.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (615
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123090.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (661
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123091.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (714
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123092.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (732
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123093.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (627
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123094.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (519
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123095.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (412
records)
%APPEND-S-APPENDED, APP_TRAF_DAT:[TRAF_ANAL_DATA1]MST_TRAF_ANAL_GSM3_00123096.DAT;1 appended to TRAF_ANAL_EXTRACT:[000000]T.T;3 (60
records)
$ exit
MNID01 >

My issue is, why does it work on the Alpha, but not the Integrity?

Re: Error when appending files on VMS

Hein van den Heuvel — Tue, 06 Jul 2010 19:58:57 GMT

And that's a cluster, so you are looking at the very same files huh?

I'm reluctantly starting to think this may be an interesting problem after all. I'll give it a second whirl in my copious spare time (later, ,much later)

You may want to move forward with SET WATCH FILE.
In you place I would also want to know what the reported attribute mismatch is.
Maybe the Itanium code reacts differently (wrongly) to that.
Check with DIR/FULL on T.T; and on the first file to report a mismatch.
Also, remove T.T and copy that first file as T.T to see whether that changes anything.

Later,
Hein
NEL - URU : 3 - 2. Oranje Boven!

Re: Error when appending files on VMS

RobCox — Tue, 06 Jul 2010 20:25:46 GMT

Ok I have done a couple things here:

1) here's what a dir/full of t.t looks like
MNID01 >dir/full t.t;3

Directory TRAF_ANAL_EXTRACT:[000000]

T.T;3 File ID: (269149,2489,0)
Size: 4001/4356 Owner: [COX_R]
Created: 6-JUL-2010 13:31:02.23
Revised: 6-JUL-2010 16:45:11.40 (3)
Expires:
Backup:
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 4356, Extend: 0, Global buffer count: 0, No version limit
Record format: Variable length, maximum 0 bytes, longest 233 bytes
Record attributes: Carriage return carriage control
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RE, World:
Access Cntrl List: None
Client attributes: None

Total of 1 file, 4001/4356 blocks.
MNID01 >

Secondly, I did as you suggested and copied the first file as t.t. When run on the Alpha, no warnings about compatible attributes were displayed. However on the Integrity, I get the same error as before. Nothing changed there.

Thirdly, yes these machines are in a cluster. The Alpha will be phased out shortly.

Can you explain the 'set watch file' command. I've never used that.

Re: Error when appending files on VMS

Hein van den Heuvel — Tue, 06 Jul 2010 20:58:22 GMT

>> Can you explain the 'set watch file' command. I've never used that.

See my earlier reply, or GOOGLE: +"set watch" +openvms

Looks like the incompatible attributes are benin in your case. So they can be ignored. For yourwon sake I;d check the attributes on the new t.t, but is is probably stream-lf or VFC versus VAR. No biggie

Hein

Re: Error when appending files on VMS

Hoff — Tue, 06 Jul 2010 21:00:23 GMT

"OpenVMS Tips: Troubleshooting File System Access Errors"

http://labs.hoffmanlabs.com/node/217

Includes information on SET WATCH, as well as using security mechanisms.

And Google can be a good friend here, too. You'll find a variety of discussions of this topic and related topics with the search string /OpenVMS set watch/ or analogous.

Re: Error when appending files on VMS

RobCox — Tue, 06 Jul 2010 21:24:08 GMT

Ok here it is.
I have modified the script to include a set priv command and the set watch file command as well. This is attached in the text file Output.txt.

Re: Error when appending files on VMS

Jon Pinkley — Tue, 06 Jul 2010 23:00:58 GMT

Rob,

It appears that someone broke append in I64. I wonder if it is now broken in 8.4 on the Alpha? (I would expect both to use the same source code)

This test is on Alpha 8.3, and as Rob stated, it works correctly.

Suggestion, create a simple reproducer that does not rely on anything (create the files, modify the creation dates with

$ set file/attribute=(credate=1-jan) a.1 ! etc

If you can provide a simple reproducer that demonstrates the issue, you will be much more likely to get a timely fix.

See attachment for log file.

Jon

Re: Error when appending files on VMS

RobCox — Thu, 08 Jul 2010 14:53:10 GMT

Guys
I have been fiddling with this and I have observations:

The command $ append/log/new application_traf_anal_data:*gsm3*.dat/sin="02-jul-2010"/before="03-jul-2010" t.t
definitely does not work on the original files. I get the "%APPEND-W-NOTCMPLT" error.
If I remove the date parameters from the command, it works fine. But this is NOT what I want since the directory contains thousands of files outside the date range that is required.

Therefore as a work around, I have to copy the required files to the current working directory, append them there and then delete. That works ok but is a bit cumbersome. I can't change the creation dates of the original files because that is critical for selecting the files for the downstream system.

So there seems to be a problem with dates. Jon, I did as you suggested and changed the dates on some other files, and they were appended without a problem. So I'm a little confused. Is it really a problem with the append command or is there a problem with the creation dates on the files? How can I know?

Re: Error when appending files on VMS

Andy Bustamante — Thu, 08 Jul 2010 15:06:30 GMT

Just to see what happens, can you use

$copy *gsm3*.dat/sin="02-jul-2010" t.t temp.dat

Re: Error when appending files on VMS

RobCox — Thu, 08 Jul 2010 15:28:25 GMT

Andy,
the copy command works fine, no problems there. In fact this is what I used as part of my work around to get the required files in the current working directory before appending.

Re: Error when appending files on VMS

Cass Witkowski — Thu, 08 Jul 2010 15:33:16 GMT

Why do you have double quotes around the dates?

Re: Error when appending files on VMS

RobCox — Thu, 08 Jul 2010 15:55:12 GMT

Cass,
the copy command is a line extracted from a script which allows the user to insert the date range of the files to be extracted and appended before passing to the downstream system. The user can enter a date like 02-jul-2010 or 02-jul-2010 12:35, hence the quotes are needed.

Re: Error when appending files on VMS

Hein van den Heuvel — Thu, 08 Jul 2010 15:57:15 GMT

Guys
>> I have been fiddling with this and I have observations:

Me too, withing limited time available.

1) They did not 'break' append on I64 as suggest earlier. It works in general, and it works (on my itanium ) for a much similar command using my files.

2) Append is actually done by 'copy'. I check the code and the NOTCMPLT is a catch-call error AFTER an other error happened... but that other error should have been signalled, and it is not.

3) You SET WATCH log suggest trouble accessing (opening) the first file. No read are done. It is a mystery why no error was signalled. I suspect it is something very specific in your environment but for now I am at a loss as to what. Maybe the directory file is is odd?
3A) is APPS: some sour of search list or is there a bound volume set in play?
3B) create a fresh directory and rename all files there? then give the new directory the old name?
3C) analyze / disk [ /repair ]

>> Therefore as a work around, I have to copy the required files to the current working directory, append them there and then delete. That works ok but is a bit cumbersome.

So use RENAME to simplify this some, and minimize overhead on the box and minimize failure opportunities.

Use $ RENA/SINCE ... APPS:*GSM3*.DAT *.DAX
Then $ append *.DAX
And put it back $ RENA *.DAX *.DAT

I deliberately suggest to use the same main directory, and DAX as alternative for DAT to avoid/minimize the directory entry movements.

Good luck!
Hein

Re: Error when appending files on VMS

RobCox — Thu, 08 Jul 2010 16:51:47 GMT

Hein,
thanks for your input here. The logical name "APPS" is actually a search list of 4 directories on 4 disks. I don't know if it significant, but all the files identified by the date criteria are actually in the first directory of the search list.
I also like your suggestion of doing a rename rather than copying the files. I tried it and it works fine.
However, I'm still bothered as to why the original script works fine on the Alpha but throws up an error on the Integrity, when both are in the same cluster.
I'm not going to put any further time into this. I have a work around that is satisfactory for my purposes. Thanks to all the experts who contributed.

Re: Error when appending files on VMS

The Brit — Fri, 09 Jul 2010 11:30:27 GMT

Hi Rob,
Note that RENAME cannot be used when moving files between devices. (You mentioned that your files were on 4 different devices).

If all of your files are on a single device, and that is the same as your work directory, then the RENAME will work (as you seemed to indicate), however if any of the files are on other devices, then that part of the rename will fail.

(Note rename does not move any data, is simply modifies the INDEXF.SYS file (as I understand it))

Dave.