ZIP and subsequent UNZIP of COM file produces Stream-LF file.

[Dan Herron]

We have a command procedure FMSSTRTUP.COM that when we Zip (v2.32 of June 19th 2006) it and then UNZIP (v5.0 of 21 August 1992) it, the resulting extracted file is a Stream-LF record format file instead of variable length record format file. Which means LSE (and other editors) barf when we try to edit the extracted file. We were under the assumption the -V switch would cause the record format file attributes to be retained.

What do we need to do to end up with an extract file that is a variable length record format file?

Here is the zip and unzip commands we use.
zip "-V" test.zip FMSSTRTUP_ALPHA.COM
unzip "-V" TEST.ZIP

Original File Attributes
FMSSTRTUP_ALPHA.COM;98 File ID: (5776,38697,0)
Size: 48/70 Owner: [1,1]
Created: 2-NOV-2005 10:52:16.65
Revised: 6-MAR-2007 16:08:13.12 (2)
Expires:
Backup:
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 70, Extend: 0, Global buffer count: 0, No version limit
Record format: Variable length, maximum 0 bytes, longest 134 bytes
Record attributes: Carriage return carriage control
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RE, World:
Access Cntrl List: None
Client attributes: None

Total of 1 file, 48/70 blocks.

Extracted file attributes:
NEW.COM;1 File ID: (16427,238,0)
Size: 48/70 Owner: [1,1]
Created: 2-NOV-2005 10:52:16.00
Revised: 2-NOV-2005 10:52:16.00 (0)
Expires:
Backup:
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 70, Extend: 0, Global buffer count: 0, No version limit
Record format: Stream_LF, maximum 0 bytes, longest 0 bytes
Record attributes: Carriage return carriage control
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RE, World:
Access Cntrl List: None
Client attributes: None

Thanks for your help,
Dan Herron

[Hoff · Accepted solution]

zip 2.32 looks recent enough, but unzip v5.0 of 21 August 1992? That's really rather old.

Try unzip 5.52, dating from last year or so.

You can get pre-built core tools from here:

http://h71000.www7.hp.com/freeware/freeware80/000tools/

or in the unzip directory on the same Freeware.

http://h71000.www7.hp.com/freeware/freeware80/unzip/

Try the unzip again with 5.52, and call us back. :-)

[Hein van den Heuvel]

First of all I consider LSE and TPU broken
in their non-handling of standard OpenVMS files like Stream_lf and fxied length.
They ought to be fixed, but will not be.
It's just silly to have RMS be use be all tools, have RMS be ready for all file formats, and then have some tools get in the way because 'they know better'. Oh well.

Second, a workaround to get back to variable length is trivial.
Just in case you did not know:

$CONVERT/FDL=NL:

Yeah I seem to recall, -V is supposed to preserve attributes best I recall, but Steven Schweda will no doubt inform us.

Finally, 1992 is way back. Things have changed since. Seems to me you would want to google around for a more recent (un)zip, and you might run into simialr problem with explanations along the way.

Cheers,
Hein van den Heuvel
HvdH Performance Consulting.

[Steven Schweda]

"UnZip 5.52 of 28 February 2005, by Info-ZIP"
is the current released version. UnZip 5.0
is before my time ("1992"), and might do
almost anything, for all I know. My advice
would be to try 5.52, which seems to work for
me:

alp $ dire /full

Directory ALP$DKA0:[SMS.ZIP_TEST]

STMLF.TXT;1 File ID: (121343,2871,0)
[...]
Record format: Stream_LF, maximum 0 bytes, longest 37 bytes
Record attributes: Carriage return carriage control
[...]

VAR.TXT;1 File ID: (135141,4065,0)
[...]
Record format: Variable length, maximum 0 bytes, longest 37 bytes
Record attributes: Carriage return carriage control
[...]

VFC.TXT;1 File ID: (156160,1017,0)
[...]
Record format: VFC, 2 byte header, maximum 0 bytes, longest 37 bytes
Record attributes: Print file carriage control
[...]

alp $ zip -V test.zip *.*
adding: STMLF.TXT (deflated 29%)
adding: VAR.TXT (deflated 30%)
adding: VFC.TXT (deflated 29%)

alp $ set defa [.unz]
alp $ crea /dire []

alp $ unzip [-]test.zip
Archive: ALP$DKA0:[SMS.ZIP_TEST]TEST.ZIP;1
inflating: stmlf.txt
inflating: var.txt
inflating: vfc.txt

alp $ dire /full

Directory ALP$DKA0:[SMS.ZIP_TEST]

STMLF.TXT;1 File ID: (121343,2871,0)
[...]
Record format: Stream_LF, maximum 0 bytes, longest 37 bytes
Record attributes: Carriage return carriage control
[...]

VAR.TXT;1 File ID: (135141,4065,0)
[...]
Record format: Variable length, maximum 0 bytes, longest 37 bytes
Record attributes: Carriage return carriage control
[...]

VFC.TXT;1 File ID: (156160,1017,0)
[...]
Record format: VFC, 2 byte header, maximum 0 bytes, longest 37 bytes
Record attributes: Print file carriage control
[...]


Note that UnZip's "-V" deals with version
numbers, not VMS/RMS file attributes. (See
also Zip's "-w".)

If you can break these, please let me know:

alp $ zip -v
Copyright (c) 1990-2006 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 2.32 (June 19th 2006), by Info-ZIP.
[...]

alp $ unzip -v
UnZip 5.52 of 28 February 2005, by Info-ZIP. Maintained by C. Spieler. Send
[...]

[Steven Schweda]

> [...] the -V switch would cause the record
> format file attributes to be retained.

It gets them saved in the archive, but it's
up to UnZip to do something with them.

> [...] Steven Schweda will no doubt inform us.

Now I have a reputation. Sigh.

Note also, that with SET PROCESS
/PARSE_STYLE = EXTENDED, you don't (and I
didn't) need to quote the non-lower-case
items on the (modern) [Un]Zip command line,
which is why my Zip command above worked as
intended. When in doubt, quoting is always
safe, but I've trained myself pretty well now
not to bother.

[John Gillings]

Dan,

IMHO the ZIP "-V" switch is a myth! In all the years I've been transferring files to and from OpenVMS systems, I have never even once seen -V actually work! No one has ever been able to give me a simple, step by step example to show how to use it correctly(sorry, MAN page style documentation doesn't count, I want to see a fully worked and proven example!).

If you want to preserve OpenVMS attributes, the most reliable means I know of is to first BACKUP your files into a saveset, then ZIP the saveset. Although the UNZIP at the other end will break the attributes of the saveset, they can always be corrected either using Hein's FIXSAVESET.COM procedure, or, on V8.3 BACKUP/REPAIR.

You can then unpack the saveset with all OpenVMS attributes intact.

[John Gillings]

Looks like Steven has posted the transcript. So I'll have to accept that it is possible to make it work!

On the other hand, if you want to make absolutely certain that no intermediary systems or utilities drop your attributes and waste your transfer bandwidth, or you can't be sure of a correct UNZIP version at the other end, use the BACKUP trick.

[Steven Schweda]

> Looks like Steven has posted the
> transcript. [...]

Yes, but that moron pasted in the same
DIRECTORY output data twice, which might
reduce one's confidence in his claim. On the
other hand, looking at the files actually
created by UnZip (this time), it really does
appear to work:

alp $ dire /full

Directory ALP$DKA0:[SMS.ZIP_TEST.UNZ]

STMLF.TXT;1 File ID: (159782,449,0)
[...]
Record format: Stream_LF, maximum 0 bytes, longest 37 bytes
Record attributes: Carriage return carriage control
[...]

VAR.TXT;1 File ID: (163936,273,0)
[...]
Record format: Variable length, maximum 0 bytes, longest 37 bytes
Record attributes: Carriage return carriage control
[...]

VFC.TXT;1 File ID: (168479,1879,0)
[...]
Record format: VFC, 2 byte header, maximum 0 bytes, longest 37 bytes
Record attributes: Print file carriage control
[...]

(Even the file IDs are different this time.)

> On the other hand, if you want to make
> absolutely certain [...]

Absolute certainty may be more difficult to
attain, but some folks seem to tolerate a
non-BACKUP solution in some cases. For
example, VMS patch kits:

ALP $ exec VMS83A_TZ-V0100.ZIPEXE -v
UnZipSFX 5.42 of 14 January 2001, by Info-ZIP (Zip-Bugs@lists.wku.edu).
Valid options are -tfupcz; modifiers are -abjnoqCLMVX. Quote uppercase options.

Now, why they're still using that particular
old, slow UnZip version is another question,
the answer to which must be sought elsewhere.

[Jon Pinkley]

Dan,

First, thanks for a good problem description.

As others have mentioned, you should update unzip. I can't remember when the file attribute save/restore was added, but it is very possible that the version you have doesn't.

Some other zip/unzip trivia:

"-V" means different things to zip and unzip.

To get descriptions of the options, enter zip and unzip with no arguments.

Zip interprets "-V" as "save vms file attributes". Unzip will use them if they were saved.

Unzip interprets "-V" as "restore original vms file version numbers". Normally these are stripped, and VMS creates a new file with the version set to one higher than the highest existing version.

To make unzip create "standard" VMS text files, use unzip -a (or -aa to force it).

My guess is that if you use

unzip -a TEST.ZIP

it is possible the version 5.0 of unzip will do what you want, at least for standard text files.

Concerning BACKUP vs. zip "-V". Backup is the "reference standard". I know that zip works for nearly everything. Security info is retained, but unless unzip "-X" is used, the behavior is similar to a backup/interchange, i.e. security related stuff doesn't get preserved, but the data does.

As a test I zipped an indexed file with 18 keys in 36 areas, allocation > (size + 1 cluster), unzipped to different location, and did a backup/compare with zero differences found. (That's a block for block compare, which is more significant than differences, which compares at the record level). The output file was not truncated.

Tests done with following versions on VMS Alpha 7.2-2

$ zip -v
Copyright (C) 1990-1997 Mark Adler, Richard B. Wales, Jean-loup Gailly,
Onno van der Linden, Christian Spieler and Igor Mandrichenko.
Type 'zip "-L"' for software license.
This is Zip 2.2 (November 3rd 1997), by Info-ZIP.
Currently maintained by Onno van der Linden. Please send bug reports to
the authors at Zip-Bugs@lists.wku.edu; see README for details.

Latest sources and executables are at ftp://ftp.cdrom.com/pub/infozip, as of
above date; see http://www.cdrom.com/pub/infozip/Zip.html for other sites

Compiled with DEC C V5.5-002 for OpenVMS (V7.1 for Alpha) on May 13 1998.

Zip special compilation options:
USE_EF_UT_TIME
VMS_PK_EXTRA
[encryption, version 2.7 of 22 April 1997]

Zip environment options:
ZIP_OPTS: [none]
ZIPOPT: [none]
$

$ unzip -v
UnZip 5.40 of 28 November 1998, by Info-ZIP. Maintained by C. Spieler. Send
bug reports to the authors at Zip-Bugs@lists.wku.edu; see README for details.

Latest sources and executables are at ftp://ftp.cdrom.com/pub/infozip/ , as of
above date; see http://www.cdrom.com/pub/infozip/UnZip.html for other sites.

Compiled with DEC C V5.5-002 for OpenVMS (K7.2 for Alpha) on Dec 15 1998.

UnZip special compilation options:
COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
LZW_CLEAN (PKZIP/Zip 1.x unshrinking method not supported)
TIMESTAMP
[decryption, version 2.8 of 15 November 1998]

UnZip and ZipInfo environment options:
UNZIP_OPTS: [none]
UNZIPOPT: [none]
ZIPINFO_OPTS: [none]
ZIPINFOOPT: [none]
$



In other words, it appears to me that:

$ zip "-V" save.zip infile;
$ unzip "-VX" save.zip

Is very similar to

$ backup/block=0 infile; save.bck/save
$ backup save.bck/save []*.*;*/own=original

and

$ zip "-V" save.zip infile;
$ unzip save.zip

is very similar to

$ backup/block=0 infile; save.bck/save
$ backup/interchange save.bck/save [];0

Of course the size of the .zip files will normally be a lot smaller than the .bck files.

If you are zipping files from another directory, consider using -j (junk directories). I prefer to set my default directory to the top of the source tree, and specify a different location for the zip file. This avoids the need for -j and also allows you to recreate a tree with all the subdirectories.

i.e.

$ set default sys$common:[sysmgr]
$ zip "-V" sys$scratch:sysmgr.zip [...]*.com;

instead of

$ set def sys$scratch
$ zip "-V" sysmgr.zip sys$common:[sysmgr...]*.com;

[Jon Pinkley]

Update:

unzip "-V" only restores version numbers if they were saved with zip -w.

So

$ zip "-Vw" save.zip infile; ! w is required
$ unzip "-VX" save.zip ! to preserve version

Is very similar to

$ backup/block=0 infile; save.bck/save
$ backup save.bck/save []*.*;*/own=original