topic Re: VMS Unzip doesn't adjust for GMT in Operating System - OpenVMS

VMS Unzip doesn't adjust for GMT

Douglas Rupp — Wed, 18 Nov 2009 20:35:54 GMT

The most recent version I've tried is Unzip V5.52. While reviewing the forum for answers I saw an announcement that 6.0 is out, but nothing about a bug fix for this problem:

We package our sources files in Europe for VMS using zip on a Unix system, transfer the file, then build our application in the US. Unzipping the source files in the US doesn't adjust the file timestamps for GMT causing Gnu make to complain about some files with future timestamps, resulting in build failure.

Please fix, or advise on how to fix. This is a high priority issue for our company.

Re: VMS Unzip doesn't adjust for GMT

Steven Schweda — Wed, 18 Nov 2009 20:54:12 GMT

Which VMS version? Not all know what GMT/UTC
is. (Pre V7.0?)

Output from "unzip -v"?

No bets, but I'd guess that it all works
where the underlying C run-time support
exists.

Around here, for example:

alp $ unzip -v
UnZip 5.52 of 28 February 2005, by Info-ZIP. Maintained by C. Spieler. Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with DEC C V6.5-001 for OpenVMS (V7.3-1 Alpha) on May 18 2005.

UnZip special compilation options:
COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
TIMESTAMP
USE_EF_UT_TIME
USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
[decryption, version 2.9 of 05 May 2000]
[...]

alp $ unzip6l -v
UnZip 6.00 of 20 April 2009, by Info-ZIP. Maintained by C. Spieler. Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with DEC C V7.3-009 for OpenVMS (V7.3-2 Alpha) on Apr 29 2009.

UnZip special compilation options:
COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
SET_DIR_ATTRIB
SYMLINKS (symbolic links supported, if RTL and file system permit)
TIMESTAMP
USE_EF_UT_TIME
USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
LARGE_FILE_SUPPORT (large files over 2 GiB supported)
ZIP64_SUPPORT (archives using Zip64 for large files supported)
USE_BZIP2 (PKZIP 4.6+, using bzip2 lib version 1.0.5, 10-Dec-2007)
[decryption, version 2.11 of 05 Jan 2007]
[...]

Note the "USE_EF_UT_TIME" in both. If you
don't see that, then you may be doomed (or
have some solvable problem, depending).

> [...] using zip on a Unix system [...]

Output fron "zip -v" there might be
informative, too. Some fossil version might
not be storing the "UT" extra fields. (Look
for "USE_EF_UT_TIME" there, too.) As I
recall, the original Zip archive format
stored (DOS) local time. More than that is a
bonus, and, while very commonly available,
it's not universal.

Re: VMS Unzip doesn't adjust for GMT

Douglas Rupp — Wed, 18 Nov 2009 21:00:30 GMT

This is on I64 VMS Version 8.3

Output from unzip -v is
$ unzip552 -v
UnZip 5.52 of 28 February 2005, by Info-ZIP. Maintained by C. Spieler. Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with DEC C V7.1-005 for VMS (XALQ-T3Z VAX) on Feb 28 2005.

UnZip special compilation options:
COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
TIMESTAMP
USE_EF_UT_TIME
USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
[decryption, version 2.9 of 05 May 2000]

UnZip and ZipInfo environment options:
UNZIP_OPTS: [none]
UNZIPOPT: [none]
ZIPINFO_OPTS: [none]
ZIPINFOOPT: [none]

Re: VMS Unzip doesn't adjust for GMT

Hoff — Wed, 18 Nov 2009 23:09:04 GMT

There's the brute-force is an option:

$ zn = "zipname.zip"
$ dt =f$file(zn,"CDT")
$ dt = f$element(0," ",dt) + ":" + f$element(1," ",dt)
$ set file/attr=credate='dt'/moddate='dt' *.*;*

There's also the possibility that the system time setting on the OpenVMS box is messed up, too. That's unfortunately an easy state to get into, given the implementation. To check that, post:

SHOW LOGICAL SYS$TIMEZONE*

Also, post a unzip -v from the source Unix box.

Post up a small zip reproducer, too. Show a timestamp, zip some non-sensitive text files, and post (attach) the zip here.

Re: VMS Unzip doesn't adjust for GMT

Steven Schweda — Thu, 19 Nov 2009 00:19:20 GMT

> Also, post a unzip -v from the source Unix
> box.

"zip -v".

> Post up a small zip reproducer, too. [...]

Or for even more fun, use "unzip -Zv" to
analyze the archive. For example, using an
UnZip 6.0 kit (created in Germany, I
believe), selecting a typical file:

ALP $ unzip6l -Zv UNZIP60.ZIP unzip60/ttyio.h
Archive: ALP$DKA0:[UTILITY.SOURCE.ZIP]UNZIP60.ZIP;1
[...]

Central directory entry #42:
---------------------------

unzip60/ttyio.h

offset of local header from start of archive: 291515
(00000000000472BBh) bytes
file system or operating system of origin: MS-DOS, OS/2 or NT FAT
version of encoding software: 3.0
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 2.0
compression method: deflated
compression sub-type (deflation): maximum
file security status: not encrypted
extended local header: no
file last modified on (DOS date/time): 2004 Sep 9 00:03:46
file last modified on (UT extra field modtime): 2004 Sep 8 17:03:46 local
file last modified on (UT extra field modtime): 2004 Sep 8 22:03:46 UTC
32-bit CRC value (hex): c6a5f5e0
compressed size: 1789 bytes
uncompressed size: 5348 bytes
length of filename: 15 characters
length of extra field: 9 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: text
non-MSDOS external file attributes: 000000 hex
MS-DOS file attributes (20 hex): arc

The central-directory extra field contains:
- A subfield with ID 0x5455 (universal time) and 5 data bytes.
The local extra field has UTC/GMT modification/access/creation times.
[...]

That "file last modified" stuff tells the
story. Apparently, the local time was
UTC+2:00. I'll admit that I don't understand
why it gets a five-hour difference between
UTC and local, when I should be on standard
time in CST6CDT, so UTC-6:00. Whatever the
cause, it seems not to be VMS-specific, as I
see the same things on an HP_UX system (where
"ls -l" is less useful for revealing the time
on a file that old.) In any case, that file
was restored on my VMS system with date-time
"8-SEP-2004 17:03:46.00".

Re: VMS Unzip doesn't adjust for GMT

Steven Schweda — Thu, 19 Nov 2009 02:33:25 GMT

> [...] I'll admit that I don't understand
> why [...]

Of course, that was a summer file, so I was
in CDT at its date-time. It's a six-hour
difference for a winter file:

ALP $ unzip6l -Zv UNZIP60.ZIP unzip60/unzip.h
Archive: ALP$DKA0:[UTILITY.SOURCE.ZIP]UNZIP60.ZIP;1
[...]

Central directory entry #47:
---------------------------

unzip60/unzip.h

offset of local header from start of archive: 322232
(000000000004EAB8h) bytes
file system or operating system of origin: MS-DOS, OS/2 or NT FAT
version of encoding software: 3.1
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 2.0
compression method: deflated
compression sub-type (deflation): maximum
file security status: not encrypted
extended local header: no
file last modified on (DOS date/time): 2009 Feb 15 19:12:54
file last modified on (UT extra field modtime): 2009 Feb 15 12:12:54 local
file last modified on (UT extra field modtime): 2009 Feb 15 18:12:54 UTC
32-bit CRC value (hex): 36eea94f
compressed size: 8010 bytes
uncompressed size: 26144 bytes
length of filename: 15 characters
length of extra field: 9 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: text
non-MSDOS external file attributes: 000000 hex
MS-DOS file attributes (20 hex): arc

The central-directory extra field contains:
- A subfield with ID 0x5455 (universal time) and 5 data bytes.
The local extra field has UTC/GMT modification/access/creation times.
[...]

Hence:

ALP $ dire /date unzip.h

Directory ALP$DKA0:[UTILITY.SOURCE.ZIP.UNZIP60]

UNZIP.H;1 15-FEB-2009 12:12:54.00

Total of 1 file.

But my point remains that if the
(unspecified) Zip program on the
(unspecified) "Unix system" stores only the
(local) "DOS date/time" and not the UT extra
fields, then UnZip on the VMS system won't
have the info it needs to do the job
correctly.

Any reasonably recent Info-ZIP Zip program on
a UNIX(-like) system should be built with
USE_EF_UT_TIME. For example:

dyi # /usr/local/bin/zip232 -v
Copyright (c) 1990-2006 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 2.32 (June 19th 2006), by Info-ZIP.
[...]
Compiled with HP C version A.06.12 for Unix (HP-UX) on Jan 24 2008.

Zip special compilation options:
USE_EF_UT_TIME
[encryption, version 2.91 of 31 May 2006]
[...]

(Any Zip before 2.31 is before my time, hence
not my fault.)

> Compiled with DEC C V7.1-005 for VMS (XALQ-T3Z VAX) on Feb 28 2005.

That's a little scary, but that "VAX" should
be "IA64" in UnZip 6.0. Just one of the many
reasons to use the newer version.

Re: VMS Unzip doesn't adjust for GMT

Douglas Rupp — Thu, 19 Nov 2009 10:34:09 GMT

This is the output of "zip -v" for the Unix program that zipped the offending file:

Copyright (c) 1990-2006 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 2.32 (June 19th 2006), by Info-ZIP.
Currently maintained by Onno van der Linden. Please send bug reports to
the authors using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.

Compiled with gcc 3.4.6 for Unix (Sun SPARC/Solaris) on Aug 29 2007.

Zip special compilation options:
USE_EF_UT_TIME
[encryption, version 2.91 of 31 May 2006]
Encryption notice:
The encryption code of this program is not copyrighted and is
put in the public domain. It was originally written in Europe
and, to the best of our knowledge, can be freely distributed
in both source and object forms from any country, including
the USA under License Exception TSU of the U.S. Export
Administration Regulations (section 740.13(e)) of 6 June 2002.

Zip environment options:
ZIP: [none]
ZIPOPT: [none]

Re: VMS Unzip doesn't adjust for GMT

Steven Schweda — Thu, 19 Nov 2009 11:05:23 GMT

> This is the output of "zip -v" [...]
> [...]
> Zip special compilation options:
> USE_EF_UT_TIME
> [...]

It's not 3.0, but it should be good enough.

It'd still be subject to user intervention
(interference), by, say, "-X" ("eXclude
eXtra file attributes") on the Zip command
line, or some other too-clever operation
whose details lie beyond the limits of my
imagination. Need to see a test archive
and/or some "unzip -Zv" output to see if the
"UT" extra fields are in there or not (and
what's in them). The mystery is getting more
subtle.

Re: VMS Unzip doesn't adjust for GMT

Douglas Rupp — Fri, 20 Nov 2009 04:41:08 GMT

This looks like its missing UT fields, what are they supposed to look like?

$ unzip552 "-Zv" foo.zip "gcc-41-vms/gcc/config/ia64/ia64.h"
Archive: FOO.ZIP;1 35222371 bytes 4989 files

End-of-central-directory record:
-------------------------------

Actual offset of end-of-central-dir record: 35222349 (0219734Dh)
Expected offset of end-of-central-dir record: 35222349 (0219734Dh)
(based on the length of the central directory and its expected offset)

This zipfile constitutes the sole disk of a single-part archive; its
central directory contains 4989 entries. The central directory is 396529
(00060CF1h) bytes long, and its (expected) offset in bytes from the
beginning of the zipfile is 34825820 (0213665Ch).

There is no zipfile comment.

Central directory entry #664:
---------------------------

gcc-41-vms/gcc/config/ia64/ia64.h

offset of local header from start of archive: 2404705 (0024B161h) bytes
file system or operating system of origin: Unix
version of encoding software: 2.3
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 2.0
compression method: deflated
compression sub-type (deflation): normal
file security status: not encrypted
extended local header: no
file last modified on (DOS date/time): 2009 Nov 19 03:18:26
32-bit CRC value (hex): a7ef4909
compressed size: 23398 bytes
uncompressed size: 80465 bytes
length of filename: 33 characters
length of extra field: 0 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: text
Unix file attributes (100664 octal): -rw-rw-r--
MS-DOS file attributes (00 hex): none

There is no file comment.

Re: VMS Unzip doesn't adjust for GMT

Steven Schweda — Fri, 20 Nov 2009 05:02:22 GMT

> This looks like its missing UT fields,

Yup. (And that's "it's".)

> what are they supposed to look like?

As shown above, in "unzip -Zv" output:

[...]
file last modified on (UT extra field modtime): 2009 Feb 15 12:12:54 local
file last modified on (UT extra field modtime): 2009 Feb 15 18:12:54 UTC
[...]
length of extra field: 9 bytes
[...]
The local extra field has UTC/GMT modification/access/creation times.
[...]

> length of extra field: 0 bytes

So, in your archive, no UT extra field (or
any other). The "zip -v" report which you
provided suggests that that Zip program can
employ them ("USE_EF_UT_TIME"), so, with no
evidence to the contrary, I'm still guessing
that a "-X" option on the Zip command (or in
the ZIP or ZIPOPT environment variable, or
somewhere) was the culprit. Or the archive
was actually created using some other (older)
Zip program. (Or some other sneaky reason
which is not leaping to mind.)

Re: VMS Unzip doesn't adjust for GMT

Steven Schweda — Fri, 20 Nov 2009 05:31:06 GMT

Around here, Solaris-wise, everything I can
find employs _multiple_ UTC extra fields.
For example, here's the old Zip which Sun
supplied with the OS:

sol# /usr/bin/zip -v
Copyright (C) 1990-1999 Info-ZIP
Type 'zip "-L"' for software license.
This is Zip 2.3 (November 29th 1999), by Info-ZIP.
Currently maintained by Onno van der Linden. Please send bug reports to
the authors at Zip-Bugs@lists.wku.edu; see README for details.

Latest sources and executables are at ftp://ftp.cdrom.com/pub/infozip, as of
above date; see http://www.cdrom.com/pub/infozip/Zip.html for other sites.

Compiled with cc for Unix (Sun Sparc/Solaris) on Jan 8 2005.

Zip special compilation options:
USE_EF_UT_TIME

Zip environment options:
ZIP: [none]
ZIPOPT: [none]

sol# /usr/bin/zip a_23.zip nil.c
adding: nil.c (deflated 29%)

sol# unzip6 -Zv a_23.zip
Archive: a_23.zip
There is no zipfile comment.

End-of-central-directory record:
-------------------------------

Zip archive file size: 285 (000000000000011Dh)
Actual end-cent-dir record offset: 263 (0000000000000107h)
Expected end-cent-dir record offset: 263 (0000000000000107h)
(based on the length of the central directory and its expected offset)

This zipfile constitutes the sole disk of a single-part archive; its
central directory contains 1 entry.
The central directory is 64 (0000000000000040h) bytes long,
and its (expected) offset in bytes from the beginning of the zipfile
is 199 (00000000000000C7h).

Central directory entry #1:
---------------------------

nil.c

offset of local header from start of archive: 0
(0000000000000000h) bytes
file system or operating system of origin: Unix
version of encoding software: 2.3
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 2.0
compression method: deflated
compression sub-type (deflation): normal
file security status: not encrypted
extended local header: no
file last modified on (DOS date/time): 2009 Aug 4 22:03:26
file last modified on (UT extra field modtime): 2009 Aug 4 22:03:25 local
file last modified on (UT extra field modtime): 2009 Aug 5 03:03:25 UTC
32-bit CRC value (hex): 8ab42934
compressed size: 143 bytes
uncompressed size: 200 bytes
length of filename: 5 characters
length of extra field: 13 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: text
Unix file attributes (100644 octal): -rw-r--r--
MS-DOS file attributes (00 hex): none

The central-directory extra field contains:
- A subfield with ID 0x5455 (universal time) and 5 data bytes.
The local extra field has UTC/GMT modification/access times.
- A subfield with ID 0x7855 (Unix UID/GID (16-bit)) and 0 data bytes.

There is no file comment.

It's all essentially similar with Zip 2.32
and Zip 3.0 (and between and beyond, all of
which which I built myself).

For the record:

sol# uname -a
SunOS sol 5.10 Generic_137137-09 sun4u sparc sun4u

Re: VMS Unzip doesn't adjust for GMT

Douglas Rupp — Fri, 20 Nov 2009 19:26:34 GMT

Further inquiry shows the UT info is indeed stripped out explicitly, for uniformity purposes.

On Unix, it is possible to get the proper timestamp back by setting TZ
variable before calling unzip. However this doesn't seem to work on
VMS. Is there a comparable feature?

Re: VMS Unzip doesn't adjust for GMT

Steven Schweda — Fri, 20 Nov 2009 20:10:53 GMT

> [...]the UT info is indeed stripped out
> explicitly, for uniformity purposes.

Please define "uniformity purposes". It's
not immediately clear to me how discarding
such useful information is wise, or what harm
it could do if retained.

Relevant parable:

"Doctor, it hurts when I do this."

"Don't do that."

> On Unix, it is possible to get the proper
> timestamp back [...]

And on a UNIX(-like) system, it's possible to
get the proper timestamp with no extra effort
at all, IF YOU DON'T THROW IT AWAY.

> [...] Is there a comparable feature?

I don't immediately see one. A quick look at
the code suggests that the (local) DOS
date-time in the archive is interpreted on
VMS as a local date-time. Time zones are
considered (only) when UTC is processed. No
doubt one could modify the code to do more,
but I wouldn't expect the Info-ZIP
maintainers to include it in the main code
stream. But if a man wants to carry a cat
home by its tail, I say, "Let him."

Re: VMS Unzip doesn't adjust for GMT

Hoff — Fri, 20 Nov 2009 21:29:53 GMT

For purposes of uniformity? That's even easier to handle. Slam it all...

$ set file/attributes=(credate=1-JAN-1980:00:00 -
moddate=1-JAN-1980:00:00) [...]*.*;*

Or (better) get Mercurial (Hg), Subversion (SVN) or some other distributed change tracking and version control software into operation, and keep one set of archives around.

Re: VMS Unzip doesn't adjust for GMT

Douglas Rupp — Fri, 20 Nov 2009 21:39:11 GMT

In reply to Hoff's comment:

The "uniformity" defense (actually the word I got was "indistinguishable") needs more probing on my part. Will advise when I figure it out, but all this is going on many timezones away....

Re: VMS Unzip doesn't adjust for GMT

Steven Schweda — Fri, 20 Nov 2009 22:04:17 GMT

> [...] needs more probing on my part. [...]

Yup. See if you can find anyone who can tell
you (exactly) what goes wrong if the UTC
extra fields are not removed. (We can see
what goes wrong if they are removed.)

Re: VMS Unzip doesn't adjust for GMT

Douglas Rupp — Sat, 21 Nov 2009 02:49:26 GMT

The reason is we generate this zip file on both sides of the pond from identical sources and want it to have the same md5 sum.

The zip files don't match unless the extra timestamp info is stripped.

Re: VMS Unzip doesn't adjust for GMT

Steven Schweda — Sat, 21 Nov 2009 03:43:04 GMT

> The reason is [...]

Well, I suppose that that is a reason. Of
course, the next question would be, why do
you care? Zip archives have a built-in
checksum scheme, so it's not immediately
obvious what value an additional checksum
has, or why two different archives, even with
the same content, must be exactly identical.

Zip archives contain a local (DOS) date-time
value for each member, so I gather that
you're fooling Zip with the same TZ-setting
trick (as described above) if you're able to
create identical archives from files which
have different actual local date-time values.

All this deception is ok while it works, but
you've found a case where it breaks down.
You can fool some of the programs all of the
time, and all of the programs some of the
time, but you can't fool all of the programs
all of the time.

> Please fix, or advise on how to fix.

It's not actually broken, so I wouldn't count
on seeing a change to the official code. My
advice is to change your requirements, and
stop defeating the program features which
properly preserve file date-time values.

All the Info-ZIP source code is available,
and its license is very accommodating, so
you're welcome to alter it as you'd like,
but, as someone may once have said, "We could
do that, but it would be wrong."

Re: VMS Unzip doesn't adjust for GMT

Douglas Rupp — Sat, 21 Nov 2009 05:01:35 GMT

It doesn't seem bad practice to want a way to *externally* compare two zip files of identical sources built in two time zones.

If I understand you correctly you're saying that since there's an internal check sum, then I must trust it? Please correct me if I'm not understanding your position here.

Re: VMS Unzip doesn't adjust for GMT

Steven Schweda — Sat, 21 Nov 2009 05:51:37 GMT

> It doesn't seem bad practice [...]

Except for the price of discarding the UTC
date-time data, and faking the local
date-time data in the archive, which I find
excessive.

> If I understand you correctly you're saying
> [...]

What I thought that I said was that I didn't
see much value in yet another checksum.
You're free to trust whatever you wish. I've
found "unzip -t" to be adequate for my
purposes.

There are all kinds of ways to create an
archive which differs from another archive,
but which would yield the same extracted
files. "zip -V" on a VMS system would add
still more extra fields. Changing the order
of the archive members should do it, too.
And so on.

I can see some value in having an md5
checksum for an archive. I can't see much
value in creating an artificially shabby
archive merely to ensure that its md5
checksum matches the one for some other
artificially shabby archive.

I don't yet see the point of building a Zip
archive which is identical to some other Zip
archive which already exists. I _can_ see
the point of building a Zip archive which
includes UTC date-time data.