topic Re: Zip/Unzip Data File Size Limitations? in Operating System - OpenVMS

Zip/Unzip Data File Size Limitations?

John T. Farmer — Thu, 28 Sep 2006 17:56:28 GMT

Just put InfoZips two utilities, Zip 2.3 and Unzip 5.50 on our Alpha/OpenVMS 7.2-1 system. I can zip and unzip small test files. Then I tried a 4.2 million block file and Zip does see the file. A small test file of the same name is zipped and unzip with ease.

Is there a file size limitation on what Zip & Unzip can handle? Is there a newer version that fixes this issue? This 4mil block file is text and should compress down significantly for ftp purposes.

Thanks,

John Farmer
john dot farmer at genworth dot com

Re: Zip/Unzip Data File Size Limitations?

jaw_1 — Thu, 28 Sep 2006 18:48:59 GMT

Yes, but do note that it is the beta version, and also requires a C compiler to build. Take a look at:

ftp://ftp.info-zip.org/pub/infozip/OLD/beta/

James

Re: Zip/Unzip Data File Size Limitations?

Steven Schweda — Thu, 28 Sep 2006 19:38:16 GMT

The file size limit is imposed by the C
run-time library, and it's 2GB (except on
some old VAX versions, where it's 4GB). At
4.2M blocks, you seem to be just over the
edge.

Not that it'd help with large files, but the
current released versions are UnZip 5.52 and
Zip 2.32

http://www.info-zip.org/
http://www.info-zip.org/UnZip.html
http://www.info-zip.org/Zip.html

There are some significant benefits to using
the newer versions, compared with UnZip 5.50
and Zip 2.3. (File I/O speed, "-V" archive
portability, less pointless Zip archive file
copying, and so on.)

> ftp://ftp.info-zip.org/pub/infozip/OLD/beta/

Yes, and at VMS V7.2-1, you may just have a
C RTL new enough for large-file support.

If you need binaries because you lack a C
compiler (and you're too lazy to use one of
the TestDrive Alpha systems), let me know,
and I can probably whip up some useful object
files.

Re: Zip/Unzip Data File Size Limitations?

John T. Farmer — Fri, 29 Sep 2006 07:44:37 GMT

Sorry, typo "and Zip doesn't see the file". But you folks already figured that out.

So what I'm hearing, is the size limitation is due to the C RTL. We don't have C installed on our system, so I'm at the mercy of someone creating that for me. I'd be happy to test it if someone can build it.

"(and you're too lazy to use one of
the TestDrive Alpha systems)"
What is a TestDrive system?

Thanks,

John

Re: Zip/Unzip Data File Size Limitations?

Robert Gezelter — Fri, 29 Sep 2006 08:07:48 GMT

John,

The "Test Drive" systems are a couple of systems operated by HP that have fairly public access (obviously within some limits). The pointer to the OpenVMS Test Drive is http://www.testdrive.hp.com/os/#ovms

- Bob Gezelter, http://www.rlgsc.com

Re: Zip/Unzip Data File Size Limitations?

Steven Schweda — Fri, 29 Sep 2006 08:33:11 GMT

> What is a TestDrive system?

http://www.testdrive.hp.com/
http://www.testdrive.hp.com/current.shtml

The price is right, but you'll need to look
elsewhere if you want to test drive a VAX.

> We don't have C [...]

Try:

http://antinode.org/ftp/info-zip/unzip60c_alphal.zip
http://antinode.org/ftp/info-zip/zip30e_alphal.zip
or:
ftp://antinode.org/info-zip/unzip60c_alphal.zip
ftp://antinode.org/info-zip/zip30e_alphal.zip

And try to ignore everything else in that
neighborhood, if you know what's good for
you.

These kits were built with "HP C V7.1-015 on
OpenVMS Alpha V7.3-2", so you'll probably
need to re-link. (For details, see
[.VMS]INSTALL_VMS.TXT.) The source is not
exactly the same as the original BETA kits,
but it shouldn't be much worse. There may be
some extra crud in there, too, as those were
my development directories. The warranty on
this stuff covers even less that the normal
it's-all-your-fault disclaimer would imply.

If a re-link fails, you may need to find
someone with a comparably old VMS version to
build the stuff. (Or perhaps install a new
C compiler kit, license or no, and/or a C RTL
patch to get the required stuff, if any.
It's been a while since I've had to worry
about linking on old VMS versions, so I'm
not sure how much can go wrong.) Wake me
when it gets worse.

Re: Zip/Unzip Data File Size Limitations?

Steven Schweda — Fri, 29 Sep 2006 09:49:16 GMT

Someone needs better (more VMS-tolerant) FTP
(proxy?) software:

29-SEP-2006 08:38:30.39 User:anonymous ident:proxy@ status:07649912 CWD dir:SYS$POSIX_ROOT^:^[info-zip^]zip30e_alphal.zip

alp $ sysmsg %x07649912
(SYS$COMMON:[SYSMSG]TCPIP$MSG.EXE;1:)
%TCPIP-E-FTP_BADDIR, invalid directory

"zip30e_alphal.zip" is not part of the
directory name. (My best guess: The request
was "CWD /[info-zip]zip30e_alphal.zip", which
didn't have a chance of being right.

Ask me why I map my anonymous FTP directory
to "/ftp/" on the Web server. Go ahead, ask.

On the bright side, having an FTP server with
which lameware can't cope does foil many
break-in attempts and abuse exploits (as
well as some legitimate users).

Re: Zip/Unzip Data File Size Limitations?

Wim Van den Wyngaert — Mon, 02 Oct 2006 08:28:05 GMT

We encountered the same problem with Sybase dumps. We now split the dump into 5 pieces by doing the dump to 5 devices.
There might be a utility to split a file into smaller pieces. Or you can write it yourself in any language.

Wim

Re: Zip/Unzip Data File Size Limitations?

John T. Farmer — Wed, 11 Oct 2006 17:05:38 GMT

Do I understand correctly, that the 2gb input file limit (C RTL) still exists even on newer versions of VMS, i.e. 8.x? Not that I'll be upgrading soon, but just wonder ed. I'm dealing with some pretty large files for data warehouse conversion.

thanks,

John

Re: Zip/Unzip Data File Size Limitations?

Steven Schweda — Wed, 11 Oct 2006 17:44:52 GMT

> Do I understand correctly, that the 2gb
> input file limit (C RTL) still exists even
> on newer versions of VMS, i.e. 8.x?

Large-file (>2GB) support in the C RTL seems
to be available beginning with VMS Alpha
V7.2 or so. After that, it's available in
everything except VAX (any version).

To make use of this large-file support, an
application (say, for example, Zip or UnZip)
may need to be re-coded to use larger
variables for file sizes and byte offsets,
and it must be re-built requesting large-file
support (defining the C macro "_LARGEFILE").
For some of the fine print, see
SYS$COMMON:[DECC$LIB.REFERENCE.DECC$RTLDEF]DECC$TYPES.H

In the case of Zip and UnZip, some new
archive formats are needed to accomodate
large file size data, so there were more
changes needed than just replacing "%d" with
"%lld" in a few places.

Zip 2.x and UnZip 5.x do not and will not
offer large-file support, anywhere, anytime.

Zip 3.x and UnZip 6.x will offer large-file
support. If you can accept the current,
unreleased, BETA versions (Zip 3.0e, UnZip
6.00c), you can have it now (with no known
major problems).

In fact, on VMS, for maximal downward
compatibility, the current builders for Zip
3.x and UnZip 6.x do not enable large-file
support by default. You need to specify
"LARGE" (somehow) at build time to get it.

Re: Zip/Unzip Data File Size Limitations?

John T. Farmer — Wed, 11 Oct 2006 23:45:01 GMT

Re: Zip 3.x and Unzip 6.X

Where do I sign up??? Seriously, I'll have a look at the beta zip and unzip, however, as most of you have read, I don't have a C compiler on my OpenVMS 7.2-1, Alpha DS20E system. Anyone know of pre-compiled binaries, or something ready-to-link?

I tried the zip/unzip from info-zip, but the link procedure failed "symbol" undefined.

Is there some other similar compression utility that doesn't have the 2gb limit?

Thanks,

John

Re: Zip/Unzip Data File Size Limitations?

Thomas Ritter — Wed, 11 Oct 2006 23:52:48 GMT

John, while not directly relevant, buy a C complier. Not expensive. C can be very useful as a general system management tool. A lot of our work is converting VMS stuff to some CSV format. C is mostly used.

Re: Zip/Unzip Data File Size Limitations?

Steven Schweda — Thu, 12 Oct 2006 00:09:49 GMT

> I tried the zip/unzip from info-zip,

Which one? I really expected the kit from
http://vms.process.com/ftp/vms-freeware/fileserv/
to work. No, wait. That was the "-V"
problem. I'm starting to get lost.

> but the link procedure failed "symbol"
> undefined.

Which symbol(s)? As usual, actual error
messages would help more than vague
descriptions of error messages.

> Is there some other similar compression
> utility that doesn't have the 2gb limit?

Yes, but you'll have the same problems with
them, and they can't preserve RMS file
attributes. See for example:

http://antinode.org/dec/sw/bzip2.html
http://antinode.org/dec/sw/gzip.html

If you don't get any better offers, I'll see
if I can find a way to build the beta kits
in a VMS V7.2-1-compatible way.

Re: Zip/Unzip Data File Size Limitations?

John T. Farmer — Thu, 12 Oct 2006 00:32:02 GMT

Error received when trying to create zip with the vms link procedure (info-zip stuff).

Directory DRA0:[USERS.FARMER.ZIP.VMS]

LINK_ZIP.COM;1 15/21 25-JAN-2006 22:48:14.00
MAKE_ZIP.COM;1 22/42 20-FEB-2006 22:54:20.00

Total of 2 files, 37/63 blocks.
$ @link_zip
%DCL-I-IGNQUAL, qualifiers appearing before this item were ignored
\SYMBOL\
%DCL-I-IGNQUAL, qualifiers appearing before this item were ignored
\SYMBOL\
Cannot find any AXP object library for Zip.
You must keep all binary files of the object distribution
in the current directory !

Directory DRA0:[USERS.FARMER.ZIP.VMS]

00BINARY.VMS;1 9/21 14-FEB-1999 02:15:26.00
00README.TXT;1 12/21 15-OCT-1997 14:09:02.00
CMDLINE.C;1 63/63 14-APR-2006 22:38:06.00
CVTHELP.TPU;1 11/21 17-APR-2005 08:59:22.00
DESCRIP.MMS;1 23/42 5-JUN-2006 19:29:22.00
LINK_ZIP.COM;1 15/21 25-JAN-2006 22:48:14.00
MAKEFILE.VMS;1 16/21 14-FEB-1999 02:15:26.00
MAKE_ZIP.COM;1 22/42 20-FEB-2006 22:54:20.00
OSDEP.H;1 6/21 29-JAN-2005 01:44:02.00
VMS.C;1 51/63 18-JUN-2006 20:17:06.00
VMS.H;1 21/21 3-AUG-2005 22:58:18.00
VMSDEFS.H;1 25/42 29-JAN-2005 01:45:16.00
VMSMUNCH.C;1 30/42 15-FEB-2005 21:59:06.00
VMSMUNCH.H;1 5/21 29-JAN-2005 01:46:10.00
VMSZIP.C;1 40/42 29-JAN-2005 01:46:24.00
VMS_IM.C;1 42/42 29-JAN-2005 01:45:18.00
VMS_PK.C;1 34/42 15-JUN-2005 23:50:16.00
VMS_ZIP.RNH;1 49/63 11-APR-2006 00:05:12.00
ZIPUP.H;1 2/21 29-JAN-2005 01:46:06.00
ZIP_CLI.CLD;1 5/21 14-APR-2006 22:39:00.00
ZIP_CLI.HELP;1 41/42 14-APR-2006 22:39:00.00

Total of 21 files, 522/735 blocks.

Apparently, I don't have all the pieces. Was just trying to get a copy of 2.3 linked for my vms version, to correct some corruption issues. Would consider the 3.x zip and 6.x unzip if their are linkable kits available.

Thanks,

John

Re: Zip/Unzip Data File Size Limitations?

Steven Schweda — Thu, 12 Oct 2006 01:13:53 GMT

It sure would help to know where from you're
starting. There's a lot of "info-zip stuff"
out there.

> Was just trying to get a copy of 2.3
> linked [...]

Don't do it. Zip 2.3 is not what you want.
For anything. However, for completeness:

%DCL-I-IGNQUAL, qualifiers appearing before this item were ignored
\SYMBOL\

I'd guess that those come from some "DELETE /SYMBOL" commands. That procedure assumes
that you haven't redefined "DELETE", which
you apparently have. At least it's a "-I-",
so that shouldn't have derailed anything.

> Cannot find any AXP object library for Zip.

And it's not obvious, but you need to be in the top-level directory to use LINK_ZIP.COM,
not in the [.VMS] directory:

@ [.VMS]LINK_ZIP.COM !! Not "@ LINK_ZIP.COM".

(Same for MAKE_ZIP.COM, not that you care.)

But, back to what you should be doing...

For a start, get the Zip 2.32 kit (with
binaries) from:

http://vms.process.com/ftp/vms-freeware/fileserv/ZIP.ZIP

and see if you can link that:

SET DEFAULT [.vms-binaries]
@link.com

Note that the builders for Zip 2.3 (and UnZip
5.50) are from before my time, and the ones
for later Zip 2.x (and UnZip 5.x) have not
been changed much. Everything's different
(possibly better) in the new (unreleased)
stuff.

> Would consider the 3.x zip and 6.x unzip if
> their are linkable kits available.

I don't have anything lying around which is
likely to be linkable on VMS V7.2-1, but
I'll see what I can do (unless someone else
leaps in before I can do it).

Re: Zip/Unzip Data File Size Limitations?

Steven Schweda — Thu, 12 Oct 2006 02:17:56 GMT

I can't test the stuff, but there's at least
a small chance. Suck down these things:

http://antinode.org/ftp/info-zip/unzip60c_alphal.zip
http://antinode.org/ftp/info-zip/unzip60c_alphal_72.zip
http://antinode.org/ftp/info-zip/zip30e_alphal.zip
http://antinode.org/ftp/info-zip/zip30e_alphal_72.zip

The plain-name archives are the full BETA
kits with modern binaries in the [.ALPHAL]
directories. The "_72" archives contain
replacement [.xxxx.ALPHAL] directories with
potentially old-compatible binaries. No
bets.

The suggested procedure is to unpack the
plain-name archives, and then move the
original [.ALPHAL] directories out of the
way:

RENAME [.ZIP30E]ALPHAL.DIR [.ZIP30E]ALPHAL_orig.DIR

RENAME [.UNZIP60C]ALPHAL.DIR [.UNZIP60C]ALPHAL_orig.DIR

Then unpack the "_72" archives to provide
the alternate [.*.ALPHAL] directories.

Finally, read [.VMS]INSTALL_VMS.TXT (Zip) or
[.VMS]INSTALL.VMS (UnZip), which should
explain how to re-link stuff.

You could also try the executables before
re-linking, just in case.

Re: Zip/Unzip Data File Size Limitations?

John T. Farmer — Thu, 12 Oct 2006 10:36:03 GMT

Steven,

You are correct, I was in the [.VMS] directory. But that asside, I will download and try the zip3 and unzip6 per your links.

You are also correct, that there is so much of this stuff floating around, it is difficult to troubleshoot if you don't know for sure where you started. I believe this was the download from process.com.

I do appreciate your detailed responses and will reply back with my results (successes!).

Thanks,

John

Re: Zip/Unzip Data File Size Limitations?

John T. Farmer — Thu, 12 Oct 2006 14:39:30 GMT

Steven,

I followed your suggestions, downloaded the zip30e and unzip60c, and the _72 zips. Tried running the zip.exe program from the [.alphal] and the [.alphal_orig] directories. Both give the error about referencing DECC$SHR.

$ show def
DKA300:[000000.FARMER.ZIP30E.ALPHAL]
$ dir .exe

Directory DKA300:[000000.FARMER.ZIP30E.ALPHAL]

ZIP.EXE;3 322/411 12-OCT-2006 01:47:48.21
ZIPCLOAK.EXE;3 127/274 12-OCT-2006 01:49:02.58
ZIPNOTE.EXE;3 116/274 12-OCT-2006 01:49:07.33
ZIPSPLIT.EXE;3 123/274 12-OCT-2006 01:49:12.69
ZIP_CLI.EXE;3 367/411 12-OCT-2006 01:48:09.81

Total of 5 files, 1055/1644 blocks.
$ run zip
%IMGACT-F-SYMVECMIS, shareable image's symbol vector table mismatch
-IMGACT-F-FIXUPERR, error when ZIP referenced DECC$SHR

Then I tried the link procedure @[.vms]link_zip and get the following error.

$ @[.vms]link_zip
Cannot find any AXP object library for Zip.
You must keep all binary files of the object distribution
in the current directory !

I would appreciate if someone could produce the executable with Large file support. Have a couple of files that I'd be happy to test with.

Also, is the commercially available somewhere? I don't have the necessary tools to build for my specific installation and I hate to keep asking for help. You folks have been very kind and helpful so far.

My concern about maybe installing a GNU C compiler (not sure if that would even help), is that currently we have one vms machine for production and dev. Our disaster-recover / development machine will not be on-line for another month or so, I've been told. I don't want to do anything to trash my prod environment.

Thanks,

John
VMS 7.2-1
Alpha DS20E

Re: Zip/Unzip Data File Size Limitations?

Steven Schweda — Thu, 12 Oct 2006 15:20:02 GMT

> %IMGACT-F-SYMVECMIS, shareable image's symbol vector table mismatch

Yup, re-linking (at least) is required.

> Then I tried the link procedure
> @[.vms]link_zip and get the following error.
> [...]

Ah-ha! Caught not reading the (new) how-to
instructions. [.ZIP30E.VMS]INSTALL_VMS.TXT
(and/or [.UNZIP60C.VMS]INSTALL.VMS):

o Re-link the executables (small-file and large-file):

@ [.VMS]BUILD_ZIP LINK
@ [.VMS]BUILD_ZIP LARGE LINK

You want the second one of those. (These
BETA kits include some left-over clutter from
the older versions, like LINK_ZIP.COM.
Presumably, the released stuff will be
cleaner.)

> [...] GNU C [...]

I've tried (an old) GNU C on an old VAX, but
not on an Alpha, so, even if it's available,
I wouldn't bet on the results.

> Also, is the commercially available somewhere?

Don't be a coward.

> I hate to keep asking for help.

Guilt builds character. Fear not. If you
get too annoying, I'll complain. (Besides,
it's still being educational. I'm dying to
know if the "_72" kits actually help.)

Re: Zip/Unzip Data File Size Limitations?

John T. Farmer — Thu, 12 Oct 2006 15:46:30 GMT

Well, I forgot to mention that I did try the build procedure and it appeared to only get worse. Lots of undefined symbol messages related to DECC$ (see attached log of the build, I edited out company specific information). Finally the resulting exe, when run, produces the following Access Violation message. So, obviously, the link failed. I appreciate your enthusiasm. Helps motivate me not to throw in the towel!

$ zip
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=00000000000B

Your student,

John