Zip/Unzip Data File Size Limitations?

[John T. Farmer]

Just put InfoZips two utilities, Zip 2.3 and Unzip 5.50 on our Alpha/OpenVMS 7.2-1 system. I can zip and unzip small test files. Then I tried a 4.2 million block file and Zip does see the file. A small test file of the same name is zipped and unzip with ease.

Is there a file size limitation on what Zip & Unzip can handle? Is there a newer version that fixes this issue? This 4mil block file is text and should compress down significantly for ftp purposes.

Thanks,

John Farmer
john dot farmer at genworth dot com

[jaw_1]

Yes, but do note that it is the beta version, and also requires a C compiler to build. Take a look at:

ftp://ftp.info-zip.org/pub/infozip/OLD/beta/

James

[Steven Schweda · Accepted solution]

The file size limit is imposed by the C
run-time library, and it's 2GB (except on
some old VAX versions, where it's 4GB). At
4.2M blocks, you seem to be just over the
edge.

Not that it'd help with large files, but the
current released versions are UnZip 5.52 and
Zip 2.32

http://www.info-zip.org/
http://www.info-zip.org/UnZip.html
http://www.info-zip.org/Zip.html

There are some significant benefits to using
the newer versions, compared with UnZip 5.50
and Zip 2.3. (File I/O speed, "-V" archive
portability, less pointless Zip archive file
copying, and so on.)

> ftp://ftp.info-zip.org/pub/infozip/OLD/beta/

Yes, and at VMS V7.2-1, you may just have a
C RTL new enough for large-file support.

If you need binaries because you lack a C
compiler (and you're too lazy to use one of
the TestDrive Alpha systems), let me know,
and I can probably whip up some useful object
files.

[John T. Farmer]

Sorry, typo "and Zip doesn't see the file". But you folks already figured that out.

So what I'm hearing, is the size limitation is due to the C RTL. We don't have C installed on our system, so I'm at the mercy of someone creating that for me. I'd be happy to test it if someone can build it.

"(and you're too lazy to use one of
the TestDrive Alpha systems)"
What is a TestDrive system?

Thanks,

John

[Robert Gezelter]

John,

The "Test Drive" systems are a couple of systems operated by HP that have fairly public access (obviously within some limits). The pointer to the OpenVMS Test Drive is http://www.testdrive.hp.com/os/#ovms

- Bob Gezelter, http://www.rlgsc.com

[Steven Schweda]

> What is a TestDrive system?

http://www.testdrive.hp.com/
http://www.testdrive.hp.com/current.shtml

The price is right, but you'll need to look
elsewhere if you want to test drive a VAX.

> We don't have C [...]

Try:

http://antinode.org/ftp/info-zip/unzip60c_alphal.zip
http://antinode.org/ftp/info-zip/zip30e_alphal.zip
or:
ftp://antinode.org/info-zip/unzip60c_alphal.zip
ftp://antinode.org/info-zip/zip30e_alphal.zip

And try to ignore everything else in that
neighborhood, if you know what's good for
you.

These kits were built with "HP C V7.1-015 on
OpenVMS Alpha V7.3-2", so you'll probably
need to re-link. (For details, see
[.VMS]INSTALL_VMS.TXT.) The source is not
exactly the same as the original BETA kits,
but it shouldn't be much worse. There may be
some extra crud in there, too, as those were
my development directories. The warranty on
this stuff covers even less that the normal
it's-all-your-fault disclaimer would imply.

If a re-link fails, you may need to find
someone with a comparably old VMS version to
build the stuff. (Or perhaps install a new
C compiler kit, license or no, and/or a C RTL
patch to get the required stuff, if any.
It's been a while since I've had to worry
about linking on old VMS versions, so I'm
not sure how much can go wrong.) Wake me
when it gets worse.

[Steven Schweda]

Someone needs better (more VMS-tolerant) FTP
(proxy?) software:

29-SEP-2006 08:38:30.39 User:anonymous ident:proxy@ status:07649912 CWD dir:SYS$POSIX_ROOT^:^[info-zip^]zip30e_alphal.zip

alp $ sysmsg %x07649912
(SYS$COMMON:[SYSMSG]TCPIP$MSG.EXE;1:)
%TCPIP-E-FTP_BADDIR, invalid directory

"zip30e_alphal.zip" is not part of the
directory name. (My best guess: The request
was "CWD /[info-zip]zip30e_alphal.zip", which
didn't have a chance of being right.

Ask me why I map my anonymous FTP directory
to "/ftp/" on the Web server. Go ahead, ask.

On the bright side, having an FTP server with
which lameware can't cope does foil many
break-in attempts and abuse exploits (as
well as some legitimate users).

[Wim Van den Wyngaert]

We encountered the same problem with Sybase dumps. We now split the dump into 5 pieces by doing the dump to 5 devices.
There might be a utility to split a file into smaller pieces. Or you can write it yourself in any language.

Wim

[John T. Farmer]

Do I understand correctly, that the 2gb input file limit (C RTL) still exists even on newer versions of VMS, i.e. 8.x? Not that I'll be upgrading soon, but just wonder ed. I'm dealing with some pretty large files for data warehouse conversion.

thanks,

John

[Steven Schweda]

> Do I understand correctly, that the 2gb
> input file limit (C RTL) still exists even
> on newer versions of VMS, i.e. 8.x?

Large-file (>2GB) support in the C RTL seems
to be available beginning with VMS Alpha
V7.2 or so. After that, it's available in
everything except VAX (any version).

To make use of this large-file support, an
application (say, for example, Zip or UnZip)
may need to be re-coded to use larger
variables for file sizes and byte offsets,
and it must be re-built requesting large-file
support (defining the C macro "_LARGEFILE").
For some of the fine print, see
SYS$COMMON:[DECC$LIB.REFERENCE.DECC$RTLDEF]DECC$TYPES.H

In the case of Zip and UnZip, some new
archive formats are needed to accomodate
large file size data, so there were more
changes needed than just replacing "%d" with
"%lld" in a few places.

Zip 2.x and UnZip 5.x do not and will not
offer large-file support, anywhere, anytime.

Zip 3.x and UnZip 6.x will offer large-file
support. If you can accept the current,
unreleased, BETA versions (Zip 3.0e, UnZip
6.00c), you can have it now (with no known
major problems).

In fact, on VMS, for maximal downward
compatibility, the current builders for Zip
3.x and UnZip 6.x do not enable large-file
support by default. You need to specify
"LARGE" (somehow) at build time to get it.

[John T. Farmer]

Re: Zip 3.x and Unzip 6.X

Where do I sign up??? Seriously, I'll have a look at the beta zip and unzip, however, as most of you have read, I don't have a C compiler on my OpenVMS 7.2-1, Alpha DS20E system. Anyone know of pre-compiled binaries, or something ready-to-link?

I tried the zip/unzip from info-zip, but the link procedure failed "symbol" undefined.

Is there some other similar compression utility that doesn't have the 2gb limit?

Thanks,

John

[Thomas Ritter]

John, while not directly relevant, buy a C complier. Not expensive. C can be very useful as a general system management tool. A lot of our work is converting VMS stuff to some CSV format. C is mostly used.

[Steven Schweda]

> I tried the zip/unzip from info-zip,

Which one? I really expected the kit from
http://vms.process.com/ftp/vms-freeware/fileserv/
to work. No, wait. That was the "-V"
problem. I'm starting to get lost.

> but the link procedure failed "symbol"
> undefined.

Which symbol(s)? As usual, actual error
messages would help more than vague
descriptions of error messages.

> Is there some other similar compression
> utility that doesn't have the 2gb limit?

Yes, but you'll have the same problems with
them, and they can't preserve RMS file
attributes. See for example:

http://antinode.org/dec/sw/bzip2.html
http://antinode.org/dec/sw/gzip.html

If you don't get any better offers, I'll see
if I can find a way to build the beta kits
in a VMS V7.2-1-compatible way.

[John T. Farmer]

Error received when trying to create zip with the vms link procedure (info-zip stuff).

Directory DRA0:[USERS.FARMER.ZIP.VMS]

LINK_ZIP.COM;1 15/21 25-JAN-2006 22:48:14.00
MAKE_ZIP.COM;1 22/42 20-FEB-2006 22:54:20.00

Total of 2 files, 37/63 blocks.
$ @link_zip
%DCL-I-IGNQUAL, qualifiers appearing before this item were ignored
\SYMBOL\
%DCL-I-IGNQUAL, qualifiers appearing before this item were ignored
\SYMBOL\
Cannot find any AXP object library for Zip.
You must keep all binary files of the object distribution
in the current directory !



Directory DRA0:[USERS.FARMER.ZIP.VMS]

00BINARY.VMS;1 9/21 14-FEB-1999 02:15:26.00
00README.TXT;1 12/21 15-OCT-1997 14:09:02.00
CMDLINE.C;1 63/63 14-APR-2006 22:38:06.00
CVTHELP.TPU;1 11/21 17-APR-2005 08:59:22.00
DESCRIP.MMS;1 23/42 5-JUN-2006 19:29:22.00
LINK_ZIP.COM;1 15/21 25-JAN-2006 22:48:14.00
MAKEFILE.VMS;1 16/21 14-FEB-1999 02:15:26.00
MAKE_ZIP.COM;1 22/42 20-FEB-2006 22:54:20.00
OSDEP.H;1 6/21 29-JAN-2005 01:44:02.00
VMS.C;1 51/63 18-JUN-2006 20:17:06.00
VMS.H;1 21/21 3-AUG-2005 22:58:18.00
VMSDEFS.H;1 25/42 29-JAN-2005 01:45:16.00
VMSMUNCH.C;1 30/42 15-FEB-2005 21:59:06.00
VMSMUNCH.H;1 5/21 29-JAN-2005 01:46:10.00
VMSZIP.C;1 40/42 29-JAN-2005 01:46:24.00
VMS_IM.C;1 42/42 29-JAN-2005 01:45:18.00
VMS_PK.C;1 34/42 15-JUN-2005 23:50:16.00
VMS_ZIP.RNH;1 49/63 11-APR-2006 00:05:12.00
ZIPUP.H;1 2/21 29-JAN-2005 01:46:06.00
ZIP_CLI.CLD;1 5/21 14-APR-2006 22:39:00.00
ZIP_CLI.HELP;1 41/42 14-APR-2006 22:39:00.00

Total of 21 files, 522/735 blocks.


Apparently, I don't have all the pieces. Was just trying to get a copy of 2.3 linked for my vms version, to correct some corruption issues. Would consider the 3.x zip and 6.x unzip if their are linkable kits available.

Thanks,

John

[Steven Schweda]

It sure would help to know where from you're
starting. There's a lot of "info-zip stuff"
out there.

> Was just trying to get a copy of 2.3
> linked [...]

Don't do it. Zip 2.3 is not what you want.
For anything. However, for completeness:

%DCL-I-IGNQUAL, qualifiers appearing before this item were ignored
\SYMBOL\

I'd guess that those come from some "DELETE /SYMBOL" commands. That procedure assumes
that you haven't redefined "DELETE", which
you apparently have. At least it's a "-I-",
so that shouldn't have derailed anything.

> Cannot find any AXP object library for Zip.

And it's not obvious, but you need to be in the top-level directory to use LINK_ZIP.COM,
not in the [.VMS] directory:

@ [.VMS]LINK_ZIP.COM !! Not "@ LINK_ZIP.COM".

(Same for MAKE_ZIP.COM, not that you care.)

But, back to what you should be doing...

For a start, get the Zip 2.32 kit (with
binaries) from:

http://vms.process.com/ftp/vms-freeware/fileserv/ZIP.ZIP

and see if you can link that:

SET DEFAULT [.vms-binaries]
@link.com

Note that the builders for Zip 2.3 (and UnZip
5.50) are from before my time, and the ones
for later Zip 2.x (and UnZip 5.x) have not
been changed much. Everything's different
(possibly better) in the new (unreleased)
stuff.

> Would consider the 3.x zip and 6.x unzip if
> their are linkable kits available.

I don't have anything lying around which is
likely to be linkable on VMS V7.2-1, but
I'll see what I can do (unless someone else
leaps in before I can do it).

[Steven Schweda]

I can't test the stuff, but there's at least
a small chance. Suck down these things:

http://antinode.org/ftp/info-zip/unzip60c_alphal.zip
http://antinode.org/ftp/info-zip/unzip60c_alphal_72.zip
http://antinode.org/ftp/info-zip/zip30e_alphal.zip
http://antinode.org/ftp/info-zip/zip30e_alphal_72.zip

The plain-name archives are the full BETA
kits with modern binaries in the [.ALPHAL]
directories. The "_72" archives contain
replacement [.xxxx.ALPHAL] directories with
potentially old-compatible binaries. No
bets.

The suggested procedure is to unpack the
plain-name archives, and then move the
original [.ALPHAL] directories out of the
way:

RENAME [.ZIP30E]ALPHAL.DIR [.ZIP30E]ALPHAL_orig.DIR

RENAME [.UNZIP60C]ALPHAL.DIR [.UNZIP60C]ALPHAL_orig.DIR

Then unpack the "_72" archives to provide
the alternate [.*.ALPHAL] directories.

Finally, read [.VMS]INSTALL_VMS.TXT (Zip) or
[.VMS]INSTALL.VMS (UnZip), which should
explain how to re-link stuff.

You could also try the executables before
re-linking, just in case.

[John T. Farmer]

Steven,

You are correct, I was in the [.VMS] directory. But that asside, I will download and try the zip3 and unzip6 per your links.

You are also correct, that there is so much of this stuff floating around, it is difficult to troubleshoot if you don't know for sure where you started. I believe this was the download from process.com.

I do appreciate your detailed responses and will reply back with my results (successes!).

Thanks,

John

[John T. Farmer]

Steven,

I followed your suggestions, downloaded the zip30e and unzip60c, and the _72 zips. Tried running the zip.exe program from the [.alphal] and the [.alphal_orig] directories. Both give the error about referencing DECC$SHR.

$ show def
DKA300:[000000.FARMER.ZIP30E.ALPHAL]
$ dir .exe

Directory DKA300:[000000.FARMER.ZIP30E.ALPHAL]

ZIP.EXE;3 322/411 12-OCT-2006 01:47:48.21
ZIPCLOAK.EXE;3 127/274 12-OCT-2006 01:49:02.58
ZIPNOTE.EXE;3 116/274 12-OCT-2006 01:49:07.33
ZIPSPLIT.EXE;3 123/274 12-OCT-2006 01:49:12.69
ZIP_CLI.EXE;3 367/411 12-OCT-2006 01:48:09.81


Total of 5 files, 1055/1644 blocks.
$ run zip
%IMGACT-F-SYMVECMIS, shareable image's symbol vector table mismatch
-IMGACT-F-FIXUPERR, error when ZIP referenced DECC$SHR


Then I tried the link procedure @[.vms]link_zip and get the following error.

$ @[.vms]link_zip
Cannot find any AXP object library for Zip.
You must keep all binary files of the object distribution
in the current directory !


I would appreciate if someone could produce the executable with Large file support. Have a couple of files that I'd be happy to test with.

Also, is the commercially available somewhere? I don't have the necessary tools to build for my specific installation and I hate to keep asking for help. You folks have been very kind and helpful so far.

My concern about maybe installing a GNU C compiler (not sure if that would even help), is that currently we have one vms machine for production and dev. Our disaster-recover / development machine will not be on-line for another month or so, I've been told. I don't want to do anything to trash my prod environment.

Thanks,

John
VMS 7.2-1
Alpha DS20E

[Steven Schweda]

> %IMGACT-F-SYMVECMIS, shareable image's symbol vector table mismatch

Yup, re-linking (at least) is required.

> Then I tried the link procedure
> @[.vms]link_zip and get the following error.
> [...]

Ah-ha! Caught not reading the (new) how-to
instructions. [.ZIP30E.VMS]INSTALL_VMS.TXT
(and/or [.UNZIP60C.VMS]INSTALL.VMS):

o Re-link the executables (small-file and large-file):

@ [.VMS]BUILD_ZIP LINK
@ [.VMS]BUILD_ZIP LARGE LINK

You want the second one of those. (These
BETA kits include some left-over clutter from
the older versions, like LINK_ZIP.COM.
Presumably, the released stuff will be
cleaner.)

> [...] GNU C [...]

I've tried (an old) GNU C on an old VAX, but
not on an Alpha, so, even if it's available,
I wouldn't bet on the results.

> Also, is the commercially available somewhere?

Don't be a coward.

> I hate to keep asking for help.

Guilt builds character. Fear not. If you
get too annoying, I'll complain. (Besides,
it's still being educational. I'm dying to
know if the "_72" kits actually help.)

[John T. Farmer]

Well, I forgot to mention that I did try the build procedure and it appeared to only get worse. Lots of undefined symbol messages related to DECC$ (see attached log of the build, I edited out company specific information). Finally the resulting exe, when run, produces the following Access Violation message. So, obviously, the link failed. I appreciate your enthusiasm. Helps motivate me not to throw in the towel!

$ zip
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=00000000000B


Your student,

John

[Steven Schweda]

> %LINK-I-UDFSYM, DECC$FEATURE_GET_INDEX
> %LINK-I-UDFSYM, DECC$FEATURE_GET_VALUE
> %LINK-I-UDFSYM, DECC$FEATURE_SET_VALUE

Those may be my fault. I'll look into that
and let you know.

> %LINK-I-UDFSYM, DECC$__OFF64_FSEEKO
> %LINK-I-UDFSYM, DECC$__OFF64_FTELLO
> %LINK-I-UDFSYM, DECC$__OFF64_UTC_FSTAT
> %LINK-I-UDFSYM, DECC$__OFF64_UTC_STAT

Those are part of the C RTL large-file
support, so if they're missing, you may need
a C RTL patch. Trusting your claim of
"7.2-1", I'd vote for:

ftp://ftp.itrc.hp.com/openvms_patches/alpha/V7.2-1/VMS721_ACRTL-V0400.PCSI-DCX_AXPEXE
ftp://ftp.itrc.hp.com/openvms_patches/alpha/V7.2-1/VMS721_ACRTL-V0400.txt

Note:

3 KIT DEPENDENCIES

3.1 The following remedial kit(s) must be installed BEFORE
installation of this, or any required kit:

VMS721_UPDATE-V0300

which is in the same neighborhood:

ftp://ftp.itrc.hp.com/openvms_patches/alpha/V7.2-1/VMS721_UPDATE-V0300.PCSI-DCX_AXPEXE
ftp://ftp.itrc.hp.com/openvms_patches/alpha/V7.2-1/VMS721_UPDATE-V0300.txt

and claims to need only VMS721_PCSI-V0100:

ftp://ftp.itrc.hp.com/openvms_patches/alpha/V7.2-1/VMS721_PCSI-V0100.PCSI-DCX_AXPEXE
ftp://ftp.itrc.hp.com/openvms_patches/alpha/V7.2-1/VMS721_PCSI-V0100.txt

PRODUCT SHOW PRODUCT /FULL should reveal if
you already have any of those installed.

Boy, this _is_ fun.

[Steven Schweda]

Ok, I blew the __CRTL_VER setting in the
first pass. The new version "_72" kits
should fix the "%LINK-I-UDFSYM,
DECC$FEATURE_*" complaints. New ones are at:

http://antinode.org/ftp/info-zip/unzip60c_alphal_72.zip
http://antinode.org/ftp/info-zip/zip30e_alphal_72.zip

The "%LINK-I-UDFSYM, DECC$__OFF64_*"
complaints are, I claim, all your fault now,
but I am filled to overflowing with hope for
the VMS721_ACRTL-V0400 C RTL patch kit.

Hey. What could go wrong?

[John T. Farmer]

I deleted the existing [.alphal] directory, downloaded the new "_72" files. unzipped, no noticable difference in link errors.

:(

I'll be talking with the VMS system manager tomorrow about the patch application and what that procedure will be. If I get it approved, maybe that will handle the C RTL issues.

thanks,

john

[Steven Schweda]

Oops (again). I think that the second Zip
"_72" kit was ok, but the UnZip one was bad
(again). There should be a third (sigh)
UnZip "_72" kit at the same old place:

http://antinode.org/ftp/info-zip/unzip60c_alphal_72.zip

That should put the remaining
"DECC$FEATURE_*" complaints to rest.

Sorry for the extra bother.

The C RTL patch (or a VMS upgrade) is likely
to be your only hope for the "DECC$__OFF64_*"
complaints.

[Jan van den Ende]

John,

from your Forum Profile:


I have assigned points to 7 of 62 responses to my questions.


Maybe you can find some time to do some assigning?

http://forums1.itrc.hp.com/service/forums/helptips.do?#33

Mind, I do NOT say you necessarily need to give lots of points. It is fully up to _YOU_ to decide how many. If you consider an answer is not deserving any points, you can also assign 0 ( = zero ) points, and then that answer will no longer be counted as unassigned.
Consider, that every poster took at least the trouble of posting for you!

To easily find your streams with unassigned points, click your own name somewhere.
This will bring up your profile.
Near the bottom of that page, under the caption "My Question(s)" you will find "questions or topics with unassigned points " Clicking that will give all, and only, your questions that still have unassigned postings.

Thanks on behalf of your Forum colleagues.

PS. - nothing personal in this. I try to post it to everyone with this kind of assignment ratio in this forum. If you have received a posting like this before - please do not take offence - none is intended!

PPS. - Zero points for this.

PPPS.
John,
take this chance to view a special phenomenon:
If you call up Steven's profile before assigning, AND you are not too scroogy, then after assigning you refresh the profile, and admire his changed headgear!
(which also gives you the chance to launch a congrats thread before anyone bets you to it)

Proost.

Have one on me.

jpe