topic Re: how to zip multiple files? in Operating System - OpenVMS

how to zip multiple files?

blaster2007 — Mon, 10 Sep 2007 13:30:18 GMT

I got hundreds files with continuous serial in the filename. e.g. doc001.dat, doc002.dat, docxxx.dat

how can I use zip to compress some of them(the situation below) to a single zip file?
say i) doc300.dat to doc320.dat
ii) doc005.dat, doc201.dat and doc412.dat

and I will transfer the zip file to windows xp, so $ zip/VMS/ should be used?

thanks,

running environment: openvms 8.2 I64, zip 2.3

Re: how to zip multiple files?

Peter Zeiszler — Mon, 10 Sep 2007 14:07:23 GMT

zip filename.zip doc*.dat
zip filename.zip doc300.dat,doc301.dat, etc.

zip filename.zip doc005.dat,doc201.dat,doc412.dat
I think you can also do the zip and then do new files to same zip file and it will add them.

Re: how to zip multiple files?

David Jones_21 — Mon, 10 Sep 2007 14:15:39 GMT

The zip program I use doesn't recognize comma as a delimiter for the filename arguments, only spaces (but wildcards work).

You can create a file containing the list of files to put in the zip archive (whitespace or newlines between filenames), define sys$input to this file, then run zip with the -@ option.

Re: how to zip multiple files?

Hein van den Heuvel — Mon, 10 Sep 2007 14:16:12 GMT

As Peter indicates, just use a list, until that become impractical.

How do yo know which files to select?
Are they listed in a file?

Well then just pass that file as SYS$INPUT and use the option '-@'.

That can also be done to read from a pipe:

$ pipe dir/col=1/nohead/notra *a*.txt; | zip test -@

What makes those selected files special?
Can you automate the selecting in a DCL/Perl script?

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

Re: how to zip multiple files?

Hoff — Mon, 10 Sep 2007 15:02:10 GMT

Welcome to ITRC.

The zip can add files into an existing zip archive (with the "-u" option), which means that a DCL loop is trivial to code; f$search can select the files in sequence, and each file gets added into the archive.

There are any number of examples of using the f$search call in a loop within DCL; here in ITRC, on the Freeware, in the "James" support datanase, in the OpenVMS manuals, and elsewhere on the 'net.

f$search("doc%%%.dat") would be at the core of the loop, repeated until no more files are found. (Or until you've extracted the trailing numerics and converted them to integers for range-testing, for instance.)

And since you're aiming for Microsoft Windows, make sure the RMS sequential data files are using Stream or Stream-LF record format. If they're not, you'll want to have your loop use CONVERT/FDL or such to convert the file structures over. This because Windows doesn't appreciate finding the RMS VFC structures, for instance.

Please upgrade your version of zip, too. You will want to use zip 2.32. There's a copy of this and of unzip 5.54 available on the Freeware V8.0 distro, over at the Process software OpenVMS archives, or likely over at the Info-Zip.org web site.

Re: how to zip multiple files?

Jon Pinkley — Mon, 10 Sep 2007 20:00:45 GMT

"And since you're aiming for Microsoft Windows, make sure the RMS sequential data files are using Stream or Stream-LF record format. If they're not, you'll want to have your loop use CONVERT/FDL or such to convert the file structures over. This because Windows doesn't appreciate finding the RMS VFC structures, for instance."

---------

I tried to reproduce the problem Hoff is describing above, but I must be doing something wrong.

I created a VFC file using the following command file, which I called test_vfc.com

-----------
$ open/write vfc vfc.dat
$ c=1
$ top:
$ write vfc c
$ c = c + 1
$ if c .le. 100 then goto top
$ close vfc
$ exit
-----------

ROOT$USERS:[JON]TEST_VFC.COM;1 8 lines

$ dir/ful vfc.dat
%DIRECT-W-NOFILES, no files found
$ @test_vfc
$ dir/ful vfc.dat

Directory ROOT$USERS:[JON]

VFC.DAT;1 File ID: (235106,52,0)
Size: 2/8 Owner: [ILS002,JON]
Created: 10-SEP-2007 19:52:12.67
Revised: 10-SEP-2007 19:52:12.70 (1)
Expires: 10-SEP-2007 19:52:12.71
Backup:
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 8, Extend: 0, Global buffer count: 0, No version limit
Record format: VFC, 2 byte header, maximum 0 bytes, longest 3 bytes
Record attributes: Print file carriage control
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:, World:
Access Cntrl List: None
Client attributes: None

Total of 1 file, 2/8 blocks.
$

Now using an old version of zip, (but I get the same results with 2.32)

$ zip
Copyright (C) 1990-1997 Mark Adler, Richard B. Wales, Jean-loup Gailly,
Onno van der Linden, Christian Spieler and Igor Mandrichenko.
Type 'zip "-L"' for software license.
Zip 2.2 (November 3rd 1997). Usage: zip=="$disk:[dir]zip.exe"
zip [-options] [-b path] [-t mmddyyyy] [-n suffixes] [zipfile list] [-xi list]
The default action is to add or replace zipfile entries from list, which
can include the special name - to compress standard input.
If zipfile and list are omitted, zip compresses stdin to stdout.
-f freshen: only changed files -u update: only changed or new files
-d delete entries in zipfile -m move into zipfile (delete files)
-R recurse into directories -j junk (don't record) directory names
-0 store only -l convert LF to CR LF (-ll CR LF to LF)
-1 compress faster -9 compress better
-q quiet operation -v verbose operation/print version info
-c add one-line comments -z add zipfile comment
-@ read names from stdin -o make zipfile as old as latest entry
-x exclude the following names -i include only the following names
"-F" fix zipfile("-FF" try harder) "-D" do not add directory entries
"-A" adjust self-extracting exe "-J" junk zipfile prefix (unzipsfx)
"-T" test zipfile integrity "-X" eXclude eXtra file attributes
"-V" save VMS file attributes -w append version number to stored name
-e encrypt -n don't compress these suffixes
$ zip vfc vfc.dat
adding: VFC.DAT (deflated 51%)
$ zip -l vfc-l vfc.dat ! that's a lowercase L to convert LF to CR LF
adding: VFC.DAT (deflated 57%)
$
$ zip232
Copyright (c) 1990-2006 Info-ZIP - Type 'zip "-L"' for software license.
Zip 2.32 (June 19th 2006). Usage: zip=="$disk:[dir]zip.exe"
zip [-options] [-b path] [-t mmddyyyy] [-n suffixes] [zipfile list] [-xi list]
The default action is to add or replace zipfile entries from list, which
can include the special name - to compress standard input.
If zipfile and list are omitted, zip compresses stdin to stdout.
-f freshen: only changed files -u update: only changed or new files
-d delete entries in zipfile -m move into zipfile (delete files)
-r recurse into directories -j junk (don't record) directory names
-0 store only -l (-ll CR LF to LF)
-1 compress faster -9 compress better
-q quiet operation -v verbose operation/print version info
-c add one-line comments -z add zipfile comment
-@ read names from stdin -o make zipfile as old as latest entry
-x exclude the following names -i include only the following names
"-F" fix zipfile("-FF" try harder) "-D" do not add directory entries
"-A" adjust self-extracting exe "-J" junk zipfile prefix (unzipsfx)
"-T" test zipfile integrity "-X" eXclude eXtra file attributes
"-V" save VMS file attributes ("-VV" also save allocated blocks past EOF)
"-R" PKZIP recursion (see manual) -w append version number to stored name
-e encrypt -n don't compress these suffixes
$ zip232 vfc232 vfc.dat
adding: VFC.DAT (deflated 51%)
$ zip232 -l vfc232-l vfc.dat
adding: VFC.DAT (deflated 57%)
$ backup/comp vfc.zip vfc232.zip
%BACKUP-E-VERIFYERR, verification error for block 1 of ROOT$USERS:[JON]VFC232.ZIP;1
$ backup/comp vfc-l.zip vfc232-l.zip
%BACKUP-E-VERIFYERR, verification error for block 1 of ROOT$USERS:[JON]VFC232-L.ZIP;1
$

Note: I did not use the "-V" switch, and that shouldn't be used when creating zip files that are going to be used on Windows.

Now from XP ftp the vfc*.zip files from VMS using binary mode transfer, and unzip using winrar 3.20 (what I use on Windows) into unique folders.

The vfc.zip and vfc232.zip create files that notepad can't deal with (everything on one line), but the vfc-l.zip and vfc232-l.zip both create text files that notepad has not problem with.

And Metapad can deal with all the vfc.dat files, it shows the files created without -l as being UNIX format.

My point is that I am not able to reproduce the problem with VFC files. If there are problems, then it was due to transferring the files or a different application unzipping the files. I even tried using the unzip built into XP, right click on .zip file, and select explore, then drag a copy to another folder. I got identical files to what winrar 3.20 extracted.

I do recommend using zip232, but I didn't see any difference in behavior for this particular test.

I am not saying that converting to stream-lf is bad, it just isn't necessary (at least in the case I tried).

If the files are text files, and you want the files to work on Windows, it is more important to use the zip "-l" switch to convert LF to CR LF than to convert to stream files, at least that is my experience.

Getting a subset of the files is accomplished best with the -@ as first mentioned by Dave, and embellished with pipe by Hein. Note that any directory qualifiers can be used, so you can limit the files matched with things like creation date.

Jon

Re: how to zip multiple files?

Hoff — Mon, 10 Sep 2007 20:05:31 GMT

Interesting. This might be something zip or the CRTL or such has remediated; I had no end of problems with low-level record structures when dealing with mixed platforms.

Re: how to zip multiple files?

Steven Schweda — Mon, 10 Sep 2007 23:04:41 GMT

> and I will transfer the zip file to windows
> xp, so $ zip/VMS/ should be used?

ZIP /VMS (zip -V) is useful only when the
destination is VMS, but it should be harmless
with Zip 2.32 (or later).

> running environment: openvms 8.2 I64, zip 2.3

Gack. Do what he said:

> Please upgrade your version of zip, too.
> You will want to use zip 2.32. There's a
> copy of this and of unzip 5.54 [...]

except that UnZip 5.52 is the latest released
version. If you need large-file (>2GB)
support, you'll need not-yet-released
versions.

> Note: I did not use the "-V" switch, and
> that shouldn't be used when creating zip
> files that are going to be used on Windows.

As I said, useless but harmless, nowadays.
(With Zip 2.3, it's a hazard.)

> Interesting. This might be something zip or
> the CRTL or such has remediated; I had no
> end of problems with low-level record
> structures when dealing with mixed
> platforms.

Files recognized by Zip as text are marked as
such in the archive, and UnZip (see "-a") can
consider this. On non-VMS systems, UnZip is
normally built with VMS_TEXT_CONV enabled,
which is intended to add the ability to
decode VMS-specific text-file formats.
Everything's complicated.

Re: how to zip multiple files?

Jon Pinkley — Tue, 11 Sep 2007 08:26:20 GMT

After Steven Schweda's note about "-V" being harmless after Zip 2.32 or later, I zipped the vfc.dat file using the command:

$ zip232 "-V" vfc232-v vfc.dat
adding: VFC.DAT (deflated 73%)
$ zip232 "-Vl" vfc232-vl vfc.dat

zip error: Invalid command arguments (can't use -V with -l or -ll)
$

Note that when you use "-V" it apparently changes the way the files are read/processed by zip, since it won't allow you to use anything that would affect the record delineation, and to me that makes sense, since you are asking it to preserve the VMS attributes.

I don't have info-zip's unzip on my PC, so it is possible that info-zip's unzip program has knowledge of how to interpret the stored VMS attributes and create a usable file for Windows. The unzip built into XP and the unzip built into WinRAR 3.20 are able to extract the file from vfc232-v.zip, but it isn't the same as what gets unzipped from vfc232-l. It is larger, 602 bytes vs. 392 bytes.

Here is where Hoff's recommendation to convert to stream format probably would make a difference, although I did not test that.

My point is that if you are targeting windows as the destination, I can see no advantage to using zip "-V" and I can see disadvantages of doing so, even if using zip 2.32, since at a minimum you will need a version of unzip that knows how to properly deal with the zip file produced with "-V".

I have attached a file zipped on my windows PC that has the following contents: I had to ftp it to VMS so I could get a listing, since I don't have info-zip unzip on my PC.

unzip -v vmsvfc.zip
Archive: ROOT$USERS:[JON]VMSVFC.ZIP;1
Length Method Size Ratio Date Time CRC-32 Name
-------- ------ ------- ----- ---- ---- ------ ----
0 Stored 0 0% 09-11-07 08:42 00000000 VMSVFC/
0 Stored 0 0% 09-10-07 20:01 00000000 VMSVFC/VFC/
292 Defl:N 142 51% 09-10-07 19:52 678bf1dc VMSVFC/VFC/vfc.dat
0 Stored 0 0% 09-10-07 20:01 00000000 VMSVFC/VFC-L/
392 Defl:N 167 57% 09-10-07 19:52 35e4e82f VMSVFC/VFC-L/vfc.dat
297 Defl:N 242 19% 09-10-07 20:15 8b221f6c VMSVFC/VFC-L.ZIP
272 Defl:N 229 16% 09-10-07 20:15 0681c3bc VMSVFC/vfc.zip
0 Stored 0 0% 09-10-07 20:15 00000000 VMSVFC/VFC232/
292 Defl:N 142 51% 09-10-07 19:52 678bf1dc VMSVFC/VFC232/vfc.dat
0 Stored 0 0% 09-10-07 20:15 00000000 VMSVFC/VFC232-L/
392 Defl:N 167 57% 09-10-07 19:52 35e4e82f VMSVFC/VFC232-L/vfc.dat
297 Defl:N 242 19% 09-10-07 20:15 a6596ab4 VMSVFC/VFC232-L.ZIP
0 Stored 0 0% 09-11-07 08:22 00000000 VMSVFC/VFC232-V/
602 Defl:N 162 73% 09-10-07 19:52 a8b729f1 VMSVFC/VFC232-V/vfc.dat
536 Defl:N 295 45% 09-11-07 08:12 4c0e4cec VMSVFC/VFC232-V.ZIP
272 Defl:N 229 16% 09-10-07 20:15 4eaf440a VMSVFC/VFC232.ZIP
1159 Defl:N 453 61% 09-11-07 08:42 88f75f49 VMSVFC/VMSVFC.md5
816 Defl:N 291 64% 09-11-07 08:42 0b2c71f4 VMSVFC/VMSVFC.sfv
-------- ------- --- -------
5619 2761 51% 18 files
$

Note the size of the unzipped files. 292 (with LF) 392 (with CR LF) 602 (from "-V")

So my recommendation stands: If you are zipping files on VMS to be read on Windows as text files, use zip -l on VMS, not zip "-V".

Steven Schweda, can you verify that you can unzip the vfc232.zip file on a windows box using info-zip unzip, and have it produce a file that is equivalent to the 392 byte version of vfc.dat? I.e. one that can be opened and displayed in Notepad. And what the correct switches to use are? (I would guess unzip -a vfc232.zip).

And I do agree with "Everything's complicated."

Jon

Re: how to zip multiple files?

Steven Schweda — Tue, 11 Sep 2007 08:56:04 GMT

> [...] can you verify that you can unzip the
> vfc232.zip file on a windows box using
> info-zip unzip, and have it produce a file
> that is equivalent to the 392 byte version
> of vfc.dat? I.e. one that can be opened and
> displayed in Notepad. And what the correct
> switches to use are? (I would guess
> unzip -a vfc232.zip).

I try my best to avoid Windows, and I succeed
pretty well (at least at home), so I can't
run that test. I'd guess "unzip -a", too.

I dimly recall getting a Kermit source kit a
while ago,as a Zip archive, which initially
got expanded into files with CR-LF line
endings. I was just starting to get upset
when I stumbled across the instruction to use
"unzip -a" to extract the stuff, and that
solved the problem. So, it did seem to
handle a case going in the other direction
(MSDOS-Windows to VMS). The same general
feature should work on a UNIX[-like] system,
too, except that the result would be
LF-delimited instead of CR-LF-. Look for
VMS_TEXT_CONV in the "unzip -v" output.

Re: how to zip multiple files?

blaster2007 — Tue, 11 Sep 2007 09:28:46 GMT

thank all,

I have tried to zip my files up, it's just fine by using wildcard and space as a delimiter.

And the -l works for the conversion of the files, but it's just if I don't use it, since it will be resorted after imported to excel, no matter if it's LF or CR LF.

Now, I still can't figure out how to use DCL command to compress files with continual serials in their filenames.
i.e. compress doc300.dat to doc320.dat in a folder that contains doc001.dat to doc999.dat

I'm just a newbie in this aspect...

Re: how to zip multiple files?

O'Malley — Tue, 11 Sep 2007 09:32:36 GMT

Using v2.3 use the /BATCH qualifier (not sure what the dash (-) character is)

$ DIR FILE*.EXT* /NOHEAD/NOTRAIL/OUT=FOO.LIS
$ ZIP/VMS ARCHIVE.ZIP /BATCH=FOO.LIS

Use /VMS incase you ever need to restore to a VMS system.

The issue I have found with ADDING files to an archive is that ZIP:
1. makes a copy the archive
2. adds the new file to the copy
3. deletes the old archive
4. copies the updated archive to the old archive name/location

All this duplication is not to noticable when the archive is small. When the archive gets large copying large archive files can take some time. Consider using /TEMP= too.

Re: how to zip multiple files?

Jan van den Ende — Tue, 11 Sep 2007 09:55:08 GMT

blaster,

>>>
i.e. compress doc300.dat to doc320.dat in a folder that contains doc001.dat to doc999.dat
<<<

Now look at O'Malley's solution:

>>>
$ DIR FILE*.EXT* /NOHEAD/NOTRAIL/OUT=FOO.LIS
$ ZIP/VMS ARCHIVE.ZIP /BATCH=FOO.LIS
>>>
Slightly modify it:

$ DIR doc3%%.DAT /..... etc

... For this, and MANY more (nearly all) VMS quetions, try
$ HELP or
$ HELP subject, eg
$ HELP DIR

and when you have found something that might look promising, then look at the example(s), and 9 out of 10 your solution should be clear.

And if not... you now have found us.

Have success, have fun!

Proost.

Have one on me.

jpe

Re: how to zip multiple files?

Jon Pinkley — Tue, 11 Sep 2007 10:06:49 GMT

One way to get only doc300.dat through doc320.dat is the following:

$ pipe dir/col=1/nohead/notrail doc30%.dat;,doc31%.dat;,doc320.dat; | zip -@j doc.zip

The "j" tells zip to ignore the directory, and is only significant if the you have your default directory set to something other than the location the doc files reside.

For example, if the files with the doc files is protected against writing, and there was a logical name "doc" that had the location of the doc files, you could use:

$ pipe dir/col=1/nohead/notrail doc:doc30%.dat;,doc:doc31%.dat;,doc:doc320.dat; | zip -@j doc.zip

Alternatively, you could use directory to get a listing of all the files with 3 character names that start with 3, and then edit that file to eliminate all the ones you are not interested in, and use that list with wither the /bat or piping the output of type into zip -@.

$ dir/nohead/notrai doc:doc3%%.dat;/out=tmp.lis
$ edit tmp.lis ! remove all non-wanted files
$ pipe type tmp.lis | zip -@j doc.zip

Re: how to zip multiple files?

blaster2007 — Tue, 11 Sep 2007 10:26:47 GMT

Thank all, again.
I think the problem is fixed here.

yeah, finally, I found u Guys.
It's just a nightmare that nobody in the workplace know how to work with the machine.

Re: how to zip multiple files?

Hein van den Heuvel — Tue, 11 Sep 2007 10:53:33 GMT

I'll tell you one more secret about OpenVMS.
It's not a secret really, just little known and little used.
The essential command being SET FILE /ENTER
Check it with help.

it creates an ALIAS for the file ON THE SAME DISK.
A little like a Unix hard link, but watch out, a DELETE will delete the file, not just the link. To remove the link use SET FILE/REMOVE. Obviously this delete danger is part of the reason it is little used.

Anyway, You can use it to readily create a collection of files to be ZIPped, or PRINTed, or FTPed, or whatever other processing may be needed on a collection.

Here it goes...

$CREATE/DIR [.TO_BE_ZIPPED]
$SET FILE/ENT= [.TO_BE_ZIPPED] doc30%.dat;
$SET FILE/ENT= [.TO_BE_ZIPPED] doc30%.dat;, doc320
$! ... and more as needed...
$DIR [.TO_BE_ZIPPED] ! Check
$ZIP test [.TO_BE_ZIPPED]*.*
$!Main Cleanup
$SET FILE/REMOVE [.TO_BE_ZIPPED]*.*.*
$!Optional further cleanup
$SET FILE/PROT TO_BE_ZIPPED.DIR ! Un-protect
$DELETE TO_BE_ZIPPED.DIR.

The whole sequence is relatively efficient as it really only manipulates a single file: TO_BE_ZIPPED.DIR. It does not copy data or crerate other files.

Alternative cleanup for those who like to live with privs:
$SET FILE/NODIR TO_BE_ZIPPED.DIR.
$DELETE TO_BE_ZIPPED.DIR. ! Poof

fwiw,
Hein.

Re: how to zip multiple files?

blaster2007 — Tue, 11 Sep 2007 11:22:29 GMT

should it be $SET FILE/REMOVE [.TO_BE_ZIPPED]*.*,*
instead of $SET FILE/REMOVE [.TO_BE_ZIPPED]*.*.* ?

it's nice to process big files, but it's so dangerous that if the cleanup step is forgot to run.

Re: how to zip multiple files?

Hein van den Heuvel — Tue, 11 Sep 2007 12:05:32 GMT

should it be $SET FILE/REMOVE [.TO_BE_ZIPPED]*.*,*
instead of $SET FILE/REMOVE [.TO_BE_ZIPPED]*.*.* ?

Well, you mean "*.*;*", that is with a semi-colon, not a comman, but NO.
That's an other little known secret I thought I'd toss in.
I'm glad you noticed :-).

You can use a "." as seperator between file-type and file-version in most cases.

While I do not encourage that usage in formal scripts, I personally do use it a lot to just delete the last version of a file. It types easier: $DELETE x.y.

>> it's nice to process big files, but it's so dangerous that if the cleanup step is forgot to run.

Well, the extra directory with aliases just sits there. No harm no foul :-).

Admittedly, depending on the exact backup tool and method deployed thay may become real files on a backup.

Hein.

Re: how to zip multiple files?

Jan van den Ende — Tue, 11 Sep 2007 12:10:49 GMT

Blaster,

>>>
should it be $SET FILE/REMOVE [.TO_BE_ZIPPED]*.*,*
instead of $SET FILE/REMOVE [.TO_BE_ZIPPED]*.*.* ?
<<<

Well, the second ( period or full stop as separator) works, and is is (at least to me) easier typing.
But "officially", the separator between file type & version# is the semicolon ( " ; " )
(This is also a demonstration of the unwise choice of display in these forums. In your first command you used a comma ( " , " ) as separator, and THAT would generate a syntax error.

hth

Proost.

Have one on me.

jpe

Re: how to zip multiple files?

Hein van den Heuvel — Tue, 11 Sep 2007 12:17:47 GMT

Hah! Beat you to it Jan! :-).

Just kidding.

I just happened to be doing my lunch break tour of the forums/c.o.v and such. :-).

>> (This is also a demonstration of the unwise choice of display in these forums.

Yeah... really hard to spot the comma/semi in the read display font. Nice and clear in the reply window and/or pasted to a terminal.

>> In your first command you used a comma ( " , " ) as separator, and THAT would generate a syntax error.

well, not a syntax error. It is valid, but useless as it fails to provide the explicit version number needed.

Hein.