Operating System - OpenVMS
1754018 Members
7511 Online
108811 Solutions
New Discussion юеВ

Re: Never ending ZIP process ...

 
SOLVED
Go to solution
Hein van den Heuvel
Honored Contributor

Re: Never ending ZIP process ...

For directories with lots of litlle files yo may also want to consider an LD devices on a file.

The LD device could then have a cluster size which works wel with the typical smal file size.

When all files are collected, dismount the LD device and zip up the file containing the device !?

Applications creating lots of file in a single directory can often, but not always, be helped transparently by using SEARCH LIST. Define that target directory as [current],[rest].
That could also be [200710],[200709],[2007],[2006]
Once a month, or once a year, or whenever appropriate, you push a new empty, but pre-allocated dircetory in from of the definition and perhaps clean up the tail.


Btw.. on my last comment on the perl helping too much, that really applies to a simple DIRECTORY also, unless your default is actually in the directory itself.

Below an example of that, still using perl.
By using the directory command, all the the slection switches (/BEFORE...) are aviable.
Perl has those also, but we all know directory better.


Hein,


$ perl [-]rename_dir.pl """2006*.*"""
wild: "2006*.*"
2006AAP.TMP;1 --> [-.tmp_helper]0000002006AAP.TMP;1
2006MIES.TMP;2 --> [-.tmp_helper]0000012006MIES.TMP;2
2006MIES.TMP;1 --> [-.tmp_helper]0000022006MIES.TMP;1
2006NOOT.TMP;1 --> [-.tmp_helper]0000032006NOOT.TMP;1
4 files moved to [-.tmp_helper]

Directory [HEIN.TMP_HELPER]

0000002006AAP.TMP;1 0000012006MIES.TMP;2
0000022006MIES.TMP;1 0000032006NOOT.TMP;1

Total of 4 files.
$ dir [-.tmp*]

Directory [HEIN.TMP]

2005MIES.TMP;1 2007MIES.TMP;1 FOO.DIR;1 NEWSRC.;1
OTS_HIST_DETAIL.DIR;1

Total of 5 files.

Directory [HEIN.TMP_RENAMED]

2006AAP.TMP;1 2006MIES.TMP;2 2006MIES.TMP;1 2006NOOT.TMP;1

Total of 4 files.

Grand total of 2 directories, 9 files.
$ type [-]rename_dir.pl
use strict;
#use warnings;

my $HELPER = "[-.tmp_helper]";
my $TARGET = "[-.tmp_renamed]";
my $i = 0;
my @files;
$_ = shift or die "Please provid double quoted wildcard filespec";

print "wild: $_\n";
s/"//g;
my $wild = $_;
foreach (`DIRECTORY/COLU=1 $wild`) {
chomp;
$files[$i++] = $_ if /;/;
}
die "Please provide double quoted wildcard filespec" if @files < 2;

# phase 1

for ($i=0; $i<@files; $i++) {
my $name = $files[$i];
my $new = sprintf("%s%06d%s",$HELPER,$i,$name);
print "$name --> $new\n";
rename $name, $new;
}

print "$i files moved to $HELPER\n";
system ("DIRECTORY $HELPER");
# phase 2

while ($i-- > 0) {
my $name = $files[$i];
rename sprintf("%s%06d%s",$HELPER,$i,$name), $TARGET.$name;
}



Hein van den Heuvel
Honored Contributor

Re: Never ending ZIP process ...

Hoff,

I think Andy did the directory shuffle improvements in 7.2 but I'm travelling and do not have my (VMS)notes here to verify that.

The actual IO size used is defined by SYSGEN param: acp_maxread

http://h71000.www7.hp.com/DOC/82FINAL/6048/6048pro_090.html

All,

Please find attached a special XFC trace our friend Mark H made, showing the XQP directory management for behaviour. The example is for a delete. For a rename the bitmap writes will go away, and the indexf.sys writes might go away (directory backlink update!?) but the directory IO is exactly the same on the old side and joied with directory IO fo rthe new name.

The trace shows most recent on top.
So you need to start at teh bottom
Te first few deletes are 'easy'.
Then about 20 lines from the bottom a delete empties a directory blocks and triggers a down shuffle.

Enjoy!
Hein van den Heuvel
HvdH Performance Consulting.


Art Wiens
Respected Contributor

Re: Never ending ZIP process ...

What would the syntax be to use /inlist= ?

$ zip_cli test.zip /inlist=test_tozip.lis /move/keep/vms/nofull_path

zip error: Invalid command arguments (nothing to select from)

Where the inlist file has:

$ ty test_tozip.lis
TEST.COM;13
TEST.COM;12
TEST.COM;11
TEST.COM;10
TEST.COM;9
TEST.COM;8
TEST.COM;7
TEST.COM;6
TEST.COM;5
TEST.COM;4
TEST.COM;3
TEST.COM;2
TEST.COM;1

Art
Art Wiens
Respected Contributor

Re: Never ending ZIP process ...

Or should I actually be using the /batch= qualifier rather than /inlist= ?

Art
Art Wiens
Respected Contributor

Re: Never ending ZIP process ...

Yes Art, use the /batch qualifier:

$ zip_cli test.zip /batch=test_tozip.lis /move/keep/vms/nofull_path
adding: TEST.COM;13 (deflated 45%)
adding: TEST.COM;12 (deflated 12%)
adding: TEST.COM;11 (deflated 15%)
adding: TEST.COM;10 (deflated 15%)
adding: TEST.COM;9 (deflated 10%)
adding: TEST.COM;8 (deflated 4%)
adding: TEST.COM;7 (deflated 10%)
adding: TEST.COM;6 (deflated 10%)
adding: TEST.COM;5 (deflated 1%)
adding: TEST.COM;4 (deflated 1%)
adding: TEST.COM;3 (deflated 63%)
adding: TEST.COM;2 (deflated 63%)
adding: TEST.COM;1 (deflated 63%)

Cheers,
Art
Steven Schweda
Honored Contributor

Re: Never ending ZIP process ...

> Yes Art, use the /batch qualifier:

Oops. Yes. Sorry. /EXCLUDE (-x), /EXLIST
(-x@), /INCLUDE (-i), and /INLIST (-i@) are
for file name filtering. /BATCH (-@) is for
the actual file name specification task.

Boy, you just can't trust me for anything.
Dean McGorrill
Valued Contributor

Re: Never ending ZIP process ...

>>> As you move stuff out of the directory, there was a directory file 'defragger' tool around once. <<<

hoff this was some tool one of the system managers had to shrink a directory, supposedly worked faster. I didn't trust it.
reason I rember it was it broke on him once
doing users mail and corrupted the directory file. wound up picking up all the
files out of syslost, he got em back with anal/disk/repair. (unhappy user)
Hein van den Heuvel
Honored Contributor

Re: Never ending ZIP process ...

Re: Directory compression

Well, there used to be a tool called 'comdir' we (Colin Blake) developped in the 80's in Valbonne.

DFU picked this up in the DIRECTORY /COMPRESS command with optional /FILL

DFU
DIRECTORY
/FILL_FACTOR=percentage

This qualifier is only valid in combination with /COMPRESS. Default behaviour for DFU is to compress a directory as tight as possible; this is equivalent to /FILL_FACTOR=100. By choosing a lower fill_factor DFU will leave some free space in each directory block. /FILL_FACTOR may be between 50 and 100 %. Caution : choosing a fill_factor lower than 100% may fail if the directory file is not large enough. In that case DFU will signal an error and advise using a higher fill_factor.


Hein.

Wim Van den Wyngaert
Honored Contributor

Re: Never ending ZIP process ...

If the application uses a logical to access the files, you can use a search list to force creation in smaller directories (without modifiying the application itself.

define logic dev:[2006.12],dev:[2006.11]

create logic:a.txt

The file will be created in the 12 directory.

In January you simply change it to

define logic dev:[2007.1],dev:[2006.12],dev:[2006.11]

and files are created in the 1 directory.

Thus you can zip a much smaller directroy.

This will of course not solve your problem now because tou already are stuck with the big directory.

Wim
Wim
labadie_1
Honored Contributor

Re: Never ending ZIP process ...

A copy of an old post


I suppose that your appli puts files in a directory named disk$appli,
and that your appli is started every morning and shut every evening.

May be you should do the following, define disk$appli as a search-list

The following will automagically roll

def disk$appli disk:-
disk:-
disk:-
disk:-
disk:-
disk:-
disk:

Of course you have to create your directory <.monday> and so

You can of course use a little dcl to have a search-list with many more
elements, and make it roll.

On monday you can quietly move the file in <.tuesday> and other
directories to otherwhere, while not disturbing the application.