topic Re: how to reduce directory size ? in Operating System - OpenVMS

how to reduce directory size ?

Lokesh_2 — Tue, 18 Nov 2003 05:19:58 GMT

Hi,

I want to reduce directoy size . I do NOT want to use backup/restore option. Here is an illustration for my problem:

1. I creat a directory testlj.dir whose initial size 1/4 (actual : 1 block, allocation: 4 blocks)
******************************************
DCCDT2> set def dsa2:[000000]
DCCDT2> creat/dir [testlj]
DCCDT2> dir/size=all testlj.dir

Directory DSA2:[000000]

TESTLJ.DIR;1 1/4

Total of 1 file, 1/4 blocks.
*****************************************

2. Now I increase the the directory size by running the following command procedure.

DCCDT2> set def [testlj]
DCCDT2> creat file.com
$loop:
$creat a.b
$creat b.c
$crea e.f
$creat c.d
$goto loop
$exit
Exit
DCCDT2> set noverify
DCCDT2> @file
Interrupt

DCCDT2> stop
DCCDT2> dir/size=all [-]testlj.dir

Directory DSA2:[000000]

TESTLJ.DIR;1 5/8

Total of 1 file, 5/8 blocks.
DCCDT2> dir

Directory DSA2:[TESTLJ]

A.B;49 B.C;49 C.D;48 E.F;49
FILE.COM;1

Total of 5 files.
DCCDT2> @file
Interrupt

DCCDT2> stop
DCCDT2> dir/size=all [-]testlj.dir

Directory DSA2:[000000]

TESTLJ.DIR;1 9/12

Total of 1 file, 9/12 blocks.
DCCDT2> purge
DCCDT2> dir/size=all [-]testlj.dir

Directory DSA2:[000000]

TESTLJ.DIR;1 5/12

Total of 1 file, 5/12 blocks.
DCCDT2>
**************************************

In the above example, now the directory size is :- 5 blocks actual & 12 blocks allocation size.

I want to reduce this allocation size back to 8 block ( cluster factor is 4 blocks)

Thanks & regards,
Lokesh

Re: how to reduce directory size ?

Bradford Hamilton_1 — Tue, 18 Nov 2003 05:48:12 GMT

Hi Lokesh,

One way to accomplish this is to issue the following command (using your example above):

set file/truncate DSA2:[000000]TESTLJ.DIR

This should reduce file allocation size back to 8 blocks.

--Brad

Re: how to reduce directory size ?

Willem Grooters — Tue, 18 Nov 2003 06:49:16 GMT

What I've learned (by study and experience - see an earlier thread) is that directory requires contiguous space.
I deduct from observations that the file system (XQP, to be exact) will locate new files based on a kind of hashing mechanism (correct me if I'm wrong). This _might_ mean that when a calculated slot within the directory-file is already occupied, it's very well possible that the new slot will be at the end of the directroy file - expanding it into the next cluster if needed.
For that reason, I doubt that limiting a directory's size is actullay useful, except you do not create any more files within it.

If you are so short of room on a disk, you definitely will run into trouble. Suppose you create a new file, and it will be placed at the end of the directory, it's possible the directory needs to be expanded into the next cluster. If no room is available at the curent disk-location, the file will have to be moved to a place on that disk that holds the whole allocated size as contiguous space. If that fails, you'll end up in an error. Or, expansion of the directory succeeds but there is too little room for the file to be allocated.

Re: how to reduce directory size ?

Ian Miller. — Tue, 18 Nov 2003 08:54:29 GMT

See also DFU DIRECTORY COMPRESS command. The DFU program is available on VMS freeware CD and also at

http://h71000.www7.hp.com/freeware/freeware50/dfu027a/

(NB DFU V3.0s due out RSN)

Note that directories should be kept below 127 blocks in size as there is a severe performance degradation above this size.

Re: how to reduce directory size ?

Hein van den Heuvel — Tue, 18 Nov 2003 10:40:23 GMT

Lokesh> want to reduce directoy size .

Why? Nothing better to do? What is the real problem you are trying to solve? What is the magnitude of the real directory (over) allocation you are addressing? You do knwo XQP/RMS will use the pre-allocated space for quicker future additions right? So shrinking is good for read-only directories, but the price to get is wrong is high!

Lokesh> I do NOT want to use backup/restore option.

Right. You don't want to because it is very wastefull, and potentially slow.
The VMS native sollution for this is RENAME.
This will NOT move the DATA, it WILL not create new files, it will just shuffle directory entries and tightly pack them as you go.

$CRE/DIR [.NEW]
$SET PROT NEW.DIR, DATA.DIR ! Allow delete access
$RENAME [.DATA]*.*;* [.NEW]
$RENAME DATA.DIR OLD.DIR ! Or delete
$RENAME NEW.DIR DATA.DIR

Lokesh> I increase the the directory size by running the following command
Lokesh> Loop: create a.b...

Creating mutliple versions of the same file is actually the most efficient, least space occupying method to store a file. It just adds a files-id and a little overhead to an array, based of the same name.
Next time, if you really want to explode a directory in a hurry make a loop with fresh,
It if you like creating work for the system, create them in reverse or random order.
long, filenames:
$loop:
$x = x=f$cvtime("",,"hundredth") + f$cvtime("",,"second")
$create 'x-get-creative-here.and-here
$if not-done then goto loop

Lokesh>

bradford> set file/truncate DSA2:[000000]TESTLJ.DIR

NO, IMHO it is NOT a good idea to truncate Directory files.
Use DFU, as Ian points out. Its great. best thing since sliced bread and all. Check out it's other functions (fragmentation report, quick file searched). It will soon ship with VMS.

Willem> _might_ mean that when a calculated slot within the directory-file is already occupied, it's very well possible that the new slot will be at the end of the directroy file

Close, but no cigar. Directory records are in strict sequential order. Use DUMP/RECO or DUMP/DIRECTORY to visualize. There is NO hashing on disk, but the XQP builds a little in memory table based on the first few characters (so don't pre-pend fluff!).
There is indeed a notion of name ranges for a BLOCK in a directory, trying to fit a new name in a logical spot. If that block is full, the whole directory will be re-copied opening up a block. This USED to be done with single block IOs as an easy way to garantuee recoverability.! (pre 7.2).

Ian> Note that directories should be kept below 127 blocks in size as there is a severe performance degradation above this size.

This is still agood guideline, but it never really was a hard and fast rule. It was there because RMS had a 127 block max size for (repeated) WILDCARD lookup ( mail$*.mai). Individual file lookup, without wildcard were/are done by the XQP with a modified binary search algoritme. RMS can now (7.2+?) deal with larger directories, making the performance 'knee' much softer. But exsessive directory sizes still hurt some.

hth,
Hein.

Re: how to reduce directory size ?

Lokesh_2 — Thu, 20 Nov 2003 05:24:52 GMT

Thanks to all for great suggestions. I am a happy now.

Regards,
Lokesh