Map area Retrieval pointers

[Mario Abruzzi]

What is the correct way interpret the retrieval pointers display when dumping a file header? Is this a good way to determine file fragmentation?

[Garry Fruth]

If by interpret, you mean determining if the file is fragmented, then I would say count them. Using DUMP/HEADER, count the number of rows of "Count:"..."LBN:". Compare the row count to the disks window size; that would be the sysgen parameter ACP_WINDOW unless you use MOUNT/WINDOW. If the number of segments in the file is greater than the window size, then there is a potential for a window turn, which can slow things down a bit. It has been a while since I refered to the documentation on this subject; so hopefully others in the forum will correct me if I am mistaken.

File headers can be extended. I would consider any file with a non-zero 'Extension file identifcation' as being fragmented.

[John Gillings]

Mario,

> Is this a good way to determine file fragmentation?

Not just "good", for individual files, this is "it".

There's not a lot to interpret. The map area describes the extents making up the file. Each extent has a Count (how many blocks it contains) and an LBN (its location on the disk). By definition, any file with more than one extent is fragmented. The more extents it has, the more fragmented the file.

Fragmentation, in and of itself, is not necessarily "bad". Most of the arguments for regular defragging of disks assume a single user operating system with strictly sequential file accesses and file systems that don't scale well. These aren't necessarily true on OpenVMS. An OpenVMS file which can fit all its extents in the primary header will probably be fine in terms of performance. That's because the file system will cache the map area as a whole. Depending on the size of the extents, that amounts to between 50 and 70 extents.

Once you're over that limit, you require extension headers. In the file header display you can see this up near the top as a non-zero "Extension file identification". For example:

File identification: (27259,102,0)
Extension file identification: (27267,94,0)

The last header in the chain looks like this:

File identification: (27275,81,0)
Extension file identification: (0,0,0)

To determine the fragmentation of a disk as a whole, use DFG (example below). Note that you can legally install DFG and use the SHOW command without a license PAK. Install from the Consolidated Distribution. You have to lie (sorry!) - Say "YES" when asked if you have a PAK and "NO" to running the IVP.

$ DEFRAG SHOW USER$DISK
Disk File Optimizer for OpenVMS DFG E2.7
© Copyright 2003 Hewlett-Packard Development Company,L.P.

F r a g m e n t a t i o n R e p o r t

USER$DISK 9-DEC-2004 11:37:31.29

The fragmentation index is 64.7
1 - 20.9 is excellent
21 - 40.9 is good
41 - 60.9 is fair
61 - 80.9 is poor
81 - 100 indicates a badly fragmented disk
Approximately 44.7 (out of 80.0 possible) is due to file fragmentation
Approximately 20.0 (out of 20.0 possible) is due to freespace fragmentation


Freespace Summary:
Total free space: 218124 blocks
Percentage free: 0 (rounded)
Total free extents: 720
Maximum free extent: 16386 blocks, LBN: 31905720
Minimum free extent: 6 blocks, LBN: 20957454
Average free extent: 302 blocks
Median free extent: 30 blocks


File Fragmentation Summary:
Number of files (with some allocation): 67137
Total file extents on the disk: 104329
Average number of file extents per file: 1.553972
Median number of file extents per file: 1

Most Fragmented File:
[USER$TSC.GILLINGS.AH277359]NTN1_CRASH2_6-AUG.DMP;1 (1619 extents)

[Hein van den Heuvel]

>> What is the correct way interpret the retrieval pointers display when dumping a file header?

Each one you see is a little contiguous chunk starting at that LBN, size = size.

>> Is this a good way to determine file fragmentation?

Count them and you'll know the number of fragments.
Look at the LBNs and you'll 'see' whether the file is spread out all over the place or just had alternating allocation with other files (less of a problem, but more avoidable).


I like the DFU freeware as a report/slection tool:

$ mcr dfu report defrag sys$disk
:
Total fragments : 73477
Average fragments per file : 1.093
File fragmentation index : 0.368 (excellent)
Average size per fragment : 97
Most fragmented file :
DRA0:[HEIN.VMS]F11X.ZIP;1 ( 4616/4617 blocks; 77 fragments)


or:

$ mcr dfu search/frag=mini=200 sys$disk:
:
NEWFOLDER.PST;1 ______259232/259236 16/1137
HEIN.BCK-DCX_AXPEXE;1 _96581/96588 __4/274
EVENTLOG_SAMPLE.BCK;1 134253/134253 _4/275
EVENTLOG_SAMPLE.IDX;1 120042/120042 10/898
%DFU-S-FND , Files found : 4, Size : 610108/610119, Hdr/Frag : 34/2584

Or i use an awk or perl 'one liner' to read the header dump:

$ pipe dump/head/blo=cou=0 [.VMS]F11X.ZIP | gawk/com="/seg/{h=1+$4;print f}/Cou/{f++;s+=$2}END{print h,f,s}" sys$pipe
77
179
3 199 4617

btw... This actually confirmed a buglette in this old dfu 2.7 (3.1 now available).
The reported worst file was the file with most fragments in the first header, not the most fragmented file!

Cheers,
Hein.

[Jan van den Ende]

John,

glad to read from an expert what was long ago explained to me, with the logic that made me believe it: let VMS care about fragmentation, it usually is NOT worth the trouble to defragment


Fragmentation, in and of itself, is not necessarily "bad". Most of the arguments for regular defragging of disks assume a single user operating system with strictly sequential file accesses and file systems that don't scale well. These aren't necessarily true on OpenVMS. An OpenVMS file which can fit all its extents in the primary header will probably be fine in terms of performance. That's because the file system will cache the map area as a whole.


.. and as it was explained to me _UNLESS_ you are VERY short of memory, set SYSGEN's ACP_WINDOW to 255, ("cathedral window") and the above goes for multi-header files as well.
Only if a multi-header file has high activity, then it might be a case to consider moving that one to a less fragmented location. (which in itself might require defragging free space first, CATCH 22).

_IF_ potential fragmentation _IS_ an issue, it is much more rewarding to handle that in advance.
So, which files are at risk? That would be (more or less continuously) growing files that are 'frequently' accessed.
Normally, that excludes LOG files. Yes, they are written chunck by chunck, more often than not requiring an extension. But also, on Average, they are accessed only two more times: usually they are only read to check for errors (if even that, I know lots of cases where they are only ever read if there is some trigger), and one more time to delete them.

But, for your application data files: start with allocating them "big enough", and then
$ SET FILE /EXTEND=.
Small complication for that last number: max value 65535. But that IS a "little" bit less fragmentation than the default value of 5..


To us, up till now this has been working fine.

YMMV, of course.

Proost.

Have one on me.

Jan

[Karl Rohwedder]

Hein,

will DFU 3.1 be in the next freeware CD and/or is there another download location available?

Hint:
We are using the fragmentation avoider FAVOID
(I think its from Glenn Everhartm don't know a source at the moment), which extents files based on their current size (customizable).

[Robert Gezelter]

Mario,

A small point that is easy to miss in the discussion on this topic:

File mapping pointers cover only a limited amount of space. A large CONTIGUOUS file will have many file headers, and may very well have extension file headers. For example (details ommitted in an attempt to maintain clarity):

Mapping Pointer 1: LBN 1000, 256
Mapping Pointer 2: LBN 1256, 256
...
Mapping Pointer n: LBN 1000+(256*(n-1)), 256

What you have is a contiguous file, admittedly a very large one.

See the file header definitions for the exact sizes of the various mapping entries.

I hope that the above is useful.

- Bob Gezelter, http://www.rlgsc.com

[Uwe Zessin]

In my book, that is not a contiguous file, because it does not have the 'contiguous' file attribute set.

[Hein van den Heuvel]

Karl

DFU V3.1-1 replaces V3.1, Ton fixed a serious issue recently, which (fortunately) just made it onto the new freeware CD... to be released with 8.2 early 2005. He suggests that if you have a copy of V3.1, to no longer use it.

Bob,

I think you are mistaken. A contiguous file has just one mapping pointer. In 'format 3' those can have a 32-bit size.
See $fm2def in sys$library:lib.mlb and the likes (Or the file systems internals book of course). Perhaps confused with the old (pre 6.0) WCB restriction of 65536/retrieval pointer?

hth,
Hein.

[Bojan Nemec]

Thats true, if the file is extended contiguos the last retrieval pointer is modified to show the actual block count.

Bojan

[Robert Gezelter]

Hein,

Thank you for the correction.

I do see quite a mix of systems, and disks which have been migrated between versions. So it is quite possible that all of the cases I have seen have involved files created by older versions.

I did, however, want to ensure that people look carefully, and avoid the "multiple mapping pointers == fragmentation" mindset.

- Bob Gezelter, http://www.rlgsc.com