topic Space available in indexf.sys in Operating System - OpenVMS

Space available in indexf.sys

K5TTT — Tue, 12 Dec 2006 14:45:25 GMT

I found this algorithm on the net to determine free space within the indexf.sys file.

(310 bytes in map area) - (6 times the number of map pointers) - (ACL size) - (reserved area size) = xxx,
where xxx is the number of bytes free in the indexf.sys header. Thus xxx/6 (6 is avg size in bytes of mapping pointer) will give the approximate number of mapping pointers that can be added.

I easily see the number of map pointers from issuing the dump/header on indexf.sys, but I don't see the ACL size or reserved area size from that information. All I see is the offset.

Can someone point me to where that info is, or show me a different method of determining how much free space I have available in the index?

I encountered file header full issues on this disk while only 700,000 files were on the disk and max files is >13 million. Someone else did a backup/restore while I was on vacation, so I don't know how many extents there were to the index before the rebuild.

$ DUMP/HEADER/BLOCK=COUNT=0 $1$dga81:[000000]INDEXF.SYS

Dump of file $1$DGA81:[000000]INDEXF.SYS;1 on 12-DEC-2006 13:30:55.88
File ID (1,1,0) End of file block 845968 / Allocated 1503448

File Header

Header area
Identification area offset: 40
Map area offset: 100
Access control area offset: 255
Reserved area offset: 255
Extension segment number: 0
Structure level and version: 2, 1
File identification: (1,1,0)
Extension file identification: (0,0,0)
VAX-11 RMS attributes
Record type: Fixed
File organization: Sequential
Record attributes:
Record size: 512
Highest block: 1503448
End of file block: 845969
End of file byte: 0
Bucket size: 0
Fixed control area size: 0
Maximum record size: 512
Default extension size: 0
Global buffer count: 0
Directory version limit: 0
File characteristics: Contiguous best try
Caching attribute: Writethrough
Map area words in use: 11
Access mode: 0
File owner UIC: [1,1]
File protection: S:RWED, O:RWED, G:RE, W:
Back link file identification: (4,4,0)
Journal control flags:
Active recovery units: None
File entry linkcount: 0
Highest block written: 1503448
Client attributes: None

Identification area
File name: INDEXF.SYS;1
Revision number: 201
Creation date: 5-JUL-1995 08:38:46.23
Revision date: 9-DEC-2006 19:01:35.44
Expiration date:
Backup date:

Map area
Retrieval pointers
Count: 16 LBN: 0
Count: 8 LBN: 1032
Count: 8 LBN: 64484520
Count: 1503416 LBN: 62981104

Checksum: 28201

I'd like to set up a monitor to alert when an index gets close to being full.

Re: Space available in indexf.sys

labadie_1 — Tue, 12 Dec 2006 15:28:25 GMT

check the field map area words in use. When it is at 155, you will get the famous headerfull error when issuing any create/copy/... command.

By the way, dfu does a wonderfull job for giving you a valuable info on the remaining headers/files on the disk.

Re: Space available in indexf.sys

Hein van den Heuvel — Tue, 12 Dec 2006 16:02:47 GMT

That indexf.sys is in good shape.
It is hard to imaging you had a 'encountered file header full issues on this disk'.
It is only half full (EOF=850K, ALL=150K).
It has only one real mappin pointer: room for dozens more.

>> I don't see the ACL size or reserved
area size from that information.

Look again:
Map area offset: 100
Access control area offset: 255

Offset is 255 WORDS = 510 BYTES + CHECKSUM = no Access Control. Which is normal.

The Map area starts at byte 100*2=200 and is (255-100)*2 = 310 bytes long. Which is normal.

>> Can someone point me to where that info is, or show me a different method of determining how much free space I have available in the index?

Use the DFU FREEWARE.

>> encountered file header full issues on this disk while only 700,000 files were on

Just use DFU REPORT.

Look for " ***** Volume info "....
:
Maximum # files : 2221056
Header count : 382467
Free headers : 276840

Regards
Hein van den Heuvel
HvdH Performance Consulting

Re: Space available in indexf.sys

K5TTT — Tue, 12 Dec 2006 16:05:47 GMT

Bingo. Thanks much. I hadn't upgraded DFU in a while and was still running 2.6. I was getting some access violations when running it and didn't remember the report function. This gives me exactly what I need, thank you.

***** Volume info for ODS5 volume $1$DGA81: (from HOME block) *****
Volume name : ECP_DISK2
Volume owner :
Volume set name :
Highwater mark. / Erase on del. : No / No
Cluster size : 8
Maximum # files : 13981013
Header count : 842522
First header VBN : 3447
Free headers : 203924

***** File Statistics (from INDEXF.SYS) *****
INDEXF.SYS fragments/ map_in_use : 4 /11 words ( 7% used)
Total files (ODS2 / ODS5) : 31894 / 603843
Empty files : 1456
Files with allocation : 634281
Files with extension headers : 820
Files marked for delete : 0
Directory files : 125772
Contiguous files : 595188
Total used/ allocated size : 83073615 /88267280 blocks
Total headers/ fragments : 638493 /988823
Average fragments per file : 1.559
File fragmentation index : 1.658 (good)
Average size per fragment : 89 blocks
Most fragmented file :
$1$DGA81:[ECP.LIVE.RESPONSE.NEIC]LHP_9N000342.RSP_OLD;1 ( 178141/178144 bloc
ks; 7879 fragments)

***** Free space statistics (from BITMAP.SYS) *****
Total blocks on disk : 125829120
Total free blocks : 37561728
Percentage free (rounded) : 29
Total free extents : 106623
Largest free extent (blocks) : 2831688 at LBN: 58116792
Average extent size (blocks) : 352
Free space fragmentation index : 0.284 (excellent)

This looks good to me. I think the rebuild probably fixed our issue. Let me know if you see anything out of whack here. I can use this data to monitor/alert and that's what I needed.

Re: Space available in indexf.sys

John Gillings — Tue, 12 Dec 2006 16:22:24 GMT

>When it is at 155, you will get the famous headerfull error when issuing any create/copy/... command.

Correct, but HEADERFULL is largely historical now. The vast majority of disks initialized post V6.2 are more or less immune to HEADERFULL because the algorithm for extending the disk was significantly improved. It will generally extend INDEXF.SYS well beyond the size required to fill the available free space quite early in the life of the volume.

Up to V5.5-2 the extension algorithm was as simple as it could be: "1000 blocks", no matter the size of the volume or the number of files. On "large" disks (ie: anything over 1GB ;-) with many small files, this rapidly led to a fragmented index and filled the map area. The new algorithm considers the used space on the disk and the number of files. It then calculates the number of headers required to fill the remaining free space, and extends INDEXF.SYS by that amount, using "contiguous try a bit harder than the standard contiguous-best-try". You have to have a very odd and highly variable disk usage pattern to fragment the header badly enough to hit HEADERFULL.

In the early 90's this issue was probably THE most frequently logged case in customer support centres. Since V6.2 I've only seen ONE case where a customer claims to have experienced it on a post V6.2 volume (and I'm skeptical about that one...)

Bottom line is, make sure you initialize your volumes with a reasonable value for /HEADERS based on your expected usage (hint - the default value of 16 is NOT reasonable!). You can then forget about index map areas and concentrate on more important things in life. If you're really paranoid, check "Map area words in use" from DUMP/HEADER. As long as it's under 120 you're fine.

Challenge... I'd be surprised if anyone can post a map area size over 100 in a volume initialised post V6.2 (assuming you haven't subjected the disk to a pathological usage pattern)

Re: Space available in indexf.sys

K5TTT — Tue, 12 Dec 2006 16:39:04 GMT

I read something very similar in one of the wizard answers. Thanks for the info.

"assuming you haven't subjected the disk to a pathological usage pattern"

Our developers snuck some code in on me that has increased the subdirectories and files on this disk by a factor of 8 since the beginning of the year.

Re: Space available in indexf.sys

John Gillings — Tue, 12 Dec 2006 20:25:43 GMT

>>pathological usage pattern
>Our developers snuck some code in on me that has increased the subdirectories and files on this disk by a factor of 8 since the beginning of the year.

This is not the kind of thing I was thinking of. The extension algorithm shouldn't have any trouble dealing with more files. If you really wanted to break INDEXF.SYS, you'd need to do something really nasty. For example:

INIT disk with default settings
Open 3 files with minimum extend quantity.
Loop writing one record to each file until
disk is full.
You now have a disk with 3 (very large)files interleaved (ie worst possible case of fragmentation).
Delete one of the files.
Now start creating small files.
Since the free space is at maximum fragmentation, INDEXF.SYS can't get contiguous extensions, and will eventually fill up.

Obviously not a real world workload!