File attributes: Allocation: 5000 - VMS 7.3-2

[Jorge Cocomess]

Hi,

Does anyone here can tell me what's the meaning of the File Attributes Allocation: 5000??

Here's are the info when I typed DIR/FULL on a directory:

File attributes: Allocation: 5000, Extend: 0, Global buffer count: 0
No default version limit, Contiguous, Directory file

Total of 1 file, 93/5000 blocks.


Am I capped at 5000 VMS blocks or something like that?? If yes, how can I change this to be unlimited??

Thank you in advance.

Jorge

[Hein van den Heuvel]

Jorge,

It just means that the particular fiel CURRENTLY has 5000 block allocateed to it.

You can 'see' exactly which physical blocks it have using DUMP/HEAD/BLOCK=COUNT=0 for the file.

Is does not mean the file can not grow if and when it needs to.

The growth with constraing first by
- disk quota (if enabled)
- free blocks on the disk (in general)
- largest free contiguous chunk on the disk in this particular case where the file is currently contiguous and must stay that way, because it is a directory file with special constraint.

Why are you asking?
Just for clarification?
Do you think there is a problem?
An error in an application?

hth,
Hein.

[Robert Gezelter]

Jorge,

The file presently has 5000 blocks allocated. In addition to the questions that Hein has asked previously, I would add the question of how was the file created? If there is an FDL file used in the file creation process, then it is likely that there is an explicit allocation in that file (and also a request to make the file CONTIGUOUS).

- Bob Gezelter, http://www.rlgsc.com

[Hein van den Heuvel]

Bob, fwiw... It's a directory!
File attributes: ... Directory file

Surely no FDL
It probably simply had 10,000+ files in it at some point in time and now there are only a few hundred left.

Hein.
[0 points for this please]

[Robert Gezelter]

Hein,

Mea culpa! I was tired and missed the "Directory" file when I read his post.

In that case, while it would seem the allocation was driven by growth, however, there remain two possibilities:

- the use of the CREATE/DIRECTORY/ALLOCATION=nnn command (beginning with the last few releases).

- the underlying granularity of the volume

Perhaps Jorge can shed some light on how this directory was created and used?

- Bob Gezelter, http://www.rlgsc.com

[Jorge Cocomess]

Hi,

The directory was mainly for writing application logs into this directory, whic mostly .txt file. Every so often, we have to clear out the directory after couple of weeks or about 1500 or so files, because the application was not able to write to this directory. This directory does not have disk space constraint or any constraints as I know of. Below, is the directory characteristic "DIR/FULL" command.

Directory DISK56:[000000]

DDR_STATS.DIR;1 File ID: (666,3,0)
Size: 93/5000 Owner: [SYSTEM]
Created: 3-NOV-2005 12:32:04.34
Revised: 21-JAN-2008 17:07:56.83 (47812)
Expires:
Backup:
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 5000, Extend: 0, Global buffer count: 0
No default version limit, Contiguous, Directory file
Record format: Variable length, maximum 512 bytes, longest 512 bytes
Record attributes: No carriage control, Non-spanned
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RWED, World:RWED
Access Cntrl List: None
Client attributes: None

Total of 1 file, 93/5000 blocks.

[Karl Rohwedder]

Jorge,

if the applicatin fails to write to the directory, this may be caused by:
- the disk is full
- diskquota enabled and no more left
- the index file of the disk is full and no new file headers can be created

The latter may be cured by initializing a new disk and doing a BACKUP/IMAGE/NOINIT to it.

Btw, the DFU utility is able to compress/truncate directories.

regards Kalle

[Hein van den Heuvel]

>> Every so often, we have to clear out the directory after couple of weeks or about 1500 or so files, because the application was not able to write to this directory.

What does 'not able to write mean'?
Privillege violation?
Allocation failure?
Not quick enough?

The most likely reason for trouble here is really fragmented free space on the drive.
Check with DFU REPORT or DFO.

And uh... yo really rally want to maintain your directories irrespective of this problems. Above 1000 blocks or so, the cost of random inserts and deletes becomes significant.

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Robert Gezelter]

Jorge,

I must concur with Hein's most recent comment. It is impressive how sparse a directory can become with an ongoing chain of creations and deletions. When I encounter problems with this at clients, it is often mail directories or directories that are used as transient storage locations.

Unless you are particularly short of disk space, I would be inclined to leave the file as it is. Directories must be contiguous, and if there is a shortage of contiguous space on that disk, one can later regret shortening the directory.

If the directory appears to be full, there are several ways to address the problem.

Certainly, as Karl has mentioned, DFU can reorganize a directory. If you are not familiar with DFU, or it is not installed, there is a simple solution that requires nothing more than straightforward DCL.

An example. Directory [AAA] shows that it is almost completely full, and it is reported that some attempts to add files to the directory are failing. Without DFU, the following will reorganize the directory [admitttedly it does require some space]:

$ CREATE/DIRECTORY [BBB]
$ RENAME [AAA]*.*;* [BBB]*.*;*
$ RENAME [BBB]*.*;* [AAA]*.*;*
$ DELETE [000000]BBB.DIR;*

If I need to do this repeatedly, I keep around the scratch directory for future use. DFU would be nicer, but in some situations, it is not available.

- Bob Gezelter, http://www.rlgsc.com

[Willem Grooters]

In addition to both Bob and Hein:

Directory files need contigous space. If a directory is full, it will be expanded (by the disk's clustersize?) and stored on a location that can hold the whole file. If there is no=t enough contigoeus space on disk, you'll get an error and the file you added will either be 'invisible' by normal means (the directory file is not chnaged so the file won't show up) or it may have a size of 0. I've seen both.

Using that amount of files in one directory is (used to be?) a not-so-good idea.

Though you have 93 blocks actually used, 5000 blocks are allocated. It could mean several things, but it all boils down the the disk's clustersize. If it has been set to 5000, creation of the directory file will allocate 5000. Each extension will add 5000 bytes. If your clustersize is 2500, it might have been the file was filled up and a newly added file would cause extension - that's where 5000 (2*2500) comnes from.

Keep in mind that restoring files from backup, the original allocation will be used, rounding up to local clustersize:

[Robert Gezelter]

Willem,

"... creation of the directory file will allocate 5000. Each extension will add 5000 bytes." should have been "blocks".

I am sure that it is a typographical error, and I merely point it out to keep the thread correct.

- Bob Gezelter, http://www.rlgsc.com

[Jorge Cocomess]

Hi,

The strange thing is that the application doesn't really give you a whole lot of error messages, it just errored out. Once the application or sub-routines errored out, I go to this directory and purge out as much as I can then I can resume my processing again.

The disk was built on 2 x 72GB Mirrored. Currently, I have about 60GB available on this disk.

We do have license for DFU -- However, this disk being accessed 24 x 7, would it make more difficult for DFU utility to run since it's being accessed 24 x 7??

I could create a new LUN and move everything over to it. Would you gentlemen think this would be wise to do so??

Any ideas what else I can do??

Thank you.

Jorge

[Robert Gezelter]

Jorge,

A new LUN will not resolve the problem. The problem is how the information within the volume is being used.

The problem can be addressed in a variety of ways. The overall volume being active is not a problem, although it does complicate things. Certainly, re-organizing this directory can be done with the only requirement being that this directory is quiesced (and I can think of a few ways around that, but they DO depend on the details).

It may be good idea to retain more senior expertise to examine this problem and identify a solution [Disclosure: We do provide consulting assistance in this type of matter].

- Bob Gezelter, http://www.rlgsc.com

[Hein van den Heuvel]

>> The strange thing is that the application doesn't really give you a whole lot of error messages, it just errored out.

Too bad. No log with a real error huh?

>> Once the application or sub-routines errored out, I go to this directory and purge out as much as I can then I can resume my processing again.

1) And when it errors out, there is still free space right?

2) There wouldn't be file with version number approaching 32K?
$mcr dfu searc/vers=mini=30000 dev:

3) I suppose there could be a (very unlikely) directory corruption. Create a fresh directory, rename or set file/enter all files into it. Rename and blow away old directory. Rename fresh directory to correct name.

4) you wouldn't happen to be out of file headers, adn this directory is just a red-herring... any file create would fail:

$MCR DFU REPORT dev:
"Maximum # files " vs "Header count"
vs "Free headers"

>> The disk was built on 2 x 72GB Mirrored. Currently, I have about 60GB available on this disk.

So that's now, after cleaning.
What about before?
It's hard to imagine there would not be a free chunk of a few MB.

MCR DFU REPORT dev:
***** Free space statistics (from BITMAP.SYS) *****
"Largest free extent"

>> We do have license for DFU -- However, this disk being accessed 24 x 7, would it make more difficult for DFU utility to run since it's being accessed 24 x 7??

DFU needs no license.

DFO is usually run while the system is in use.

>> I could create a new LUN and move everything over to it. Would you gentlemen think this would be wise to do so??

Not if you do not rally know what is wrong.

Hein.

[John Gillings]

Jorge,

>The strange thing is that the application
>doesn't really give you a whole lot of
>error messages, it just errored out.

If you can't fix the application to issue proper error messages, then next time this happens, instead of purging the directory try some diagnosis first. Check the used size of the directory. Is it full? Try creating some files from DCL:

$ CREATE dev:[yourdir]X.X

If that works, try different file names, perhaps lexically less than the first file in the directory, lexically after the last, and some spread around in the file. See if you can get a sensible error message from CREATE.

Look for files with abnormally large numbers of versions, or abnormally high version numbers.

Try creating another directory on the same disk, and filling it up with files. How many files with unique names can you create before you get a DIRALLOC error? How big is the directory at that point? (use DFU to clear out the directory otherwise it will probably take a week!)

My guess would be DIRALLOC - the disk is too fragmented to extend the directory beyond its 5000 block allocation.

Long term the only solution is to defrag the disk. Short term you can purge the directory, or create multiple directories and link them together with a search list. Put the most empty directory up the front of the list, that's where the new files will be created, but the application will still see existing files in directories further down the list.

DFU DIRECTORY/COMPRESS might help, but consider that a 5000 block directory doesn't perform terribly well, especially for DELETE.

[Art Wiens]

Seeing as how that's a nice neat "5000 blocks", could that be the default cluster size of the disk? If multiple files are being created at the same time, and if they all go for another extent at the same time, there might not be enough free space, yet when you check, there still seems to be some.

Just a thought,
Art

[Jorge Cocomess]

Since the time I posted this question, the file size is now "Total of 1 file, 281/5000 blocks". It translate to 1972 files under this directory.

So, once it hit 5000 mark, I should see some issues, right??

Thanks and everyone. If you're wondering, I will take care of the points shortly.

Jorge

[Hein van den Heuvel]

>> So, once it hit 5000 mark, I should see some issues, right??

Well, yes, no, and maybe.

As you work your way up to the 5000, more and more file apparently are created, each needing 1 (at least) file header, (you can run out of those) and 1 (at least) allocation cluster (you can run out of those).

When you cross the 5000 mark, and more room is needed then the file system (F11X/SHFDIR) will allocate a new, larger, contiguous, file and copy all blocks over.

Obviously that will cost some energy, but it might not be an 'issue'. That particular issue is only marginally worse from adding a directory entry with a low alphabetical value into a full directory block in which case the same IOs occur, but just within a single file.

It becomes a real (DIRALLOC) issue if at that point not enough contiguous free space is available.

For your education & entertainment you may want to try $DUMP/DIRECTORY/BLOCK=(START=x,COUNT=1) x.DIR
Compare with $DUMP/RECORD=(START=y,COUNT=z) x.DIR (look at the RFA's!)

Cheers,
Hein.

[John Gillings · Accepted solution]

Jorge,

>> So, once it hit 5000 mark, I should see some issues, right??

You will potentially see issues long before you get to 5000 blocks (though you shouldn't have any trouble *creating* files).

Remember that on a Files-11 structured disk, directories are a kind of illusion layed over the top of the file system, to help HUMANS find files. Since most humans have trouble dealing with thousands of files in one big chunk, the directory mechanism was never designed to cope with huge numbers of files.

It wasn't very long ago (V4?) when directory files were hard limited to 128 blocks. Although V5 increased the limit, it wasn't until very recently (V7?) that the SEVERE performance knee at 128 blocks was raised. Despite that, for most practical purposes it's advisable to keep your directory files below 1000 blocks.

What problems are you likely observe? As directory files get bigger, searching for, and inserting files in the directory is likely to get significantly slower, as the search mechanism requires more and more sequential scanning to find stuff in the directory.

However, the BIG hitter for preformance is DELETE. Consider DELETE *.*;* for a directory less than 128 blocks this will be reasonably fast. Maybe a few minutes, but as the directory file size increases, the time increases geometrically, as does the CPU and I/O load. Once a directory is over 1000 blocks, the command can literally take DAYS to complete. Although DFU can solve this for deleting the ENTIRE directory, it won't help for mass deletions of some of the files, say "DELETE A*.DAT;*"

Some might say this is a flaw in the way directories are implemented, but it's more a reflection on the changes in the way directories are being used. Files-11 directories are fine for their intended purpose, but they don't scale up very well.

If you have large numbers of files to store, you may want to consider alternatives to a single directory. Perhaps create some substructure, or use search lists to spread the files across multuple directories (or even multiple devices).