Disk Initialization Parameters for good performance

[sartur]

We have a OpenVMS CLUSTER ( V8.3 ) with disk storage in a EVA8000.
A disk RMS sometime gives the message SYSTEM-F-HEADERFULL
It's a 344GB disk (vraid 1 in EVA) having 451 directories and 736,000 files.
We proceeded to create a new disk and perform disk to disk backup
The new disc was initialized with the following characteristics

initialize /cluster_size=18/directories=1000/header=1500000/nohighwater/maximun_files=1500000/index=end $1$dga75: PAN_RMS2

Initialization parameters are adequate for good performance?

[Hein van den Heuvel]

What to you typically do with those files?
Is there a DB product involved? Simple sequential or indexed?

Are they all in the 1/2 MB (1000 block) space) or some large and many small?

I like to make my clustersizes a power of 2 or a multiple of 16.
That improves the odds that rms (indexed file) buckets are aligned with XFC cache lines, and it some flavors of the EVA firmware liked it. But if I recall correctly that was mostly for Raid-5 full-stripe writes which are not in play here. That, and it makes fo easy math! :-)

/DIRETORIES is irrelevant
/HEADER as shown waste 1/2 GB
/INDEX=END 'suggests' maximum seeks for maximum time. The EVA will outsmart you by actually giving it chucks early on the disks, because INDEXF.SYS is actaull initialized and touched earlier. Still, I prefer /INDEX=BEG or the default of middle for historical reasons.
/NOHIGH if the environment/security can stand it is best for speed during certain accessed/creates.
/MAXIMUM should be big enough

You may want to add

/EXTEN=1024 for nice pre-allocation during file create/extend

/LIMIT for expansion

And you may want to start put with /STRUCT=5 but that, not /LIMIT, has direct performance implications

hth,
Hein

[Jon Pinkley]

sartur,

You have a 344 GB disk and you are pre-allocating all the headers you will ever allow? My advice is to let maximum_files be as high as possible, it doesn't cost much (1/4096th of a block for each possible file, it is controlling a bitmap that represents file headers). Not having it large enough means you will have to reinitialize the disk to extend the bitmap.

I would recommend either cluster of 16 instead of 18 (but this is because I like cluster sizes that are a power of 2). VMS folklore says it will be more efficient with 16 block clusters, but I have never seen any convincing evidence. The EVA cache likes 8KB (16 block) transfers, but the cluster size has only indirect affects on I/O transfer sized. But a cluster size of 16 won't be worse than 18, unless you have most of your files that are a multiple of 18 blocks in length.

I would recommend

/limit ! allow expansion and maximize /maximum_files (this is an EVA, and it is easy to expand a vdisk if you need to)
/maximum_files ! I would use /limit and leave this qualifier off. Then it will be 16 million.
/cluster_size=16 ! this will be the default if you use /limit
/directories=16000 ! this just pre-allocates some space (1000 blocks) to [000000]000000.dir if you want less, use something less. (n/16 will be # of blocks) Nearly irrelavent.

Other things can be controlled at mount time (like /window size and /extension). Be aware that if you have poorly behaved programs that reopen files for every write, having a large /extension can have a detrimental performance effect, as the file gets extended and possibly truncated when closed.

Jon

[John Gillings]

sartur,

I'd go with what Hein suggested for qualifiers, especially avoid /INDEX=END. Without knowing the average size of files, it's difficult to say what the cluster size should be. If the disk is nearly full, the files would average around 500KB, so the cluster size should probably be more than 18, maybe 128?. That may be too extreme, if there are many smaller files, but, in general, on a disk that size, I'd go for something much bigger than 18.

I'm a bit more interested in your HEADERFULL - I can't see how you can get an INDEXF.SYS HEADERFULL on the disk you describe. If you still have the faulty disk, could you post the output of:

$ SHOW DEVICE/FULL disk
$ DUMP/HEADER/BLOCK=COUNT:0 disk:[000000]INDEXF.SYS

It might also help if you describe how the files are created - maybe there a a large number of files being created in parallel, with small cluster and extend sizes, then gradually deleted in random order, leaving a highly fragmented disk?

[sartur]

John

Currently the disk is not giving HEADERFULL, that was months ago, which is why we proceeded to initialize it this way.

My question now is whether the initialization in this manner could be negatively affecting the performance, because currently we are having some slowness.

SHOW DEVICE/FULL $1$dga6

Disk $1$DGA6: (PANA00), device type HSV210, is online, mounted, file-oriented
device, shareable, available to cluster, device has multiple I/O paths,
error logging is enabled.

Error count 0 Operations completed 7472577
Owner process "" Owner UIC [SYSTEM]
Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G:R,W
Reference count 323 Default buffer size 512
Current preferred CPU Id 3 Fastpath 1
WWID 01000010:6005-08B4-0009-2478-0000-A000-0017-0000
Total blocks 671088640 Sectors per track 128
Total cylinders 40960 Tracks per cylinder 128
Logical Volume Size 671088640 Expansion Size Limit 671440896
Allocation class 1

Volume label "PAN_RMS2" Relative volume number 0
Cluster size 18 Transaction count 320
Free blocks 275076432 Maximum files allowed 1500005
Extend quantity 5 Mount count 2
Mount status System Cache name "_$1$DGA50:XQPCACHE"
Extent cache size 64 Maximum blocks in extent cache 27507643
File ID cache size 64 Blocks in extent cache 1343574
Quota cache size 0 Maximum buffers in FCP cache 8346
Volume owner UIC [SYSTEM] Vol Prot S:RWCD,O:RWCD,G:RWCD,W:RWCD

Volume Status: ODS-2, subject to mount verification, write-back caching
enabled.
Volume is also mounted on PANA06.

I/O paths to device 4

Path PGA0.5000-1FE1-5004-238C (PANA00), primary
Error count 0 Operations completed 31
Last switched to time: Never Count 0
Last switched from time: 23-MAY-2010 09:00:41.04

Path PGA0.5000-1FE1-5004-2388 (PANA00)
Error count 0 Operations completed 31
Last switched to time: 23-MAY-2010 09:00:41.04 Count 1
Last switched from time: 23-MAY-2010 09:00:53.17

Path PGB0.5000-1FE1-5004-2389 (PANA00)
Error count 0 Operations completed 31
Last switched to time: Never Count 0
Last switched from time: Never

Path PGB0.5000-1FE1-5004-238D (PANA00), current
Error count 0 Operations completed 7472484
Last switched to time: 23-MAY-2010 09:00:53.17 Count 1
Last switched from time: Never

PANA00=>

DUMP/HEADER/BLOCK=COUNT:0 $1$dga6:[000000]INDEXF.SYS









Dump of file $1$DGA6:[000000]INDEXF.SYS;1 on 24-MAY-2010 16:40:09.38
File ID (1,1,0) End of file block 788275 / Allocated 1500444

File Header

Header area
Identification area offset: 40
Map area offset: 100
Access control area offset: 255
Reserved area offset: 255
Extension segment number: 0
Structure level and version: 2, 1
File identification: (1,1,0)
Extension file identification: (0,0,0)
VAX-11 RMS attributes
Record type: Fixed
File organization: Sequential
Record attributes:
Record size: 512
Highest block: 1500444
End of file block: 788276
End of file byte: 0
Bucket size: 0
Fixed control area size: 0
Maximum record size: 512
Default extension size: 0
Global buffer count: 0
Directory version limit: 0
File characteristics: Contiguous best try
Caching attribute: Writethrough
Map area words in use: 11
Access mode: 0
File owner UIC: [SYSTEM]
File protection: S:RWE, O:RWE, G:RWE, W:RWE
Back link file identification: (4,4,0)
Journal control flags:
Active recovery units: None
File entry linkcount: 0
Highest block written: 1500444
Client attributes: None

Identification area
File name: INDEXF.SYS;1
Revision number: 4580
Creation date: 19-AUG-2004 19:44:08.00
Revision date: 24-MAY-2010 10:14:56.98
Expiration date:
Backup date: 15-MAY-2010 04:18:50.51

Map area
Retrieval pointers
Count: 36 LBN: 0
Count: 18 LBN: 1026
Count: 18 LBN: 669579138
Count: 1500372 LBN: 669588264

Checksum: 5697
PANA00=>

[Jon Pinkley]

sartur,

In short, I don't see anything in your init qualifiers themselves would be ausing "poor" performance.

What is the indication of poor performance? Is it getting worse slowly or did something change abruptly?

Do you have DFU installed on your system?

If so, can you do

$ def/job dfu$nosmg 1
$ dfu report $1$dga75: /graph

Also, if you have T4 loaded and collecting info, you may be able to see trends.

what does "$ monitor fcp,io" show for split io and window turn rate? Non-zero values indicate fragmention that is affecting performance.

What about file open rate?

Disk initialization parameters are pretty low on the list of performance affecting factors (in my opinion).

Also, I don't think that /index=end is going to make a noticable difference compared to /index=beg or the default /index=middle.
This is because you are using an EVA, which is going to spread the i/o over all the disks in a disk group, and you really have no control over what portion of the disk a particular LBN will land. Disk allocation on the EVA is similar to virtual memory, the disk allocation is split into psegs which correspond to "pages", and you don't have control of what psegs are allocated to your vdisk.

Jon

[Jon Pinkley]

I see your last comment was using $1$DGA6 instead of $1$dga75

So for DFU report use whatever disk you are interested in.

DFU report will give some fragmentation statistics, but they are not as extensive as what DFG (DFO defrag) will display. You can install and use the reporting features of DFG without having a license PAK istalled. It is worth installing DFG if you like the graphs it can provide.

Jon

[Jon Pinkley]

sartur,

Is the "slowness" only on the $1$DGA6 device or is everything slow?

The slowness could be related to other activity on the EVA that is using the same disk group.

Also, since this is a Cluster, what does monitor cluster show? If there is high locking activity, you may want to use monitor DLOCK, as there may be a file that is being modified on multiple nodes and causing remote locking (indicated by incoming and outgoing in the monitor dlock output).

In short, you need to determine what the real cause of the slowness is before you can determine how to fix it.

Jon

[John Gillings]

sartur,
Your output is $1$DGA6, is that the disk you (apparently) got HEADERFULL from?

There's definitely nothing wrong with INDEXF.SYS on that disk:

Map area words in use: 11
...
Map area
Retrieval pointers
Count: 36 LBN: 0
Count: 18 LBN: 1026
Count: 18 LBN: 669579138
Count: 1500372 LBN: 669588264

This is as contiguous as it can ever be. It has not, and I'd guess never will hit HEADERFULL.

"because currently we are having some slowness."

Relative to what? Can you describe the operation you believe to be "slow" and what you're comparing it with?

Your cluster size is a bit small, and in theory, /INDEX=END is not a great idea, but I'd have thought you'd have to be hammering the disk VERY hard to notice any difference.

[John Gillings]

Oh, and just a general point about the assumption implied in your title:

"Disk Initialization Parameters for good performance"

If there were a single set of qualifiers (parameters) which always gave good performance, they would be hard coded, not variable.

The best settings for your disk depend on how the disk is used. For example, a disk which contains a single large file would need very different settings from one containing large numbers of small files. You also need to consider if files are created once and stay forever, or if there's a high turnover, if all the files are about the same size, or if there's a mix of very large and very small files, are they written one at a time, or many in parallel, extended over a long period of time, if they're single spindles vs RAID sets, direct attach vs SAN etc... Lots of variables. You've given us some information, but not much.

Rather than present us with a possible cause and ask if it might be responsible for a nebulous symptom, please give us more hard detail of the symptom you're interested in and ask what might be possible causes.