Very bad performance on disk with a lot of files and directories

[Rene Heeres]

On one of our servers we have a disk on which scanned documents are archived.

At the moment this disk contains:
- approx. 4.000.000 files
- approx. 25.000 directories

Every month aproximately 3000 files are added to this disk.

For about 3 months ago the daily incremental backup we ran of this disk increased from approx. 4 hours to approx. 12 hours at the moment.
The full backup of this disk that we make during the weekend takes about 17 hours.

When I do a directory on this disk it also takes forever, so I think the problem lies in the index file or the directory structure/files itself. The fragmentation can't be the problem, DFU says that the fragmentation level is excellent (fragmentation level 0.000).

I tried finding the reason for this, but I cannot determine what is causing this slow performance. Things I looked for are:
- large directory files --> > 127 blocks
- errors in the disk-structure
- fragmentation

None of these items seemed to be a problem.
I also tried to mount the disk with it's own XQP cache (PROCESSOR=UNIQUE), but this also didn't help.

The index file is quite big ( approx. 7.5 mill. blocks), could this be a problem?

Can anybody give me a clue on how to get the backup back to normal proportions?
Is converting this disk to ODS5 an option?

[Volker Halle]

Rene,

before trying to speculate what 'might be wrong', could you provide some more detailled data ?

When you say 'DIR takes forever', did you collect some data with MONITOR FILE,FCP,DISK ? Maybe you can collect that data from a directory operation and provide the output screens as an attached .TXT file ?

Volker.

[Wim Van den Wyngaert]

If you have a lot of time :
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=625667

If not : the directory may get locked when the directory file is extended. If you create a lot of files, this can be the case. And the lock takes longer as your directory grows because the directory file is re-created.

Wim

[Hein van den Heuvel]

4 Million files is lot, You have gone beyond where most systems operate.
Until not too long you could not create more than 1 M file, as the allocation bitmap was restricted to 1M bits. We can now exceed that, but you may well need special tuning, both for the SYSGEN ACP params and for teh BACKUP quotas.

How do the FCP stats look like?
Did you try AUTOGEN FEEDBACK and all that?
What are the SYSGE/ACP settings? Did you attempt to tweak them?

Are the performance problem most notable under concurrent load (not enough spindles behind the files), or also for single stream access. Backup is a 'funny' single stream as it does async IO. You may need to DE-tune it to limit it some. Think hundred IOs, not thousands as we used to tune the backup process.

Met vriendelijke groetjes,

Hein van den Heuvel
HdvH Performance Consulting.

[Robert Gezelter]

Rene,

I concur with my colleagues. More detail about what is happening on the system while the BACKUP is running would be very helpful. The increase in backup duration is merely the alarm that something undesired is happening.

- Bob Gezelter, http://www.rlgsc.com

[Wim Van den Wyngaert]

Also report your VMS version(s).
If dir extention is the problem create/dir/alloc can be your hero.

Wim

[Andy Bustamante]

A few more questions.

Besides the version of OpenVMS, is XFC enabled? If so, how much memory is in use? What is the hardware behind this disk? What other I/O is being handled on this device?

Andy

[Rene Heeres]

I have enclosed a file with the output of:
- monitor disk
- monitor file
- monitor fcp

I gave the following command:
directory icr$disk:[000000...] /size=all/gra

after that I started the monitors for about 20 minutes. The disk with the problems is ICR$DISK ($1$dga306:) with label ICR.

We are using openVMS 7.3-2 with XFC enabled (current size 3,5GB)

I will send some more monitor info that I will collect during the backup that runs tonight.

[Jan van den Ende]

Rene,

not a solution to your problem, but an idea for a way around it. It may, or may not, be suitable for your situation.

You _CAN_ create a LD drive as one big file a disk. Then you MOUNT the LD drive like you would any other drive. Use it just as you would any other drive.
But, to BACKUP, it is only ONE BIG file, and the performance of such backup is as any other big file.
One drawback: you can only restore the ENTIRE container.

hth

Proost.

Have one on me.

jpe

[Volker Halle]

Rene,

the 'problem' are the XQP IOs reading the file headers. The file header cache won't work 'well' in this case, the attempt rate is about 200 attempts/sec, hit rate 10%, which ends up in about 180 IOs/sec (synchronous) going to INDEXF.SYS on this disk. Increasing the cache won't do any good, as the DIR command won't ever re-use the file headers read into the cache.

At this IO rate, it would take about 6 hours to read all 4,000,000 file headers.

Volker.

[Volker Halle]

Rene,

did you ever check, that you are not 'overloading' your EVA/HSG80 ?

$ ANAL/SYS
SDA> FC STDT/ALL
SDA> EXIT

Check for non-zero values in 'QF seen' and/or 'Seq TMO' colums.

Volker.

[Rene Heeres]

Volker,

You are right that a directory-listing of all files takes about 6 hours. Is there any way to speed things up?

The storage system that is behind this disk is HSG80 based. The disk is based on a RAID5-set.

The thing that I cannot understand is that this is becoming a problem at this time. We have been using this disk for years now and there has never been a problem.

We also have another disk with more than 5.000.000 files and this disk disk doesn't have this problem at all. This disk is also a RAID5-set on HSG80 based storage system. A difference on this disk is the amount of directory-files, on this other disk this is 'only' about 2300.

[Volker Halle]

Rene,

now that you've looked at the MONITOR data for your 'problem' disk, you could easily compare that with the data on your 'well behaving' disk. What kind of IO-rate (DISK IO or FCP Disk Read Rate or File ID attempt rate) do you get when executing the same DIR command on that disk ?

The no. and size of the directories on the disk does not have any influence on that DIR [*...]/SIZ=ALL kind of operation. Every file hdr of every file needs to be read into memory and processed. For file lookups based on filenames, this may be different as the directory size comes into play.

Is those 2 disks on the same cluster/HSG80 ?

Volker.

[Hein van den Heuvel]

It might be the order of the headers, which you can check by doign a (quick!) DIR/FIL.

If the other disk was created more 'nicely' then the IO controller may recognize a serial access pattern adn perform read aheads. The VMS File header IOs would then be resolved in the Controller cache and would not get seek/rotate delays.

The 5M file disk may be the 'odd one out'.
For now it sounds like the 4M file drive is pretty much behaving as expected.

You can speed it up by doing a DIR/FIL, SORT by FILE IS and then roll your own tool to get the desired data driven by that sorted list. Not too usefull a suggestion huh?

The right way to solve this problem is DFU from our Dutch friend Ton Dorland!

Good luck,
Hein.

[Thomas Ritter]

Rene, I wish I could add to what the other experts have written. We also deal with big directories where we have millions of files.
Our volumes were initialized with that expected usage. For example

$ init/directories=16000/headers='big_number'

Would you post the results of

$dir/full 'device'

and

$ dump/header/block=count=0 'device'[000000]indexf.sys


Thanks

[Volker Halle]

Thomas,


$ init/directories=16000/...


The /DIRECTORIES=n qualifier just specifies the initial allocation size of the MFD = 000000.DIR. It does not help anything with any other directories on the disk.

If you expect other directories to get quite large, you can pre-allocate space with CRE/DIR/ALLOC=n disk:[dir]

Volker.

[Volker Halle]

Hein,

I was thinking about the same lines...

Reading each file header (from INDEXF.SYS) is done with single-block synchronous IOs. Not the fastest method to get data from a disk. No kind of caching inside OpenVMS helps in this situation.

If the files in the directories have been created in a way, that their FILE IDs are largely different, non-adjacent blocks must be read from INDEXF.SYS and even read-ahead caching in the controller can't help.


The right way to solve this problem is DFU from our Dutch friend Ton Dorland!


Now I wonder, how DFU would be able to change this ?!

Volker.

[Jan van den Ende]

Volker reated to Hein:



The right way to solve this problem is DFU from our Dutch friend Ton Dorland!


Now I wonder, how DFU would be able to change this ?!


Well, _I_ would have thought (as described above) that LD would be a (the?) solution.
Also by a Dutchie: Jur van den Burg.

Zoals je ziet, Rene, best wel Hollandse activiteit in VMS-land!

Proost.

Have one on me.

jpe

[Ian Miller.]

parhaps Hein was referring to DFU DIRECTORY/COMPRESS ?

[Volker Halle]

There are 2 different symptoms of maybe the same problem here:

$ DIR disk:[*...]/SIZE=ALL takes a very long time (about 6 hours).

$ BACKUP disk: takes an even longer time

The MONITOR data captured during the DIR operation shows, that the 180 IOs/sec on this disk are spent reading the file headers. If the FILE IDs of the file entries in a directory would be adjacent, i.e. FIDs 201, 202, 203,... the file headers would also be adjacent in INDEXF.SYS. But if the FIDs of file entries in a directory would be: 201, 699, 5499 etc., even the read-ahead cache of the HSG80 would not help.

DFU could compress a .DIR file, but it can't shuffle file-ids and file headers in INDEXF.SYS around...

Volker.

[Robert Gezelter]

Rene,

Re Volker's comment. Do a DIRECTORY/FILE_ID on some of these directories and see if his observation is correct.

One of the more general limitations of caches (at all levels) is that they dramatically increase performance in many BUT NOT ALL (emphasis mine) cases. In particular, caches are very effective at improving sequential operations, in either direction.

- Bob Gezelter, http://www.rlgsc.com

[Jan van den Ende]

Rene,

in case you are still interested, LDdriver and info are to be found at
http://www.digiater.nl/lddriver

Proost.

Have one on me.

jpe

[Wim Van den Wyngaert]

In the monitor statistics I see that there are 5 drives working harder than the ICR drive. These statistics don't prove much.

If I do the same dir on a disk on a GS160 with 91K files, it takes about 5 minutes (2000 blocks header cache) and cache hit rate is 10% too. So for 4M this wouild be about 200 min or more than 3 hours.

Wim

[Hein van den Heuvel]

Volker> "Now I wonder, how DFU would be able to change this ?!"

DFU can only solve one problem Rene mentions:
" gave the following command:
directory icr$disk:[000000...] /size=all/gra"

It solves that by reading INDEXF.SYS in large chunks.

It can not solve the backup problem.

Hein.

[Wim Van den Wyngaert]

One thing : on my GS160 the cpu tick rate (% cpu used for file operations) is 3.6% for 340 reads per sec. In your case it's 85% for 180 IOs.

Wim