Very bad performance on disk with a lot of files and directories

[Volker Halle]

Wim,

this high CPU tick rate seems to indicate, that maybe even the CPU has become the bottleneck ! There is also a quite substantial Window Turn rate (about 12/sec).

If we assume nothing else was going on on the ICR disk, a file system read rate of 180 and an IO rate of 176 on that disk seem to match pretty well.

Volker.

[Wim Van den Wyngaert]

Volker,

And where are the about 2000 IO's to the other disks ? Solved by xfc ?

Wim

[Volker Halle]

Wim,

MONI FCP only shows IOs issued by the File System (XQP). All the other IOs on all the other disks will be application IOs not involving the XQP.

Volker.

[Rene Heeres]

Thank you all for your replies.

I have tried to put all the information you have asked for in the attached textfile.

Both disks are on the same system. They are howeven on 2 different HSG's

We initially initialized this disk in a way that it could accomodate a whole lot of files.

init/nohigh /block=/headers=/maximum_files=

The degree of changes on this disk is not very great, about 2000-3000 files are added monthly.

I don't think LD would help, when using LD, I have to make a full backup of the whole every day, this also takes a lot of time (size=300GB). We also thought about making a physical backup of this disk, but that has the same drawbacks.

I however am willing to try both options.

When I look at the File-ID's, they seem quite sequential.

It is true that DFU is much faster in reading the directories, that is because DFU accesses the directories in a intelligenter way. This doesn't my solve my backup problem, which was the reason I started this thread.

I am still a bith lost in solving this problem. Is there any solution to this problem. Could ODS-5 be an option.

I also have the option to migrate this disk to an EVA5000 storage system. It is nog a problem to do this, but I would like to be sure that things will get better doing that.

[Jan van den Ende]

Rene,

Backup 300Gb in one (or some small numer) of files (and LD containers ARE just 1 file) should be done at about the max speed of your driev (tape or disk - the slower is determining).
That should beat the MANY,MANY, directory and header Lookups by a considerable factor!

hth

Proost.

Have one on me.

jpe

[Volker Halle]

Rene,

please consider to look at the STDT data for QF Seen and Seq Tmo (SDA commands given above) to rule out any HSG80 overload problems.

If your 2 big disks are so similar, then the achieveable IO rates should also be similar, which apparently is not the case. If you run a DIR/SIZ=ALL/GRA disk:[*...] on your 'well performing' disk, what kind of disk IO rate or File System Read rate and CPU Tick Rate do you get ? On the ICR disk, you seem to even saturate the CPU (FCP CPU Tick Rate = 86%).

Can you look at old accounting data/batch log files for your backup of the ICR disk. As the elapsed time has increased, did Direct IOs and CPU time also increase at the same rate ?

Another possible test would be to use a disk performance test utility (like DISKBLOCK or DISKPERF) to check the achievable single block synchronous read IO rate on those 2 disks in your given configuration.

The AUTOGEN report shown seems to indicate an unusal situation regarding feedback data for ACP_HDRCACHE (Too many ACP hits and attempts have occurred).

Do not bother to try to convert the disk to ODS-5 or copy to the EVA, before you understand the underlying problem.

Volker.

[Wim Van den Wyngaert]

"For about 3 months ago the daily incremental backup we ran of this disk increased from approx. 4 hours to approx. 12 hours at the moment."

This worries me. There must be something wrong. Do you know how many files there were when it took 4 hours ?

Wim

[Andy Bustamante]

From the let's touch all bases department: Please check the HSG-80 and make sure the RAID-5 unit has all members present. Has there been any change in the physical disks?

Andy

[Martin Hoogenboom]

Hi Rene,

Here's my â ¬ 0,02:

I assume you spoke to Menco, who signaled a similar performance decrease about a year ago. I don't recall if it was the same disk, but i'm sure it was the same system.
He was messing with DFU and compressing directories. IIRC solved the problem.

Some other questions pop to mind. I assume you've already concidered them, but the worst question is the one not asked:
Are you sure you are using your HBA's and HSG controllers as efficiently as you can, spreading the load ?
Are there any other systems backing up from the same HSG / controller at the same time ? Maybe even using the same controller ?

Regards
Martin

[Wim Van den Wyngaert]

I tested a partial backup on 15000 files. If you mount the disk with /nocache you consume 20% more cpu but elapsed time descreases with 10%. May be the result is better with more files.

Only a pitty that all caching is disabled. Only header would have been better (diesctory files are re-used).

FWIW

Wim

[Rene Heeres]

We found the solution for our problem.

I have tried/checked several of the hints given in the various replies:

- mounting with nocache
- checked the load on the HBA's / HSG80's --> no problem
- ..

This didn't solve the problem.

In the beginning of this week we had to reboot this server and we then took the opportunity to AUTOGEN the system.

We did the AUTOGEN not based fully upon the Feedback, but started with INITIAL and added then several parameters to get the situation we desired.

After this AUTOGEN and reboot.
We started the backup on this disk en and it only ran for 2h:45m!!!!

So I can only conclude that the problem was in the tuning of the system, I assume it's the ACP_ parameters that dit the trick.

I have added the changed parameters in which you can that these parameters are increased.
Also the paged pool has increased quite a bit, this could also be of influence.

So for now our problems are solved and I would like to thank everybody that helped me in solving this problem.

[Volker Halle]

Rene,

thanks for updating this thread with your findings. Looking at the system parameter changes, nothing jumps out. Maybe even a clean reboot (without changing anything) would have 'solved' the problem.

Consider to closely watch the backup performance on that disk and try to archive accounting information (like DIRIOs, elapsed CPU time and elapsed time), so you could start comparing them, should the problem show up again in the future.

Volker.

[Hein van den Heuvel]

I have to agree with Volker.. it is unlikely that those minor sysgen param changes made a dramatic performance effect.

Allthough I'd feel bad about rebooting fixing it (sort of against the VMS spirit), that is more likely in my opinion.

Did you use BACK/IGNOR=INTERLOCK?
Our friend Mark Hopkins described a situation he analyzed to me which had an interesting performance cliff.
If /ignore=interlock is used for backup then backup will re-visit the file header after copying the data. This revisit is normally a hit in the ACP caches. However, if you manage to open just enough files then the later files in a backup batch might push out the headers for a first files causing all headers to be re-read. This can cause 1/2 more IO, and they can be 'bad' IOs with large seeks.
Please realize that for smallish sized files, like you have, there will be just 1 data IO and 1 header IO per file and that would become 2 headers IO for just one data.

REDUCING the quotas for the account used in the backup woudl help in such case.
Is your backup account 'old school' (thousands of ASTs/DIRios) or new school (Hundreds) ?

(small: 400M blocks space, 100M free, 4M files suggest a max of 300/4 = 75 block per file).


hth,
Hein.

[Volker Halle]

Rene,

another thought crossed my mind:

When was that node last rebooted before ? You described, that the time used to backup this disk kept increasing over time. If there has been a reboot inbetween and the backup time still kept increasing, forget about this idea, otherwise:

If nonpaged pool becomes severely fragmented, this could cause a massive CPU load when allocating/deallocating packets. Note that the MONI FCP CPU Tick Rate was quite high on your system.

If this problem shows up again, use SHOW MEM/POOL/FULL and look at the difference between:

Number of Free Blocks - Free Blocks on Lookasides

The difference will be the number of nonpaged pool packets in variable length pool. If this number becomes significant (many thousand), this will cause a dramatic effect on the CPU time.

Volker.

[Ian Miller.]

Setting the folllowing system parameters to 100
NPAG_GENTLE
NPAG_AGGRESSIVE
turns off an algorithim which has been known to lead to excessive nonpageed pool fragmentation (extreme cases of which result in SPINWAIT bugchecks).

These are dynamic parameters so it can be done without a reboot. In VMS V8.2 the default value of these was chanegd to be 100.

[Wim Van den Wyngaert]

May be the reboot killed a looping process that was doing something with the drive.

In the last 2 years I found several of these cases simply by keeping a list of top IO images (VPA).

Wim