Problem after migration of VMS cluster

[Vladimir Fabecic]

I am having a strange problem after migration of VMS cluster from old MA8000 storage to new XP storage.
OS was image backuped in "stand alone" mode (using OS installed on local disk). Data disks were reorganized (now ODS5, ODS2 before).
No system parameter was changed.
The problem is that oracle exports submited in batch queue work much slower than before. Procedure was not changed, just output device for writing export files.
While doing exports (there are several exports running at same time) there is small CPU usage and very big number of I/O-s on export disk.
Other strange thing is that I did migrations on three locations, and on first location everything works fine (no problem, works even faster than before).
When doing exports interactively, everything work OK (even faster).
Can not see the reason of problem.
OS version is 8.2 alpha.

[Hein van den Heuvel]

Oracle export files are good candidates for very large default RMS file extents. Did anything change in that area?
- SHOW RMS
- SHOW DEV ... "Extend quantity"
- SHOW DEV ... "Cluster size"

Has the level of parallelism increased?
Concurrently growing export files with small extents and small cluster size will produce maximally fragmented files.
Check with DFU REPORT

Check the IO sizes with $SHOW MEM/CACHE/FULL. Use SET CACH/RESET just before, and check the IO size histogram.

Check with stats for the volume with XFC extension in SDA?

And uh.. why would anyone go to V8.2 Alpha... and stay there to boot. Willly nilly uninformed pointy haired management decision? Weak and wimpish system management team? Bullies in the development group? I know _you_ know better!

Hope this helps,
Hein van den Heuvel
HvdH Performance Consulting

[Jan van den Ende]

Vladimir,

Let me start we the obvious: newer storage is much more sensitive to "right size" chunks.
Is the cluster size an integer multiple of 16?
Are the relevant RMS parameters? (and NOT the defaults!) For the export disk I would especially look at EXTEND size - make that a BIG multiple of 16; typically 1/4 - 1/2 the average size of the export files.

Oh, and have you turned OFF highwater marking?

VMS 8.2? Alpha or IA64? Anyway, DO seriously consider upgrading.

-- just a few first things.

Proost.

Have one on me.

jpe

[Robert Gezelter · Accepted solution]

Vladimir,

I will not repeat the comments made earlier, but will note that it would be worthwhile to check the configuration of the volumes on the storage arrays.

Extend sizes, minimum quotas, and other factors all interact, I would recommend comparing the settings on one of the "working" systems with the settings from the one that is "not performing as well".

I would also suggest verifying that the XP arrays and the virtual volumes are configured correctly.

I would also consider creating a virgin test volume and seeing if that has an affect. The volumes were converted from ODS-2 to ODS-5. Was this conversion done by an in-place convert, or was it done by creating a new volume and restoring the files?

- Bob Gezelter, http://www.rlgsc.com

[Vladimir Fabecic]

Today I am not able to access those systems, will do it tomorrow.
So tomorrow I will give details.
Disk was initialized, not converted.
And was initialized with cluster size 64 (but so was on working system).
XP arrays and the virtual volumes were created at same way.
Why 8.2 and not 8.3?
Because of old Oracle 8.
Of course I agree that upgrade should be done (but can not be while using Oracle 8).
Migration to Oracle 10 is in process.

[Jan van den Ende]

Vladimir,

>>>
Vladimir,

>>>
Disk was initialized, not converted.
And was initialized with cluster size 64 (but so was on working system).
<<<

That makes HIGHWATERMARKING my first suspect, and EXTENDsize a good second.

dobr dan.

Proost.

Have one on me.

jpe

[Hein van den Heuvel]

Ayup, Jan makes a good point with HWM.
If that is in play it will have a bigger effect than extend sizes.
To clrear/test, use SET VOLUME /NOHIGH

Yeah yeah, heard the old Oracle release before.
But in this case, Oracle 8 is not supported anywhere, not on 8.2 nor on 8.3.
It just happened to be never validated agains 8.3 because OpenVMS 8.3 wasn't there (yet) when there was still a modicum of interest in Oracle 8.

Surely it (Oracle 8 on OpenVMS 8.3 Alpha) will work just fine, and likely better than on 8.2?
I don't recall issues, but I suppose there could have been some. If so, someone can perhaps remind us here? It's all so long ago.

Anyway, it sound like a moot point as work is being done to get to Oracle 10.

I seem to recall some applications, notably with simple singleton selects, had performance issues upgrading from Oracle 8.
Also, the dominance of cost based optimization now, versus there rules based typically used in 8i, has initially surprised some implementation. Still, that should prove to be goodness in the long run.

hth,
Hein van den Heuvel
HvdH Performance Consulting

[Steve Reece_3]

As others have mentioned, highwater marking is a likely candidate. How come you did an ODS2 to ODS5 migration at the same time though? Have you verified that the problem exists on an ODS2 volume on the new storage?

My usual question in cases where performance drops suddenly is, "what's changed?" In this case, the whole storage subsystem has changed so try and rule out bits of changes since you say that the first site wasn't an issue yet I'm guessing the same process was followed at each site?

[Vladimir Fabecic]

I was away from case and job because of big private problem (health related).
Please do not take me wrong for not updating thread.
Problem was fixed by reinitializing disks to ODS2.
Thanks everybody for your time and help.

[Robert Gezelter]

Vladimir,

From the resolution reported, I would suspect that the problem is not ODS-2 versus ODS-5. Rather, I would suspect that the problem is incidentally related to that change.

ODS-5 allows a smaller cluster size on large disks than ODS-2. Perhaps, the observed performance is collaterally related to a large number of file extends resulting from a smaller cluster size on the ODS-2 volume. to wit, extension size for a file must round up to the nearest whole cluster.

For example, an extend size of five blocks on a disk with a cluster size of one block would extend the file every five blocks. On a disk with a cluster factor of 255 blocks, the minimum extension would be 255 blocks. This would produce a difference of 5100% in number of extend operations.

This should be easily confirmed using a temporary test volume. The extension size can be temporarily overridden using the SET RMS command on a per-process basis. If the Oracle export utility is still extending in small quanta, then it should be reported as a bug, as this would imply that the file extension quantity is hard-coded).

I hope that the above is helpful.

- Bob Gezelter, http://www.rlgsc.com