I'd run various series of tests here, tweaking a setting on (or off) and looking for the bottlenecks.
The first limit you are reporting here is apparently involving the I/O bandwidth. It appears saturated.
The RAID volumes are definitely backed up; the Integrity rx3600 series server is tossing I/O significantly faster than the storage can deal with it. (This based on the queue length listings shown in the monitor report. All I/O is waiting. Sometimes massively waiting. The DPA11 device is in really, really, really bad shape.)
General: an I/O queue length of 0.5 means that half of all I/O is waiting. I see one queue -- if those values from MONITOR can be trusted here -- with a queue length of 1.99. That means each I/O is waiting for two other I/Os ahead of it. A continuous rate 0.5 is ugly. A continuous rate of 1.99 is really ugly. Yes, you can also see transient I/O spikes, and you can usually ignore those.
Tools such as the Freeware DFU or the DFO defragger can be used to detect the level of fragmentation at the volume level. If RMS indexed files are in use, I'd look at tuning and file conversions and cleanup. These can help reduce the I/O load; defragmentation and file conversion can reduce the need for I/O.
I'd also look around for processes generating massive I/O. Run-away, looping or otherwise.
Also look at the size of the I/O cache. If something has managed to disable the cache -- like that honking big DECram reservation setting out there -- then VMS has no choice but to toss I/O at the disk. There are SHOW MEMORY commands and SDA XFC extensions and such for evaluating the efficiency of the XFC cache; commands such as SHOW MEMORY /CACHE /FULL.
I don't see large volumes of reads, but I do see stuff writing to the disk, and looking up and opening files. Do you expect to have a heavy write I/O load as compared with read? (That's not what I usually expect to find on most systems. Are there looping or run-away processes lurking somewhere? Something here is generating direct I/O.)
Find out what process(es) or function(s) are causing that volume of I/O. If it's normal, either work into the applications to reduce it or work into the hardware to increase the I/O bandwidth out to the spindles. Or both. Or spread the I/O load across (more) spindles.
See if there is a volume shadowing operation running.
Snoop around and see if you can figure out what's running the lock manager.
See if the auditing and accounting logs show something odd -- or show volumes of something unexpected, like process failures.
The OpenVMS V8.3 quota defaults are just defaults. Applications can require more. Sometimes far more. For instance, there is a set of process quotas for the BACKUP operator username over in the system manager's manual. The quota ratios over there are tuned for better BACKUP performance.
If you are reading and writing from the same spindles during a BACKUP, you might well be exceeding the bus or controller or spindle, too. DLT tapes are very fast devices, and going direct to tape can expedite operations. If you go disk-to-disk, do look at what I/O paths are involved.
FWIW: OpenVMS I64 memory usage, image sizes and other such are generally somewhere between two and four times the equivalent usage and sizes found with OpenVMS Alpha.
Look at the disk structures, too. Did you reinitialize the disks for OpenVMS I64 and reload the contents, or were the disk settings from the 9GB disks brought across from the OpenVMS Alpha system? As a rule of thumb, disk cluster sizes should generally be 16 to 32, or some multiple of 16. This tends to keep transfers aligned, and there are I/O controllers around that are quite sensitive to this. (This recommendation from the OpenVMS I/O engineering team.)
There's a whole lot of information in the tuning and performance manual, if that's of interest. That's a systematic approach toward tuning, and can help you both with some of the details and with the general tuning process you can approach this with. At its simplest (given I'm typing this reply into a slot about a third the size of a VT terminal display) seek details of the problem and a potential trigger for the problem, tweak something related, test it, and either leave it tweaked or untweak, it depending on the results of the testing.
Grab a copy of the T4 tools stuff, and set up to run MONITOR over time. MONITOR has a recording option, and it can be handy and useful to have longer-scale data around what the system is doing, and when. T4 ties into this, too, and allows easier reporting.
I'd start running a continuous MONITOR with recording, and recording at every 1 or 5 or 15 minutes or other such value, depending on the available disk storage and the granularity of the typical system activity. I might well choose to run two passes, one when necessary and at a shorter interval (1 min?) when tuning, and another parallel pass at 15 minute intervals, for long-term trends and tuning and resource tracking.
And a general side note: whilst working at this all, I'd probably choose to move the disks out of allocation class one and keep everything out of allocation class two where you can, these classes should be reserved for potential/eventual use of Fibre Channel disk and tape. This adjustment is not required, and simply as the opportunity presents itself.
As a second general note: apply current ECOs for OpenVMS I64 V8.3.
And my apologies in advance: this is a big topic area, and this is a very small text box, and I'm aiming at some immediate hot-spots, and spots which may or may not be the trigger. As for next steps, watch for the replies here. Also consider that there are vendors around that can assist with monitoring and with system tuning -- and full disclosure: HoffmanLabs is one -- and these can include HP services and various HP partners. And start running the monitoring; start collecting data.
Stephen Hoffman
HoffmanLabs
