Janet,
Not laughing, but wondering if the costs of retaining such an old system outweigh the cost of upgrading? (Though I guess if management won't even let you touch SYSGEN, the idea of a nice shiny new Integrity box is complete fantasy? ;-)
As you know, there are fundamentally three resources to deal with. Memory, CPU and I/O bandwidth. One of these must be the bottle neck. Tuning involves changing stuff to move the bottle neck around so that you maximise the utilisation of the resources.
You've already told us that memory is not the bottleneck. What about CPU?
What are your storage systems? I'd also guess that the internals of your data base are rather tangled, so you may be spending a lot of time thrashing around. I'd also look at locking between systems. On V7.1 you may even be bouncing lock trees back and forth (but I think it's too old to use MONITOR RLOCK to even see).
Counterintuitive thing to try... stop ALL the RDB processes on one node. If you're thrashing locks, you may find that you get better performance on one node than on both.
Check PQL_MWSEXTENT. I'm not sure if V7.1 is young enough, but in recent versions, AUTOGEN will set it to WSMAX. This effectively disables all UAF WSEXTENT settings, and eliminates artificial ceilings on working set sizes. I'm not sure how you convince the fearful ones that a change like this is virtually 0 risk. If memory is already saturated nothing will change. If it's not saturated, it will allow processes to use it, but then if you're not paging it's unlikely to affect anything.
Depending on the physical distribution of your data base files, a defrag might help.
This is especially true if the drop in performance was very sudden. If a data base or RMS indexed file is fragmented across multiple headers, and you split a block high up in an index tree, one branch can end up at the end of the header chain. You then spend half your time chasing down the header list, since they don't get cached.
Maybe a standalone (as in system DOWN) image backup and restore of all disks would help? If possible get hold of some spare disks, do the image backups to a new disk and swap old for new. That way you only do one copy, it's disk to disk, and you always have a fallback. forget going to tape and back. Too slow, and risky. If a tape breaks on the restore you've lost everything. For systems of that age, spare disks could be very cheap on eBay.
You can install DFG and use the SHOW function without a license (DFO? whatever the nomdejur of the HP defrag utility). Look for critical files with multiple extension headers. If you suspect particular files, use DUMP/HEAD/BLOCK=COUNT:0 and count the extension headers.
Note that running DFG itself may not help, if the files are always open.
If all else fails, recommend hiring a performance consultant. Bring in someone like Hein (see other postings), tell him what YOU think, he can then tell management, and since they have to pay him for the advice, they're more likely to believe it! ;-)
A crucible of informative mistakes
