System slow

[John Russell_13]

Hello Gentleman,

The job that people are suspicous about is a real time interface between our patient billing/scheduling system and the patient/billing system that is used by the hospitals in our healthcare network. It runs 24/7 and used to be a problem until we upgraded from 1 to 3 CPU's.

About a month ago we had a major software upgrade and I've been told that physical memory is an issue. Being a financially poor healthcare system, we've been told that we cannot purchase memory at this time and must make due with what the alpha system has. As I mentioned before, out VMS guru is on leave and now we're not sure when he's coming back so I'm having to learn OpenVMS very quickly.

I'll attach another .txt of what's happening with the system as we speak. It's actually flying today. Some days it's very sluggish. All of the information I've gotten in this string has been very helpful in knowing what to look for. I really appreciate all of you giving me this information. I think I want to be an OpenVMS guru when I grow up!! I love the monitor utility. It's fun!!

Anyway, DSA4 is our heavy hitter. I'm going to have someone from HP look into our HSG controller configuration and make sure it's behaving efficiently/correctly.

I'm just going to assume that a few large jobs were running yesterday and killing our Finance database and I'm going to do what I can to prevent this from happening again.

Thank you all!!! Any other suggestions would be greatly appreciated!!

John

[Hein van den Heuvel]

John,
For a windows guy you seem to be doing an admirable job trying to make sense of this VMS system! There could be a carreer here :-).

Bob brings up a good point high-lighting this JOB0f UZ02DRV process. That's an awfull lot of CPU and a good chunck of the total IOs. It that job trying to accomplish a single tasks, or is it a slave waiting for work items?

Still, that would have been a problem all along, and best dealt with by the regular staff.

I suspect (WAG) that for your immediate (now past) incident you had some shadowing event. Something where the system decided to read entire disks and tried to compare & resync them? Dunno. An other good chunk of the IOs has indeed come from the SHADOW_SERVER.

The XFC hit rate is abysmall. That is not normal. It would be good to try to understand that, but deep(er) application knowledge is needed for that. Large file scans? Private (non-vms-backup) backup technique (simple copy?)?

Good luck,
Hein.

[Hein van den Heuvel]

Looks like you were re-replung while I posed mine.

Nice to see the MONI MODE. It excludes for example and RMS or Oracle problem (no exec mode)

Looks like an intersystems cache application.
You really should try to work with that DB vendor to see if the caches are set up correctly, the execution plans are behavign and to forth. You may want to give more memory to its cache an not have the, ineffective for you, XFC cache try to waste time on blocks that are not going to be re-requested.

Check with intersystems whether this low XFC is typical for their VMS solution (Maybe they should learn to issue 'bypass cache' style IOs for the main DB work).

Hein.

[Lawrence Czlapinski]

John,
1. Can you attach a MONITOR CLUSTER and MONITOR PROC/TOPCPU outputs when you have this situation again. It would give an overview of the situation.
2. If you can get AVAIL_MAN installed, it would give you a continous look at % CPU usage and CPU queues on all the nodes. It would also show you % memory usage, etc. I often notice runaway CPU processes that way. It would show which jobs are getting CPU time and which are waiting for CPU time.
3. If it is some large jobs taking up a lot of the CPU time perhaps they should be running as batch jobs at a lower priority than interactive jobs. Or perhaps something is running at a higher priority and keeping the rest of the jobs from getting a quantum of time. Batch jobs should be running at a lower priority than interactive jobs.
4. I would also recommend looking to see if you have a "Guide to OpenVMS Performance Mangement" manual around. It has some troubleshooting trees.
Lawrence

[Lawrence Czlapinski]

John, after rereading your question it looks like you have one system with 3 CPUs. So disregard the MON CLUSTER.
Lawrence

[Robert Gezelter]

John,

When you increased the number of CPUs, the CPU hog job basically got its own CPU. This is has the potential to cause other problems, as may be the case here.

The last time a client presented a similar situation to me, we were able to identify and eliminate the source of the high CPU consumption, with a substantial savings in resources.

The SHOW MEMORY command will show the memory utilization. Even if you see a low free page count, you may not need additional memory, it may be a straightforward tuning issue.

I hope that the preceeding is helpful, it is difficult to "get a handle" on the situation when one is this remote from it.

- Bob Gezelter, http://www.rlgsc.com

[Ian Miller.]

look in SYS$MANAGER:OPERATOR.LOG to see if there are messages from the shadowing software particularly DSA4.
What is CACHE.DAT ?

Does

$ SHOW MEMORY/CACHE=(TOPQIO,VOLUME=DSA4)

Show any particular file?

[Jeff Chisholm]

The Intersystems Cache product uses it's own global sections and maps chunks of physical memory for improving disk io performance. The VIOC and XFC are not recommended by them, since both of these take memory resources that could be assigned to the application.

It's possible to turn on XFC, but you'd want to limit the max size and exclude the Cache database volumes.

Take care what you recommend, and what recommendations you implement, unless you've got some experience tuning Cache.

[Antoniov.]

John,
welcome to vms world :-)

Because you are newbie in vms you could run $ @SYS$UPDATE:AUTOGEN SAVPARAMS GETDATA
This command write two files SYS$SYSTEM:AGEN$FEEDBACK.DAT and SYS$SYSTEM:PARAMS.DAT
Looking at AGEN$FEEDBACK.DAT you can see how os evaluates system parameters.
The above command doen't make any change, so it's no dangerous and you can understand more about system parameters of vms.

Antonio Vigliotti

[Antoniov.]

John,
I've forgotten you can find all techincal vms documetantion here
http://h71000.www7.hp.com/doc/os73_index.html

Antonio Vigliotti