topic Re: Bad performance Openvms in Operating System - OpenVMS

Bad performance Openvms

geir_2 — Mon, 27 Jun 2005 02:33:16 GMT

Hi,

The system performance is decreasing the last month.

The show process command list all process on the system with belonging state and priority.
Is there some special state or columns that i must be aware on? Other suggestions?

The system have one CPU, no NFS.

Rgds Geir

Re: Bad performance Openvms

Robert Gezelter — Mon, 27 Jun 2005 03:30:53 GMT

Geir,

Many things can cause performance problems that appear to steadily get worse. The problems can range from poorly designed applications to steadily worsening hardware problems.

The SHOW SYSTEM command will show you the process states and the priorities, but that will not necessarily lead you to a result. That report is based upon the exact instant in time that the command is processed. MONITOR PROCESS/TOPCPU would be more helpful in identifying if there is a process consuming all available CPU time.

However, as I mentioned earlier, that is but one of the possible underlying causes of decreasing performance. There are many others.

Do you have some additonal information about the system workload?

- Bob Gezelter, http://www.rlgsc.com

Re: Bad performance Openvms

Wim Van den Wyngaert — Mon, 27 Jun 2005 03:35:53 GMT

Rather old but try http://starlet.deltatelecom.ru/~laishev/perf-cookbook.txt

Wim

Re: Bad performance Openvms

Ian Miller. — Mon, 27 Jun 2005 03:36:17 GMT

What are you measuring to know the system performance is decreasing? User reports or actual figures of reduced application thoughput?

Do you have records of performance data (e.g from MONITOR) which cold be compared?

Re: Bad performance Openvms

Karl Rohwedder — Mon, 27 Jun 2005 03:39:17 GMT

Geir,

1st question: has something changed, e.g.
number of users, new version of software (system or application), hardware, network,...?

2nd: has something gone bad, e.g. memory, disk?

Check Errorlog, SHOW MEMORY, MONITOR *.
SHOW SYSTEM -> processes with xxxO (outswapped, e.g. COMO)

Do you have hard data for the performance degradation or is this more a feeling/user complaint?

regards Kalle

Re: Bad performance Openvms

geir_2 — Mon, 27 Jun 2005 04:07:06 GMT

Hello,

Everyting seems to be normal:

1) Normal traffic on the network according to cisco router
2) No swapped out process
3) 15 interactive users logged in on the system
4) According to monitor process/topcpu
no process have more than 20% CPU. Is there other commands that show the workload on system?
5) The performance is degraded, I have not installed new software, users and hardware.
6) The system is running DECnet IV, TCP/IP and LAT

Re: Bad performance Openvms

Antoniov. — Mon, 27 Jun 2005 05:19:46 GMT

Hi Geir,
Do your application running using RMS files? May be internal structure is degraded.

Antonio Vigliotti

Re: Bad performance Openvms

Jan van den Ende — Mon, 27 Jun 2005 05:55:27 GMT

Geir,

have you somehow passed the threshold of starting to use paging?

SHOW MEM/FILE should indicate the amount used, and with
MONI PAGE
you can get an indication of paging activity.

Depending on the answers, totally different things need further investigating.

Proost.

Have one on me.

jpe

Re: Bad performance Openvms

labadie_1 — Mon, 27 Jun 2005 06:37:30 GMT

what applications run on your system ? database, rms files ?

Anyway, you should run a performance collector tool (monitor is on any Vms system, Ecp...), and you should then be able to say you have a Cpu saturated problem, or an I/O problem (disk, bus, file saturated), or a memory problem, or a lock problem...

Re: Bad performance Openvms

Andy Bustamante — Mon, 27 Jun 2005 07:57:31 GMT

The first question is what is the source of "system performance is decreasing"? What application and how are you measure system performance?

You have a few system replies so far, in addtion has there been any network changes? You can check the Alpha NIC for errors with

$ mc lancp show dev /counter

Re: Bad performance Openvms

Douglas Graver — Mon, 27 Jun 2005 15:14:05 GMT

What does $ MONITOR DISK/ITEM=QUEUE show?

Re: Bad performance Openvms

Richard White_5 — Mon, 27 Jun 2005 15:29:43 GMT

Good Morning Geir...

As if you have not already received enough responses to your dilemma, I thought that I would add a few comments as well. You have already verified that no processes are in the "out-swapped" state, but you should also verify that there are no processes that are constantly/continuously in any sort of MWAIT/RWAIT state. This can be checked via the $mon stat or $mon sys utility or $show sys command. Any process in RWAST/RWJIB usually indicate that process/job has insufficient process quotas in Sysuaf.Dat.

A few other commands for the "Monitor" utility may prove beneficial.

$mon clus is an excellent utility to quickly check how much cpu or memory utilization the system is using, (are you approaching 100% saturation,) as well as the top 6 disk-volumes for QIO's. The system does NOT need to be in a cluster to use this command. But, if this system is in a cluster, you may compare this node to the other nodes (up to 6) in the cluster, to verify consistency amongst the other nodes in the cluster.

$mon rlock and $mon dlock will show you any "dynamic-lock-remastering" as well as local vs. remote locking that is happening, if this system is indeed part of a cluster.

$mon proc/topdio may point to a process that is performing heavy direct-io, possibly to one of the top 6 disk-volumes previously shown in the $mon clus utility.

$mon fcp will show you some of the XQP/ACP File system statistics. Typically the FCP-Call-Rate is a combination of Disk-Reads + Disk-Writes, along with File-Opens and File-Lookups. Excessive overhead/stall times can be seen if the system is performing a high number of "window-turns", which would indicate fragmented files. Also Erase-Rates and Volume-Lock-Waits should be minimal-to-none, for best performance.

$mon proc/topbio will show you any processes that may be executing buffered-io perhaps to a mailbox or a network device. You want to make sure that the Mailbox Reader is at the same priority or higher than the Mailbox Writer; and that the mailbox buffer-quota is large enough, in case you have multiple mb-writers to the same mailbox. If not, you could notice a process going into RWMBX Wait-State.

$mon time will show you the number and types of TQE's the system is performing per second. Depending on your applications and user-workload, you will typically have more "System-Subroutine" TQE's (100-200), followed by "Timer" TQE's (50-150), and "Wakeup" TQE's (1-50). However, be advised that no two systems or applications are exactly the same - your mileage-numbers may vary...

I'm not sure that I saw what version of OVMS you are running, nor what platform, but a couple of these utilities were not implemented until v7.x Hope this helps..

Thanx,
whynot3k

Re: Bad performance Openvms

Doug_81 — Mon, 27 Jun 2005 16:41:34 GMT

Hi Geir:
You've had a lot of good suggestions to start working on. I thought I'd add one more possibility.

I had a system that was normally running great until suddenly, for no apparent reason, it became a real dog. i.e. 10-15 second response to a keyboard entry.

It turned out to be a workstation which was connected to the network with a bad cable (the connector was loose), so it kept connecting, loosing connection, reconnecting, etc.

This workstation was also setup to automatically log a user on, so each time it "connects", it would get a login prompt.

When you get a login prompt on VMS, you're are running loginout process and it runs at a very high priority so it will kill the system performance.

I found this when I issued a multiple
$) show user/node/fu
commands. This showed me that someone was connecting and trying to logon, then it would disappear, then re-appear, etc.

Since there was no actual login completed, I couldn't determine where it was coming from, so I got some assistance from the network boys using a sniffer.

Good luck,
Doug

Re: Bad performance Openvms

Garry Fruth — Mon, 27 Jun 2005 19:15:47 GMT

This is a rather complex subject. You may want to use the OpenVMS Performance Management manual at http://h71000.www7.hp.com/doc/73final/6491/6491pro.html. Appendix A has a decision tree which will give you a rough outline of how to investigate.

Re: Bad performance Openvms

comarow — Tue, 28 Jun 2005 19:35:15 GMT

Ok,
The cookbook is great place to start

Watch mon system,
what do you see?

Does the cpu state grow?

What version of vms?

Does it gradually get slow?

Mon disk/item=q do any of your disks
(except when backup is running have a queue depth ON THE AVERAGE of more than 1),

Show mem? Do you have free memory?
Are you low on page file space?

Is it also slow on the console?

Are you logging errors?

Do an
Anal/system
sda>show mem/free

do you have support?

Do you reorganize index files?

Bob

Re: Bad performance Openvms

geir_2 — Wed, 29 Jun 2005 05:37:22 GMT

Hi,
Thanks to everyone for outstanding suggestions. I have tried all commands, but it was impossible to find the reason for performance degradation.

At end I rebooted the system, and now it seems to be faster than ever. If not, I will go through the suggeastions once more.

Thanks to everyone:-)

Re: Bad performance Openvms

Ian Miller. — Wed, 29 Jun 2005 06:25:30 GMT

If you are not recording performance data then start now. Use T4 or EPC or other product but do use something.

Re: Bad performance Openvms

Galen Tackett — Wed, 29 Jun 2005 06:43:51 GMT

If this crops up again, Availability Manager or AMDS might also provide some useful realtime information.

Re: Bad performance Openvms

comarow — Sat, 02 Jul 2005 04:54:43 GMT

Of performance gradually gets slower turn off memory reclamation.

set npage_gentle=100
set npage_aggresive=100

otherwise non page pool with fragment.

Make sure npagedyn hasn't grown
(show mem/full/pool)
and compare current versus initial value.

To see how fragmented npagedyn is
Analy/system
clue mem/free

There should only be a few free elements.
If you see thousands, that's a problem there.

Performance problems are rarely a single factor, but this one is a killer.

In 8.2 they set those parameters to 100 by default.

Bob

Re: Bad performance Openvms

Ian Miller. — Mon, 04 Jul 2005 04:14:59 GMT

in extreme cases non-paged pool fragmentation can cause SPINWAIT bugchecks with some versions of VMS.