Bad performance Openvms

[geir_2]

Hi,

The system performance is decreasing the last month.

The show process command list all process on the system with belonging state and priority.
Is there some special state or columns that i must be aware on? Other suggestions?

The system have one CPU, no NFS.

Rgds Geir

[Robert Gezelter]

Geir,

Many things can cause performance problems that appear to steadily get worse. The problems can range from poorly designed applications to steadily worsening hardware problems.

The SHOW SYSTEM command will show you the process states and the priorities, but that will not necessarily lead you to a result. That report is based upon the exact instant in time that the command is processed. MONITOR PROCESS/TOPCPU would be more helpful in identifying if there is a process consuming all available CPU time.

However, as I mentioned earlier, that is but one of the possible underlying causes of decreasing performance. There are many others.

Do you have some additonal information about the system workload?

- Bob Gezelter, http://www.rlgsc.com

[Wim Van den Wyngaert]

Rather old but try http://starlet.deltatelecom.ru/~laishev/perf-cookbook.txt

Wim

[Ian Miller.]

What are you measuring to know the system performance is decreasing? User reports or actual figures of reduced application thoughput?

Do you have records of performance data (e.g from MONITOR) which cold be compared?

[Karl Rohwedder]

Geir,

1st question: has something changed, e.g.
number of users, new version of software (system or application), hardware, network,...?

2nd: has something gone bad, e.g. memory, disk?

Check Errorlog, SHOW MEMORY, MONITOR *.
SHOW SYSTEM -> processes with xxxO (outswapped, e.g. COMO)

Do you have hard data for the performance degradation or is this more a feeling/user complaint?

regards Kalle

[geir_2]

Hello,

Everyting seems to be normal:

1) Normal traffic on the network according to cisco router
2) No swapped out process
3) 15 interactive users logged in on the system
4) According to monitor process/topcpu
no process have more than 20% CPU. Is there other commands that show the workload on system?
5) The performance is degraded, I have not installed new software, users and hardware.
6) The system is running DECnet IV, TCP/IP and LAT

[Antoniov.]

Hi Geir,
Do your application running using RMS files? May be internal structure is degraded.

Antonio Vigliotti

[Jan van den Ende]

Geir,

have you somehow passed the threshold of starting to use paging?

SHOW MEM/FILE should indicate the amount used, and with
MONI PAGE
you can get an indication of paging activity.

Depending on the answers, totally different things need further investigating.

Proost.

Have one on me.

jpe

[labadie_1]

what applications run on your system ? database, rms files ?

Anyway, you should run a performance collector tool (monitor is on any Vms system, Ecp...), and you should then be able to say you have a Cpu saturated problem, or an I/O problem (disk, bus, file saturated), or a memory problem, or a lock problem...

[Andy Bustamante]

The first question is what is the source of "system performance is decreasing"? What application and how are you measure system performance?

You have a few system replies so far, in addtion has there been any network changes? You can check the Alpha NIC for errors with

$ mc lancp show dev /counter

[Douglas Graver]

What does $ MONITOR DISK/ITEM=QUEUE show?

[Richard White_5 · Accepted solution]

Good Morning Geir...

As if you have not already received enough responses to your dilemma, I thought that I would add a few comments as well. You have already verified that no processes are in the "out-swapped" state, but you should also verify that there are no processes that are constantly/continuously in any sort of MWAIT/RWAIT state. This can be checked via the $mon stat or $mon sys utility or $show sys command. Any process in RWAST/RWJIB usually indicate that process/job has insufficient process quotas in Sysuaf.Dat.

A few other commands for the "Monitor" utility may prove beneficial.

$mon clus is an excellent utility to quickly check how much cpu or memory utilization the system is using, (are you approaching 100% saturation,) as well as the top 6 disk-volumes for QIO's. The system does NOT need to be in a cluster to use this command. But, if this system is in a cluster, you may compare this node to the other nodes (up to 6) in the cluster, to verify consistency amongst the other nodes in the cluster.

$mon rlock and $mon dlock will show you any "dynamic-lock-remastering" as well as local vs. remote locking that is happening, if this system is indeed part of a cluster.

$mon proc/topdio may point to a process that is performing heavy direct-io, possibly to one of the top 6 disk-volumes previously shown in the $mon clus utility.

$mon fcp will show you some of the XQP/ACP File system statistics. Typically the FCP-Call-Rate is a combination of Disk-Reads + Disk-Writes, along with File-Opens and File-Lookups. Excessive overhead/stall times can be seen if the system is performing a high number of "window-turns", which would indicate fragmented files. Also Erase-Rates and Volume-Lock-Waits should be minimal-to-none, for best performance.

$mon proc/topbio will show you any processes that may be executing buffered-io perhaps to a mailbox or a network device. You want to make sure that the Mailbox Reader is at the same priority or higher than the Mailbox Writer; and that the mailbox buffer-quota is large enough, in case you have multiple mb-writers to the same mailbox. If not, you could notice a process going into RWMBX Wait-State.

$mon time will show you the number and types of TQE's the system is performing per second. Depending on your applications and user-workload, you will typically have more "System-Subroutine" TQE's (100-200), followed by "Timer" TQE's (50-150), and "Wakeup" TQE's (1-50). However, be advised that no two systems or applications are exactly the same - your mileage-numbers may vary...

I'm not sure that I saw what version of OVMS you are running, nor what platform, but a couple of these utilities were not implemented until v7.x Hope this helps..

Thanx,
whynot3k

[Doug_81]

Hi Geir:
You've had a lot of good suggestions to start working on. I thought I'd add one more possibility.

I had a system that was normally running great until suddenly, for no apparent reason, it became a real dog. i.e. 10-15 second response to a keyboard entry.

It turned out to be a workstation which was connected to the network with a bad cable (the connector was loose), so it kept connecting, loosing connection, reconnecting, etc.

This workstation was also setup to automatically log a user on, so each time it "connects", it would get a login prompt.

When you get a login prompt on VMS, you're are running loginout process and it runs at a very high priority so it will kill the system performance.

I found this when I issued a multiple
$) show user/node/fu
commands. This showed me that someone was connecting and trying to logon, then it would disappear, then re-appear, etc.

Since there was no actual login completed, I couldn't determine where it was coming from, so I got some assistance from the network boys using a sniffer.

Good luck,
Doug

[Garry Fruth]

This is a rather complex subject. You may want to use the OpenVMS Performance Management manual at http://h71000.www7.hp.com/doc/73final/6491/6491pro.html. Appendix A has a decision tree which will give you a rough outline of how to investigate.

[comarow]

Ok,
The cookbook is great place to start

Watch mon system,
what do you see?

Does the cpu state grow?


What version of vms?

Does it gradually get slow?

Mon disk/item=q do any of your disks
(except when backup is running have a queue depth ON THE AVERAGE of more than 1),

Show mem? Do you have free memory?
Are you low on page file space?

Is it also slow on the console?

Are you logging errors?

Do an
Anal/system
sda>show mem/free

do you have support?

Do you reorganize index files?

Bob

[geir_2]

Hi,
Thanks to everyone for outstanding suggestions. I have tried all commands, but it was impossible to find the reason for performance degradation.

At end I rebooted the system, and now it seems to be faster than ever. If not, I will go through the suggeastions once more.

Thanks to everyone:-)

[Ian Miller.]

If you are not recording performance data then start now. Use T4 or EPC or other product but do use something.

[Galen Tackett]

If this crops up again, Availability Manager or AMDS might also provide some useful realtime information.

[comarow]

Of performance gradually gets slower turn off memory reclamation.

set npage_gentle=100
set npage_aggresive=100

otherwise non page pool with fragment.

Make sure npagedyn hasn't grown
(show mem/full/pool)
and compare current versus initial value.

To see how fragmented npagedyn is
Analy/system
clue mem/free

There should only be a few free elements.
If you see thousands, that's a problem there.

Performance problems are rarely a single factor, but this one is a killer.

In 8.2 they set those parameters to 100 by default.

Bob

[Ian Miller.]

in extreme cases non-paged pool fragmentation can cause SPINWAIT bugchecks with some versions of VMS.

[Karl Rohwedder]

Pls. note the typo on parameter names:
it is NPAG_GENTLE and NPAG_AGGRESSIVE

regards Kalle

[comarow]

The typo corrections are important. Thanks.