Operating System - OpenVMS
cancel
Showing results for 
Search instead for 
Did you mean: 

Too Much Free Memory on a VAX V7.1

 
Highlighted
Advisor

Too Much Free Memory on a VAX V7.1

Go ahead and laugh and get it over with. :)

We have a vax 78xx two node cluster that looks to me like we're not using the memory, not using the page files, not using anything, and both nodes are dog slow.

We have an RDB database (actually several) that haven't had an export/import since the mid 90's because it cannot complete over a long weekend.

I walked into the dilemma and am having a hard time convicing management we need to change sysgen parameters, change UAF working set limits, etc.

I've been a VMS System Manager since V1.2 (July 1980). Copying pages from the Performance Manuals doesn't seem to help my case. My experience doesn't help my case.

Is there a white paper or something I can use to help rid management of the "change fear factor"?

I can send any /all reports you like.

In advance Thank You for your time and consideration.
23 REPLIES 23
Highlighted
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

Welcome to the ITRC OpenVMS Forum.

It is probably best to start with the basics.

What does MONITOR show for IO rates and CPU utilization?

Another good place to start would be to run MONITOR/RECORD for a period of time. Then the recorded data can be analyzed.

There are many things that can cause poor performance. Tuning is always an ongoing process. My first overall recommendation is generally the performance manual. Before talking about changes with clients, the first step is understanding what is happening.

One of the most egregious cases I encountered was a client with a VAX 4000-class system, who had never increased the working set quotas from the baseline. Even log ins were egregiously slow. Fortunately, I was able to demonstrate that the change produced multiple orders of magnitude improvement immediately.

- Bob Gezelter, http://www.rlgsc.com
Highlighted
Advisor

Re: Too Much Free Memory on a VAX V7.1

I have monitor and accounting stats going back a year. I/O is nearly under control at this time. Per disk that is. We only experience IO QUE lengths on two drives. Down from all drives. Simple load balancing.

When I do $SHOW MEM at any given time on any day all I see is free this and free that and free the other thing. Page file, slots, everything.

The question isn't where to start or how to do it, but white paper on case studies or success stories? Where a system performance tune-up, or simple UAF WS changes rocked their world versus, "If it ain't broken, why fix it?"

Highlighted
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

There are some articles in the OpenVMS Technical Journal on T4 (which uses MONITOR/RECORD-produced files to gather statistics; and I have used the resulting graphs to identify problems on client systems).

Performance is a very large area, and there are many ways to dissipate performance.

From some experiments I did for a DECUS presentation, it is possible for tuning of the system and application to produce significant improvements. While mileage varies, I have had several cases of multiple orders of magnitude gains. The first step is reviewing the performance statistics to identify what is the bottleneck or excessive consumption of resources.

Unfortunately, that presentation is in the backlog of past sessions that I have not yet posted to our www site. If you want a copy of the slides, please let me know offline and I will be happy to send it to you.

- Bob Gezelter, http://www.rlgsc.com
Highlighted
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

let me add my welcome to Bob's.

>>>
and both nodes are dog slow.
<<<

>>>
or simple UAF WS changes rocked their world versus, "If it ain't broken, why fix it?"
<<<

ehhm, "If it ain't broken"...
Well, to ME, "both nodes are dog slow" is saying it definitely _IS_ broken!!

So, FIX IT!!
Once you manage to get THAT message over, you passed the main hurdle. From there on, it will be 5 % inspiration and 95 % transpiration, but.. you WILL be moving, and from the position you seem to be in, it will be mainly forward.

Been there, done that. And I am sure that like Bob, Volker and Hoff and Hein and John and Wim will likely give similar meanings.

By all means, use this post (SHOW it) to the decision-makers, and point out the ITRC list of specialists that say so.

Good luck.

Proost.

Have one on me.

jpe
Don't rust yours pelled jacker to fine doll missed aches.
Highlighted
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

You'll need to understand why your current approach with your management is not gaining any traction.

Your management has reason(s) why the issues here including heat, power, performance, parts, storage have not been resolved. You need to know what those reason(s) are. It might well be budgetary limits or interpersonal issues, for instance.

In general terms, consider why are you concerned about this. Also particularly consider how or why or when the concerns here might "blow up" on your boss, and why, and what that'll mean for your boss and your business.

Tell your management (politely) about the problem(s) you've encountered, why you think these are a problem (and what might happen if the problems aren't resolved) and then abide by their decision. You'll want to communicate that decision with your user base.

With low memory usage and queue lengths below about 0.5 on average, the technical considerations here would then be locking or mutexes and such, database caching settings, database contention, database overhead, and that the VAX 7000 is simply old and slow. (I'd seriously wonder if a Mac Mini Server with MySQL might be faster and more capacious.)

It may simply be that the VAX boxes are considered Good Enough.

And (if you're sufficiently unhappy about this) it might be time to dust off your own resume and think about what's involved in another challenge, and to have a nice career chat with your manager.
Highlighted
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

Not laughing, but wondering if the costs of retaining such an old system outweigh the cost of upgrading? (Though I guess if management won't even let you touch SYSGEN, the idea of a nice shiny new Integrity box is complete fantasy? ;-)

As you know, there are fundamentally three resources to deal with. Memory, CPU and I/O bandwidth. One of these must be the bottle neck. Tuning involves changing stuff to move the bottle neck around so that you maximise the utilisation of the resources.

You've already told us that memory is not the bottleneck. What about CPU?

What are your storage systems? I'd also guess that the internals of your data base are rather tangled, so you may be spending a lot of time thrashing around. I'd also look at locking between systems. On V7.1 you may even be bouncing lock trees back and forth (but I think it's too old to use MONITOR RLOCK to even see).

Counterintuitive thing to try... stop ALL the RDB processes on one node. If you're thrashing locks, you may find that you get better performance on one node than on both.

Check PQL_MWSEXTENT. I'm not sure if V7.1 is young enough, but in recent versions, AUTOGEN will set it to WSMAX. This effectively disables all UAF WSEXTENT settings, and eliminates artificial ceilings on working set sizes. I'm not sure how you convince the fearful ones that a change like this is virtually 0 risk. If memory is already saturated nothing will change. If it's not saturated, it will allow processes to use it, but then if you're not paging it's unlikely to affect anything.

Depending on the physical distribution of your data base files, a defrag might help.

This is especially true if the drop in performance was very sudden. If a data base or RMS indexed file is fragmented across multiple headers, and you split a block high up in an index tree, one branch can end up at the end of the header chain. You then spend half your time chasing down the header list, since they don't get cached.

Maybe a standalone (as in system DOWN) image backup and restore of all disks would help? If possible get hold of some spare disks, do the image backups to a new disk and swap old for new. That way you only do one copy, it's disk to disk, and you always have a fallback. forget going to tape and back. Too slow, and risky. If a tape breaks on the restore you've lost everything. For systems of that age, spare disks could be very cheap on eBay.

You can install DFG and use the SHOW function without a license (DFO? whatever the nomdejur of the HP defrag utility). Look for critical files with multiple extension headers. If you suspect particular files, use DUMP/HEAD/BLOCK=COUNT:0 and count the extension headers.

Note that running DFG itself may not help, if the files are always open.

If all else fails, recommend hiring a performance consultant. Bring in someone like Hein (see other postings), tell him what YOU think, he can then tell management, and since they have to pay him for the advice, they're more likely to believe it! ;-)
A crucible of informative mistakes
Highlighted
Trusted Contributor

Re: Too Much Free Memory on a VAX V7.1

Hi Janet,

My questions here are:

- what's changed?
- has the performance suddenly nosedived or gradually?
- what's AUTOGEN telling you it wants to do?

If AUTOGEN tells you it doesn't want to do anything and it's all ticketyboo, then there's something more fundamental that AUTOGEN is disregarding or doesn't take account of.

What is RDB spending its time doing when it's running? Is it hunting for space to put records in? Is it extending files hand over fist?

How much CPU time are you using and in what modes? Are you running lots of user mode with very little anything else (which should be good) or are you running lots of kernel/supervisor/exec/interrupt mode?

For white papers and performance improvement, I'd probably be looking for RDB info rather than VMS info.

FWIW, I sympathize with the change fear factor after the earlier thread (back in November I think) where we tried to put an extra CPU into a VAX 7000-610 and ended up crashing the system several times.

What's the performance like with a nice shiney new database using the same disks as the existing ones? Does the new one slow to a crawl? If so, that's IO or process driven. If it's super quick, maybe its because of database size or something that RDB is doing because the tables are getting too big to be efficient?

Steve
Highlighted
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet

Could you post the result those commands

$ @sys$examples:working_set
$ sh mem/po/fu
$ mc agen$feedback
$ ty sys$system:agen$feedback.dat
$ sh dev d
$ rmu/sh sys
$ sh sys/noproc

(the command mc agen$feedback will just give back the prompt, but create the file we type in the next command).

Highlighted
Advisor

Re: Too Much Free Memory on a VAX V7.1

Thank all of you for reponding.

I'm not sure if it's the cost or the software conversion time that keeps us treading water in

the VAX ocean. My current approach to management is to suggest options to my manager and wait

for approval from the developers (backwards) and it never happens. I care because I know I can make a difference. Dusting off the resume has crossed my mind. They hired an HP guy who pointed out the very same things I suggested.

Putting politics aside it may be possible
this thread will be the spark that lights their fire.

Down to business. T4 = V7.3-2 or higher. MONI/RLOCK, same thing. We have two Intgrity servers that have been in the closet since 2005.

I have attached documents as requested and added a few more. Again. Thank You. So much.