Operating System - OpenVMS
cancel
Showing results for 
Search instead for 
Did you mean: 

Too Much Free Memory on a VAX V7.1

 
oldskul
Advisor

Too Much Free Memory on a VAX V7.1

Go ahead and laugh and get it over with. :)

We have a vax 78xx two node cluster that looks to me like we're not using the memory, not using the page files, not using anything, and both nodes are dog slow.

We have an RDB database (actually several) that haven't had an export/import since the mid 90's because it cannot complete over a long weekend.

I walked into the dilemma and am having a hard time convicing management we need to change sysgen parameters, change UAF working set limits, etc.

I've been a VMS System Manager since V1.2 (July 1980). Copying pages from the Performance Manuals doesn't seem to help my case. My experience doesn't help my case.

Is there a white paper or something I can use to help rid management of the "change fear factor"?

I can send any /all reports you like.

In advance Thank You for your time and consideration.
23 REPLIES 23
Robert Gezelter
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

Welcome to the ITRC OpenVMS Forum.

It is probably best to start with the basics.

What does MONITOR show for IO rates and CPU utilization?

Another good place to start would be to run MONITOR/RECORD for a period of time. Then the recorded data can be analyzed.

There are many things that can cause poor performance. Tuning is always an ongoing process. My first overall recommendation is generally the performance manual. Before talking about changes with clients, the first step is understanding what is happening.

One of the most egregious cases I encountered was a client with a VAX 4000-class system, who had never increased the working set quotas from the baseline. Even log ins were egregiously slow. Fortunately, I was able to demonstrate that the change produced multiple orders of magnitude improvement immediately.

- Bob Gezelter, http://www.rlgsc.com
oldskul
Advisor

Re: Too Much Free Memory on a VAX V7.1

I have monitor and accounting stats going back a year. I/O is nearly under control at this time. Per disk that is. We only experience IO QUE lengths on two drives. Down from all drives. Simple load balancing.

When I do $SHOW MEM at any given time on any day all I see is free this and free that and free the other thing. Page file, slots, everything.

The question isn't where to start or how to do it, but white paper on case studies or success stories? Where a system performance tune-up, or simple UAF WS changes rocked their world versus, "If it ain't broken, why fix it?"

Robert Gezelter
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

There are some articles in the OpenVMS Technical Journal on T4 (which uses MONITOR/RECORD-produced files to gather statistics; and I have used the resulting graphs to identify problems on client systems).

Performance is a very large area, and there are many ways to dissipate performance.

From some experiments I did for a DECUS presentation, it is possible for tuning of the system and application to produce significant improvements. While mileage varies, I have had several cases of multiple orders of magnitude gains. The first step is reviewing the performance statistics to identify what is the bottleneck or excessive consumption of resources.

Unfortunately, that presentation is in the backlog of past sessions that I have not yet posted to our www site. If you want a copy of the slides, please let me know offline and I will be happy to send it to you.

- Bob Gezelter, http://www.rlgsc.com
Jan van den Ende
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

let me add my welcome to Bob's.

>>>
and both nodes are dog slow.
<<<

>>>
or simple UAF WS changes rocked their world versus, "If it ain't broken, why fix it?"
<<<

ehhm, "If it ain't broken"...
Well, to ME, "both nodes are dog slow" is saying it definitely _IS_ broken!!

So, FIX IT!!
Once you manage to get THAT message over, you passed the main hurdle. From there on, it will be 5 % inspiration and 95 % transpiration, but.. you WILL be moving, and from the position you seem to be in, it will be mainly forward.

Been there, done that. And I am sure that like Bob, Volker and Hoff and Hein and John and Wim will likely give similar meanings.

By all means, use this post (SHOW it) to the decision-makers, and point out the ITRC list of specialists that say so.

Good luck.

Proost.

Have one on me.

jpe
Don't rust yours pelled jacker to fine doll missed aches.
Hoff
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

You'll need to understand why your current approach with your management is not gaining any traction.

Your management has reason(s) why the issues here including heat, power, performance, parts, storage have not been resolved. You need to know what those reason(s) are. It might well be budgetary limits or interpersonal issues, for instance.

In general terms, consider why are you concerned about this. Also particularly consider how or why or when the concerns here might "blow up" on your boss, and why, and what that'll mean for your boss and your business.

Tell your management (politely) about the problem(s) you've encountered, why you think these are a problem (and what might happen if the problems aren't resolved) and then abide by their decision. You'll want to communicate that decision with your user base.

With low memory usage and queue lengths below about 0.5 on average, the technical considerations here would then be locking or mutexes and such, database caching settings, database contention, database overhead, and that the VAX 7000 is simply old and slow. (I'd seriously wonder if a Mac Mini Server with MySQL might be faster and more capacious.)

It may simply be that the VAX boxes are considered Good Enough.

And (if you're sufficiently unhappy about this) it might be time to dust off your own resume and think about what's involved in another challenge, and to have a nice career chat with your manager.
John Gillings
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

Not laughing, but wondering if the costs of retaining such an old system outweigh the cost of upgrading? (Though I guess if management won't even let you touch SYSGEN, the idea of a nice shiny new Integrity box is complete fantasy? ;-)

As you know, there are fundamentally three resources to deal with. Memory, CPU and I/O bandwidth. One of these must be the bottle neck. Tuning involves changing stuff to move the bottle neck around so that you maximise the utilisation of the resources.

You've already told us that memory is not the bottleneck. What about CPU?

What are your storage systems? I'd also guess that the internals of your data base are rather tangled, so you may be spending a lot of time thrashing around. I'd also look at locking between systems. On V7.1 you may even be bouncing lock trees back and forth (but I think it's too old to use MONITOR RLOCK to even see).

Counterintuitive thing to try... stop ALL the RDB processes on one node. If you're thrashing locks, you may find that you get better performance on one node than on both.

Check PQL_MWSEXTENT. I'm not sure if V7.1 is young enough, but in recent versions, AUTOGEN will set it to WSMAX. This effectively disables all UAF WSEXTENT settings, and eliminates artificial ceilings on working set sizes. I'm not sure how you convince the fearful ones that a change like this is virtually 0 risk. If memory is already saturated nothing will change. If it's not saturated, it will allow processes to use it, but then if you're not paging it's unlikely to affect anything.

Depending on the physical distribution of your data base files, a defrag might help.

This is especially true if the drop in performance was very sudden. If a data base or RMS indexed file is fragmented across multiple headers, and you split a block high up in an index tree, one branch can end up at the end of the header chain. You then spend half your time chasing down the header list, since they don't get cached.

Maybe a standalone (as in system DOWN) image backup and restore of all disks would help? If possible get hold of some spare disks, do the image backups to a new disk and swap old for new. That way you only do one copy, it's disk to disk, and you always have a fallback. forget going to tape and back. Too slow, and risky. If a tape breaks on the restore you've lost everything. For systems of that age, spare disks could be very cheap on eBay.

You can install DFG and use the SHOW function without a license (DFO? whatever the nomdejur of the HP defrag utility). Look for critical files with multiple extension headers. If you suspect particular files, use DUMP/HEAD/BLOCK=COUNT:0 and count the extension headers.

Note that running DFG itself may not help, if the files are always open.

If all else fails, recommend hiring a performance consultant. Bring in someone like Hein (see other postings), tell him what YOU think, he can then tell management, and since they have to pay him for the advice, they're more likely to believe it! ;-)
A crucible of informative mistakes
Steve Reece_3
Trusted Contributor

Re: Too Much Free Memory on a VAX V7.1

Hi Janet,

My questions here are:

- what's changed?
- has the performance suddenly nosedived or gradually?
- what's AUTOGEN telling you it wants to do?

If AUTOGEN tells you it doesn't want to do anything and it's all ticketyboo, then there's something more fundamental that AUTOGEN is disregarding or doesn't take account of.

What is RDB spending its time doing when it's running? Is it hunting for space to put records in? Is it extending files hand over fist?

How much CPU time are you using and in what modes? Are you running lots of user mode with very little anything else (which should be good) or are you running lots of kernel/supervisor/exec/interrupt mode?

For white papers and performance improvement, I'd probably be looking for RDB info rather than VMS info.

FWIW, I sympathize with the change fear factor after the earlier thread (back in November I think) where we tried to put an extra CPU into a VAX 7000-610 and ended up crashing the system several times.

What's the performance like with a nice shiney new database using the same disks as the existing ones? Does the new one slow to a crawl? If so, that's IO or process driven. If it's super quick, maybe its because of database size or something that RDB is doing because the tables are getting too big to be efficient?

Steve
labadie_1
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet

Could you post the result those commands

$ @sys$examples:working_set
$ sh mem/po/fu
$ mc agen$feedback
$ ty sys$system:agen$feedback.dat
$ sh dev d
$ rmu/sh sys
$ sh sys/noproc

(the command mc agen$feedback will just give back the prompt, but create the file we type in the next command).

oldskul
Advisor

Re: Too Much Free Memory on a VAX V7.1

Thank all of you for reponding.

I'm not sure if it's the cost or the software conversion time that keeps us treading water in

the VAX ocean. My current approach to management is to suggest options to my manager and wait

for approval from the developers (backwards) and it never happens. I care because I know I can make a difference. Dusting off the resume has crossed my mind. They hired an HP guy who pointed out the very same things I suggested.

Putting politics aside it may be possible
this thread will be the spark that lights their fire.

Down to business. T4 = V7.3-2 or higher. MONI/RLOCK, same thing. We have two Intgrity servers that have been in the closet since 2005.

I have attached documents as requested and added a few more. Again. Thank You. So much.
Robert Gezelter
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

The T4 kit (emphasis "Kit") requires 7.3-2 or later.

The basic data files are fully compatible with earlier releases. In fact, I have used the T4 tools on earlier releases by de-constructing the kit and putting it together manually. The basis of T4 is MONITOR/RECORD, and that is far older than 7.3-2 (actually, it is pre-Alpha and then some),

- Bob Gezelter, http://www.rlgsc.com
Robert Gezelter
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

I took a quick look at the ZIP archive. Did I miss CPU utilization information?

Thank you for all of the data. You said the machine is not using the page files and is dog-slow. What is it doing?

- Bob Gezelter, http://www.rlgsc.com
oldskul
Advisor

Re: Too Much Free Memory on a VAX V7.1

If T4 is in vmsinstall kit format I can easily disect. We do Monitor/Record to a file and it runs all day every day. Reports at night for 24 summary and prime time summary. I sent the prime time summary. It's in the zip file.
Robert Gezelter
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

BTW, I unearthed the tuning DECUS presentation I referred to in an earlier posting. It is with the Fall 1997 DECUS presentations at http://www.rlgsc.com/decus/usf97.html

At the risk of being long-winded, I will re-tell a story from more than a decade ago. I was called by a site, who had an early VAX 6200 system. In those days, memory prices were a few orders of magnitude higher than today. The VP of IS was being told that he needed a significant memory upgrade, and the cost would be about US$ 30,000. He called me to see if there were any alternatives.

A check of his configuration showed that his system was doing a great deal of swapping. Each user had a complete copy of a multi-megabyte image. Some additional research showed that the images could be installed as shareable with many beneficial side effects (e.g., less disk IO, less memory, better caching). The problem was completely resolved in less than one day (I was even able to take lunch). Needless to say, the invoice was for far less than the cost of the proposed, but unneeded and never purchased memory.

Not every performance story is so dramatic, but the overall experience is in line with that particular case. Interestingly enough, the software vendor [name deleted] never did adopt the change I recommended, they continued to tell their customers to purchase more memory.

- Bob Gezelter, http://www.rlgsc.com
Andy Bustamante
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Ultimately, you're going to need management and user buy in to make platform changes. If you have new systems sitting around not in use what will it take to get the resources for migration? Are there any vocal complaints about performance? If no one is reporting a problem, then all's quiet and everyone's happy as far as resource allocation is concerned.

The service costs on these systems might be argument for migration, especially if you have hardware on site for migration. Is Rally part of the environment? That could be a significant hold up, although there are paths to migration with appropriate funding and/or time.

You might consider running your monitor files through T4 on an Integrity system. this can provide some nifty pictures for presentation.

As far as current changes, you might try providing steps in a tuning plan, making small quickly restorable changes and moving up to more complicated phases. The "it just works" (and we have no idea of how and why)can leave a good enough level of service in place. If the application is really business critical, I'd really want to be on more recent equipment and a supported version of OpenVMS.

An outside consultant might report the same recommendations with more weight, a common syndrome with some managers. There are performance consultants a plenty here or e-mail me for a few recommendations.

As Hoff points out, if you don't agree with the long term plan and don't feel the position is interesting, consider moving on.


Good luck
If you don't have time to do it right, when will you have time to do it over? Reach me at first_name + "." + last_name at sysmanager net
John Gillings
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

Fragmentation looks fine. The biggest issue I can see is your processes are paging, because of working set constraints. WSMAX, WSEXTENTs and WSQUOTAs are WAY too low. WSMAX is set to 75MB (151604), but you have 3.5GB of memory, of which 80% is free. What is wrong with this picture?

So you paid lots of money for that expensive memory but you're forcing all your processes into 1/4 of it! Not a good ROI.

If you can get one thing past your managers, I'd suggest WSMAX=1500000 and PQL_MWSEXTENT=1500000. This is in line with current AUTOGEN practice. I'd also crank up the UAF WSQUOTAs, if only for username EDA (8196 pages = 4MB that's laughable in this century). It would be simpler to set PQL_MWSQUOTA to (say) 250000. Give them all 100MB so they've got room to do something!

To try and get this across to those who are blocking these simple changes, tell them it's like paying rent on a large warehouse, but walling off three quarters of it, and spending all your time shuffling stored goods around in the remaining quarter to try and make it fit, THEN renting more warehouse space across town to move stuff into because it won't all fit in the quarter you're using, and paying for truck drivers to cart stuff backwards and forwards (indeed with the disparity of memory and disk speeds, it's more like your overflow warehouse is in Antarctica!).

Pull out your iPod nano and explain that it has TEN THOUSAND TIMES the memory that you're allowing your production processes to use, even though there's plenty of space available.

You paid good money for that memory. Please let your system actually use it. You could allow it with some very simple changes in SYSGEN, with virtually ZERO risk. The nice thing about jacking up WSEXTENTs is it's completely self limiting.

Four digit WSQUOTAs were a necessary evil back in the 1980's when 3.5GB of memory cost more than your house, but today when it's a buck fifty at a supermarket checkout it's just plain silly.
A crucible of informative mistakes
oldskul
Advisor

Re: Too Much Free Memory on a VAX V7.1

I want to let everyone know I didn't give up, I've been out of town. I will repond after I catch up. THANK YOU too!
Steve Reece_3
Trusted Contributor

Re: Too Much Free Memory on a VAX V7.1

I'm confused John.

You say:
"The biggest issue I can see is your processes are paging, because of working set constraints. WSMAX, WSEXTENTs and WSQUOTAs are WAY too low. WSMAX is set to 75MB (151604), but you have 3.5GB of memory, of which 80% is free. What is wrong with this picture?"

I don't see any paging to the page files and the EDA processes are only working with WSDEF, not WSQUO or WSEXTent according to the process display. They're page faulting quite a lot, but using next to no working set. I'm confused!
Steve
EdgarZamora_1
Respected Contributor

Re: Too Much Free Memory on a VAX V7.1

Steve, they move to the Modified List first, not to the Page File(s).
oldskul
Advisor

Re: Too Much Free Memory on a VAX V7.1

Steve (and everyone),

The EDA processes are detached "servers" logging in users from our corporate website. User EDA has a wsmax of 151600 but the users that login have a wsmax of 8192. I'm hoping to change that today.
Steve Reece_3
Trusted Contributor

Re: Too Much Free Memory on a VAX V7.1

>>>Steve, they move to the Modified List first, not to the Page File(s).<<<

Nope, still confused. Sorry to be clueless on this.
Hoff
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

>Nope, still confused. Sorry to be clueless on this.

Pages of memory that overflow (page out) from the working set out to the modified list and can move from there out to the pagefile (slow) or back into the working-set.

Faults from the modified list are so-called soft faults and (while there's some overhead here) not nearly as expensive as a so-called hard fault of a memory page back in from disk.

See "7.2.1.1 Hard and Soft Page Faults" here:

http://h71000.www7.hp.com/doc/73final/6491/6491pro_006.html
Robert Gezelter
Honored Contributor

Re: Too Much Free Memory on a VAX V7.1

Janet,

Understanding the usual dangers of "prescribing over the phone" with many of the details unknown, I note that rather than increasing the quotas, an often higher-performance pathway is to do the needed homework to INSTALL the images used by the application.

INSTALLED shareable images are far and away a huge performance win if the majority of the pages are image pages (as opposed to read/write data pages).

- Bob Gezelter, http://www.rlgsc.com
oldskul
Advisor

Re: Too Much Free Memory on a VAX V7.1

Again, everyone. Thank you.

I am at this very moment creating the presentation notes for a meeting tomorrow to change WSMAX on the cluster. One baby step is better than no baby steps.

I installed every image I could last year though we still have many open files on the system disk, it does not have an IOREQ Len greater than .10 ever.

We have a high image activation and some of those are really large cobol programs that are not written with predication in mind.

The biggest system bovine we have is the 4GL FOCUS from IBI. Application design is not VMS saavy.

The EDA user is a WebFocus (TSCOM3) interface. This is the EDA user, which by the way, are detached processes waiting for web users. I apologize if I repeated this.

Reply only gives me the original question. I'm new. I'll get with the program and figure out a better way to answer.