topic Re: swapped out in Operating System - OpenVMS

swapped out

adarsh_4 — Sun, 15 Mar 2009 21:21:48 GMT

i have open vms7.3 and at times it go slow. i did a show system and i have copied it to the attachment. it says something about swapped out. what can be the problem and how can we solve it

Re: swapped out

Hein van den Heuvel — Sun, 15 Mar 2009 21:50:43 GMT

Well, it looks like those processes are swapped out as the system tried to free up memory. The processes that are not swapped out are trimmed down to bare bones.

Next command you want to use is SHWO MEMORY/FULL.

It looks like the memory is totally overcommited. How much is there? What is it used by?

My guess? You have some large (RDB?) global sections still stuck in memory waiting for a Database to shut down or something like that.

Maybe that INSTALL LIST /GLOBAL can identify some good sized chunks of pages under the pgltcnt columns? check also for pending delete?

Or maybe on the last reboot the system came up with a good chunk of memory missing?

Hope this helps,
Hein.

Re: swapped out

Jon Pinkley — Sun, 15 Mar 2009 21:58:20 GMT

Short answer: You need more memory.

You have many processes in COMO (Waiting for CPU, but outswapped) and RWMPB (Resource Wait, Modified Pagewriter Busy).

Solution: Buy more memory, or reduce the load.

Jon

Re: swapped out

adarsh_4 — Sun, 15 Mar 2009 22:13:47 GMT

thanks for the advise.

Re: swapped out

Robert Gezelter — Sun, 15 Mar 2009 23:29:56 GMT

dharsh,

As Hein and Jon have noted, it is quite possible that there is simply a need for more memory than there is presently in the system.

However, the original post did not specify the hardware or architecture involved.

It also mentioned nothing about what images are being run on this system.

Before purchasing additional memory, I would investigate tuning the system. I have seen many situations where there are things that can be done to the system software configuration to address this problem.

As Hein mentioned, a SHOW MEMORY/FULL is certainly a starting point. A look at the images being run by different users is also an important step.

I once had a client who had almost the precisely same problem. System performance was unacceptable, and they were getting ready to ask upper (non-IT) management for significant funds to increase memory capacity or purchase a large system.

However, they decided to ask me to take a look at the system. I found a large number of users running the same, multi-thousand page image. I checked the image, installed it, and then watched all of the swapping and paging stop as if by magic.

Needless to say, the memory expansion project never happened.

The moral of the tale: Tune before purchasing hardware, then one better knows if one actually NEEDS the extra hardware.

- Bob Gezelter, http://www.rlgsc.com

Re: swapped out

Wim Van den Wyngaert — Mon, 16 Mar 2009 08:08:32 GMT

May be a process took A LOT of memory and then finsihed and this before you looked.
Check in accounting/full the field "peak working set".

Wim

Re: swapped out

The Brit — Mon, 16 Mar 2009 10:21:05 GMT

Adarsh,
I would also check to see if automatic memory reclaimation is turned on. (check value of sys param PFRATL, if = 0 then mem reclaimation is turned off.) Set PFRATL to a value >0 (i.e. 1 or 2).

This will have the affect of reducing the Working sets of processes which have more than they need, and putting those pages back on the free list.

Dave.

Re: swapped out

adarsh_4 — Mon, 16 Mar 2009 21:02:41 GMT

that problem i have is that i cant log in the system to see what is going on when it starts to freeze. i only manage to get the sh system command running. how can i know what is chewing up the memory after i restart the system and its working fine.

Re: swapped out

Robert Gezelter — Mon, 16 Mar 2009 21:40:15 GMT

adarsh,

I have seen similar problems. The system is not actually frozen, just bogged down.

One useful piece of information would be the precise hardware configuration. There are several avenues to address this.

One is to check the question I posed earlier: Are the various users running the same image (or a small number of images)? Is that image installed, so that memory is used more efficiently?

If one must reboot at the point of failure, taking the steps to record a crash dump will allow analysis of precisely what happened, and what steps can be taken to prevent it from happening in the future.

- Bob Gezelter, http://www.rlgsc.com

Re: swapped out

Jon Pinkley — Mon, 16 Mar 2009 21:59:37 GMT

adarsh,

How frequently does this happen?

Does the system eventually recover on its own, or do you have to crash/restart it?

When did you first notice the problem? Was something changed before you noticed the problem?

Is there a pattern? I.e. does it correlate to a specific time of the day, day of the week, or day of the month, possibly when extra work is being done? Does it occur when doing a specific job (for example backups of RdB)?

What type of system is this?

$ write sys$output f$getsyi("HW_NAME")

What does the following command show?

$ show memory/all/full

Please paste the output of the above commands into notepad, save as a .txt file, and attach to a note here.

Does the system eventually recover on its own, or do you have to crash/restart it?

If you are resorting to restarting the system, a good crash dump forced while the system is thrashing should be able to provide some useful info.

Alternatively if the system does eventually recover on its own, turning on image accounting so it will be active the next time the system experiences problems will at least let you look that the things that exited just before the system recovered.

Once the system's memory is overcommitted, and it has to use the page/swap files to "warehouse" processes memory, things quickly become unbearably slow, as the system is getting little real work done. It spends its resources trying to preserve to contents of users working sets by copying memory to and from the page/swap files, and disk access is orders of magnitude slower than memory access.

Jon

Re: swapped out

Hein van den Heuvel — Mon, 16 Mar 2009 22:20:37 GMT

>> that problem i have is that i cant log in the system to see what is going on when it starts to freeze.

See if DecAMDS is available for 7.3 and install that. With that you can access the box at driver level, below the stuck processed, from an other system (OpenVMS or Windoze)

>> how can i know what is chewing up the memory after i restart the system and its working fine.

Force a crashdump (google!), analyze, or rent someone to analyze for you.

hth,
Hein.

Re: swapped out

RBrown_1 — Tue, 17 Mar 2009 00:08:09 GMT

I think you also have to ask if it is reasonable to have this many processes active.

Do you have multiple copies of the the same process? Is it supposed to work that way? Or are more started because of a problem in the application (creating the resource problem)? Or because response time was so bad that the user / application started another one (because of a resource problem)?

Re: swapped out

adarsh_4 — Tue, 17 Mar 2009 01:07:25 GMT

COMPAQ AlphaServer DS20E 666 MHz is the system specification. please also find attached the result of show memory/all/full

Re: swapped out

Hein van den Heuvel — Tue, 17 Mar 2009 01:31:13 GMT

Hmm,

1) It only has 1 GB memory. While that may be enough, it is not worth your time, our time, or the end customer aggrevation to even think about it. Double that up... at least!
Rip out those small DIMMS and stick in some good-sized ones. HP, Ebay, islanco.com what-ever it takes. Go. Now!

2) You don't indicate when the SHOW MEM was taken, but I guess there was not problem at that time, with 100 processes happily in memory. Keep running this report once a day or aonce every few hours to see if the system 'leaks' memory somewhere.

Personally I like T4 for that, but can not be bother to reflect on whether that will work on V7.3 unless a serious compensation package is in place :-). Upgrade that box to 7.3-2 already!

3) This appears to be a system with a database with its own caching. As such, the XFC has little value to add but burned almost 10% of the memory trying. Witness:
In use (MBytes) 98.57 Percentage Read I/Os 91%
Read hit rate 21% Write hit rate 0%

So contrary to classic advice... tune it DOWN to a max of for example 30MB, monitor and re-tune. It's a dynamic sysgen param, at least it is on current OpenVMS versions.
(And turn it back to 100MB when you get more memory!)

4) Nonpaged Dynamic Memory ...Maximum Size (MB) 135.42

That's a good bit if memory for a tiny box. It may well be correct, or it may indicate a problem. Is it creeping up?
Use ANAL/SYS poke at the problem with commands like SDA> SHWO POOL/PAGES/SUM
Where is the bulk allocated (LNM? KFE? ... )

Good luck,

Hein van den Heuvel ( at gmail dot com )
HvdH Performance Consulting

Re: swapped out

Hoff — Tue, 17 Mar 2009 02:36:39 GMT

Repost: ITRC glitched.

Current laptops feature two and often four gigabytes.

As for options available to you here, there are several AlphaServer DS20e boxes with four gigabytes of memory and a full system configuration for under US$900 for the full server configuration available on eBay. International shipping to your location (in the Falklands/Maldives or the South Georgia Islands?) probably won't be cheap, but you'll also end up with spare parts to match your current Alpha box. (The AlphaServer DS20e is rather heavy, though.) Or work with somebody to buy one of these boxes and ship just the DIMMs. Or purchase the DIMMs directly; the going street price for DIMMs looks to be around US$250 for a four-slot gigabyte. Prices are higher, if you need assurances.

As for alternatives, a very nice replacement HP rx2600 series Integrity Itanium box can be had for US$200. A DVD drive, SCSI disks, software licenses and media, and arraigning international shipping, are extra.

Or talk to your local HP rep or HP reseller, and start looking to upgrade the memory (and various other pieces as needed) on this box, or upgrade to a newer Alpha or to a new Integrity server.

Otherwise, start removing active users and start removing active processes from this box. Spread the load around the clock using off-hour batch and whatever you can manage. This AlphaServer DS20e hardware is grievously overloaded, in its current configuration.

Re: swapped out

Hakan Zanderau ( Anders — Tue, 17 Mar 2009 05:21:47 GMT

I hate to get involved in an already ongoing discussion, but.....

I have once investigated a system where things went veeeery slow ( a lot of outswapped processes ) and the customer bought an extra Gb of memory, but it didn't help.

It was the SYSGENparameter BALSETCNT that was to low. ( and it was out of synch with MAXPROCESSCNT.

Sys_Parameters

BALSETCNT

BALSETCNT sets the number of balance set slots in the system page
table. Each memory-resident working set requires one balance set
slot.

You can monitor the active system with the DCL command SHOW
MEMORY or the MONITOR PROCESSES command of the Monitor utility to
determine the actual maximum number of working sets in memory. If
this number is significantly lower than the value of BALSETCNT,
this parameter value could be lowered. If all balance set slots
are being used, raise the value of BALSETCNT.

Never set BALSETCNT to a value higher than 2 less than
MAXPROCESSCNT. If physical memory is a significant system
constraint, consider lowering this value even further. However,
if your system runs with a number of processes nearly equal to
MAXPROCESSCNT, lowering BALSETCNT forces swapping to occur, which
can affect system performance.

BALSETCNT is no longer a strict setting of the number of
processes that might be resident in memory. The swapper tries
to reduce the number of resident processes down to BALSETCNT.
However, if the total number of active processes and processes
that have disabled swapping exceeds BALSETCNT, the swapper does
not force processes out of memory just to meet the BALSETCNT
setting.

Note that on VAX systems, virtual balance slots (VBS) can affect
the values of BALSETCNT and MAXPROCESSCNT.

BALSETCNT is an AUTOGEN, GEN, DYNAMIC, and MAJOR parameter.

fwiw,

Hakan

Re: swapped out

Hakan Zanderau ( Anders — Tue, 17 Mar 2009 06:16:19 GMT

The previous output was from VMS 8.3

This is VMS 7.3

BALSETCNT (A,G,M)
BALSETCNT sets the number of balance set slots in the system page table. Each memory-resident working set requires one balance set slot.

You can monitor the active system with the DCL command SHOW MEMORY or the MONITOR PROCESSES command of the Monitor utility to determine the actual maximum number of working sets in memory. If this number is significantly lower than the value of BALSETCNT, this parameter value may be lowered. If all balance set slots are being used, raise the value of BALSETCNT.

Never set BALSETCNT to a value higher than 2 less than MAXPROCESSCNT. If physical memory is a significant system constraint, consider lowering this value even further. However, if your system runs with a number of processes nearly equal to MAXPROCESSCNT, lowering BALSETCNT will force swapping to occur, which can affect system performance. Note that virtual balance slots (VBS) can affect the values of BALSETCNT and MAXPROCESSCNT.

Hakan

Re: swapped out

Jon Pinkley — Tue, 17 Mar 2009 07:17:39 GMT

Here's the version I intended to paste...

Hakan makes a good point. Setting BALSETCNT to a low value can cause swapping when there is plenty of free memory.

However, that does not seem to be the case here, as evidenced by

Slot Usage (slots): Total Free Resident Swapped
Process Entry Slots 555 452 102 1
Balance Set Slots 553 452 100 1

Perhaps MAXPROCESSCNT is higher than it should be for a system with 1GB of real memory, but it depends on what those processes are doing.

adarsh,

When was the last time you ran Autogen with feedback? It may recommend lowering the MAXPROCESSCNT if the system hasn't seen a high number of concurrent processes. Don't forget to save feedback after the system has been up for at least 24 hours, and has been under a typical load, in other words if you don't do processing on the weekends, and you rebooted on Friday night, saving feedback on Sunday wouldn't give an accurate representation of the load. Ideally the system should have been through a high load period prior to saving feedback.

As Hein suggested, you really should strongly consider an upgrade to at least 7.3-2. I am not sure about this, but I believed the XFC had some "bugs" in the code that freed memory that have been improved in more recent versions, although that fix may be from the 8.3 era.

This from the vms73_xfc-v0200.txt file http://vms.tuwien.ac.at/deccampus/patches/vms73_xfc-v0200.txt

5 PROBLEMS ADDRESSED IN VMS73_XFC-V0200 KIT

o Because of numerous problems reported against XFC in V7.3,
customers were instructed to disable XFC for V7.3 until these
issues could be addressed. This kit fixes all problems with
XFC reported by customers as of 16 July 2002. Once this kit is
installed XFC can be safely re-enabled. Note that all these
issues were corrected for OpenVMS V7.3-1.

Here's a thread describing problems in 7.3-2 where XFC was reluctant to free memory.

http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1231011

This described what the intended behavior is:

http://www.arnes.si/news/archive/si.comp.os.vms/msg00412.html

It is possible that the XFC was holding more (up to 1/2 of physical memory) and not releasing it when it should have. So I would strongly consider following Hein's advice to restrict the maximum amount of memory that the XFC can allocate, at least until you determine that is not a contributing factor.

Increasing your memory will with high probability boost your performance more than any other hardware upgrade, i.e. I would expect it to help more than either a faster CPU or faster disk storage.

Jon

Re: swapped out

Robert Gezelter — Tue, 17 Mar 2009 12:58:02 GMT

adarsh,

With all due respect, I will differ somewhat with the suggestion from both Hein and Hoff to simply increase memory.

Their argument is correct on the aspect of cost. However, I have seen more than my share of cases where increasing resources merely made a problem happen less, not disappear completely.

It could very well be that your environment simply needs more memory. However, as has been pointed out by myself, Hakan, Jon, and others, there are quite a few possibilities of incorrect configuration that could be giving rise to the problem.

My normal recommendation to clients is a quote from one of the final pages in Michael Crichton's "Andromeda Strain", to wit: "... That's the important thing. That we understand."

- Bob Gezelter, http://www.rlgsc.com

Re: swapped out

Hoff — Tue, 17 Mar 2009 15:47:00 GMT

There's the well-known "even virtual memory systems need physical memory" command and (as was used on signage at DECUS panels in years past) "buy more memory".

Is adding more physical memory a performance panacea? No. Beyond its use for overtly memory-starved systems (such as this one), simply more memory can compensate legions of tuning and configuration errors.

A gigabyte is often adequate for an Alpha workstation, but much less appropriate with an AlphaServer DS20e with this much activity. This box almost certainly has XFC caching disabled, and has the overhead of the PFRAT-based trimming. Slow image activations due to the lack of free memory. Among other issues.

There wouldn't be questions posted here if you were able to off-load this box. Which means you're going to be looking at an incremental (memory) upgrade as a short-term problem for this (overloaded) server, or (better, but also more expensive) a more substantial server upgrade.

I don't know what the going business rates for this sort of a specialization might be in your area of the world, but I doubt that the purchase (US$250 or so?) and installation price ($?) of a gigabyte memory upgrade here would cover what HP or a regional specialist performing formal OpenVMS troubleshooting and a performance review of this box would tend to charge. (Does this box need performance tuning? Likely yes. But right now, the box needs to be offloaded and needs more physical memory more than it needs to be tuned.)

ps: once you add a new bank of DIMMs into this box, do remember to change the SYS$SYSTEM:MODPARAMS.DAT setting for the physical memory value to reflect the additional memory (PHYSICAL_MEMORY), and run an @SYS$UPDATE:AUTOGEN pass with a reboot.