Operating System - OpenVMS
cancel
Showing results for 
Search instead for 
Did you mean: 

insvirmem on sort

 
Guillou_2
Frequent Advisor

insvirmem on sort

Hi,

Since the upgrade of our server alpha 2100 vms6.2 on a ds10 vms7.3-1, on a sort we have error message like "sort-w-syserror, system service error" followed by "lib-f-insvirmem, insufficient virtual memory error".
On the 2100 we didn't have this kind of problem.

Maybe someone could help me to understand this behaviour and lead me to the solution?

regards

Steph.
18 REPLIES
Bojan Nemec
Honored Contributor

Re: insvirmem on sort

Hi,

As the first try to increase the /PGFLQUOTA parameter in UAF.

Bojan
Volker Halle
Honored Contributor

Re: insvirmem on sort

Steph,

a quick Google search for +sort +insvirmem turns up a couple of articles. You may want to read the following article from Ask The Wizard:

http://h71000.www7.hp.com/wizard/wiz_4523.html

Volker.
Ian Miller.
Honored Contributor

Re: insvirmem on sort

often this sort of thing is due to higher resource requirements for processes in newer versions of VMS. The quotas for existing accounts are not changed during the upgrade. Some quotas are affectively forced upwards by the PQL_M system parameters but not all. I expect the relevent quota to be increased in this case is PGFLQUOTA. Check that the value is at least the same as the system paramter PQL_DPGFLQUOTA then increase from there.

There are various tools to monitor realtime process quota use including AMDS,Availability Manager and DCL procedures that use GETJPI
____________________
Purely Personal Opinion
Kris Clippeleyr
Honored Contributor

Re: insvirmem on sort

Steph,

As Bojan stated, try increasing PGFLQUOTA in SYSUAF.DAT. Maybe it can also be necessary to increase SYSGEN parameters PQL_MPGFLQUOTA & PQL_DPGFLQUOTA.

Regards,

Kris
I'm gonna hit the highway like a battering ram on a silver-black phantom bike...
Jan van den Ende
Honored Contributor

Re: insvirmem on sort

Guillou,

togethet with Bojans advice (which I agree with), you may also re-evalute the WSQUOTA params for your user accounts, maybe SYSGEN WSMAX as well.

If you don't, you will probably run into many more of those errors, not only on SORT but on a lot of functions, in all kinds of programs.

Moving from V6.x to V7.x (and even from V7.2 to V7.3) many functions are made 'to fit better' in today's bigger memory systems, and now make use of the possible performance gains to be reached at the price of being more liberal in memory use.
At current memory prices that makes sense!

Jan
Don't rust yours pelled jacker to fine doll missed aches.
Guillou_2
Frequent Advisor

Re: insvirmem on sort

Hi Bojan,

Thanks for the quick reply, i increase that quota from 100000 to 250000, it seems to solve the problem...but i doesn't like applying a solution without really understanding the origin of a problem, can you explain me why i have this kind of behaviour on the new server?

Steph.
Ian Miller.
Honored Contributor

Re: insvirmem on sort

I think the from Jan covers why
____________________
Purely Personal Opinion
Willem Grooters
Honored Contributor

Re: insvirmem on sort

ALL above on pagefilequota is true - but one thing should not be forgotten: the actual size of the pagefile.
I tend to think that if a process's pagefilequota would be exhausted, the message would be "Exceeded quota", not "Insuficient Virtual memory"; that would not really mean "You exhausted your quota" but merely "You exhausted the page file" - you would have no more empty, phyisical space available.
But I make be wrong here.

Willem
Willem Grooters
OpenVMS Developer & System Manager
Ian Miller.
Honored Contributor

Re: insvirmem on sort

unfourtunally the exceeded quota error message is not used for some quotas. If you actually fill the pagefill then the system usually hangs. If you are lucky there is a message on the console telling you the pagefile is nearly full.
____________________
Purely Personal Opinion
Jan van den Ende
Honored Contributor

Re: insvirmem on sort

Willem,

in almost all practical cases nowadays, you will not even HAVE anything in the pagefile (apart from something like a reference to your process header), but you WILL have exhausted your RIGHTS for pagefile use.
If your system is not configured with a MUCH too small podified page list, ALL your pages in excess of WSEXTENT are still in memory; only NOT on your WS list, but on the Modified list.
And since the ModifiedPageWriter MUST be able to write them to the pagefile IMMEDIATELY if needs arises, they are already accounted against your pagefilequota.
Perhaps easiest to visualise if you consider the ModifiedPageList as some kind of LevelOne pagefile, or pagefile cache.

.. to get some idea of the extend of this feature, look at $ SHOW MEM/FIL, and watch for PageFile Used vs. Reservable. The latter may (on bigger systems with many processes)well get (very) negative, signifying that IF ALL currently running processes would simultaniously use ALL allowed pagefilequota, the pagefile would be too small. But that condition would be rare indeed.


Jan
Don't rust yours pelled jacker to fine doll missed aches.
Bojan Nemec
Honored Contributor

Re: insvirmem on sort

Steph,

The response was already given by Ian and Jan. I suggest you that you also look at other things that they mentioned.

And take my appologies for my "telegraphic" response. I received a telephon call yust after opening the reply to yours question and I went out. Now I see that other guys give you all the responses.

Bojan
Hein van den Heuvel
Honored Contributor

Re: insvirmem on sort

Guillou>> On the 2100 we didn't have this kind of problem.

Nah, probably nothing to do with 2100 vs DS10, more with 6.2 vs 7.3

Two potential reasons:
1) "LRL" as memtioned in the wizard article. This can cause explosive virtual memory needs. Can be fixed on a per file basis with SET FILE/ATT=LRL=

2) System/account tuning. Sort is very aware of memory and quota settings trying to use memory sort if it can. Of the new system setup 'lied' about those, then it can draw the wrong conclusion. Check DOC and Google and wizard for earlier descriptions.
Note: It can handle the error.

You report "lib-f-insvirmem, insufficient virtual memory error", but that is the 'second line' is is not? What was the real sort status? If it is just 'Warning' then the sort succeeded and it is just complaing about being mis-informed.
If is is fatal, then it is not this qutao mismatch I refer to.

General recommandation for sort: Make sure page file quota is at least 3x working set extent.

3) Is the file to be sorted per chance a VFC file? There is a 7.3-1 bug in this space. FInd yourself a more recent sort (V07-006)! (Btw sort from 7.3-2 could work on 7.3-1, or a patch kit).



Ian> often this sort of thing is due to higher resource requirements for processes in newer versions of VMS.

Like you say, 'often'. But I am pretty sure this is not the case this time.

Jan> If you don't, you will probably run into many more of those errors, not only on SORT but on a lot of functions, in all kinds of programs.


Often yes, but not this time. SORT is special.

Willem> ALL above on pagefilequota is true - but one thing should not be forgotten: the actual size of the pagefile.

Nah, Sort is special. Sort should NOT actually need the pagefile ever. It tries to avoid that. It knows how to use less memory by switching to workfiles. Now if you lied to it claiming there is more (virtual) memory then there really is.... Plus, like Jan writes, it is not the file, but the reservations for the files.



Guillou,

If you need further help or explanations, and to satisfyl our curiosity, please show us DIR/FULL for a failing file, and accoutn information for the two systems: SHOW WORK
and/or:

$ write sys$output "WSEXTENT =''F$GETJPI("","WSEXTENT")'"
$ write sys$output "PGFLQUOTA=''F$GETJPI("","PGFLQUOTA")'"


hth,
Hein.
John Gillings
Honored Contributor

Re: insvirmem on sort

The commonest reason for INSVIRMEM from SORT is that it's being executed by a process with WSEXTENT > PGFLQUOTA.

This is telling the system that you can have more real memory than you'd be able to fit in your page file quota - not a good idea!

SORT tries to make maximum use of memory by sizing it's virtual address space according to WSEXTENT. If other processes need memory, your process may need to relinquish some, and when it attempts to write the excess to the page file discovers there isn't enough room -> INSVIRMEM.

How can this happen? The most likely reason is the change in AUTOGEN which now sets PQL_MWSEXTENT = WSMAX, which effectively means your WSEXTENT values in SYSUAF are meaningless (overridden by PQL_MWSEXTENT).

So, just make sure PGFLQUOTAs are higher than WSMAX and all should be OK. Setting PQL_MPGFLQUOTA is a good idea. Also make sure you have plenty of page file available.

In general you should look at ALL your quotas - remember that there's lots more memory around now, and it's much cheaper. You paid for the resources, you may as well make sure your processes make maximum use of them.
A crucible of informative mistakes
Martin P.J. Zinser
Honored Contributor

Re: insvirmem on sort

Hi Steph,

for diagnosing quota problems a small DCL from
one of the older Freeware CDs can be pretty
handy, since it shows the quota usage for an executing process.

The DCL can be found at

http://zinser.no-ip.info/vms/sw/show_quota.htmlx

Greetings, Martin
Wim Van den Wyngaert
Honored Contributor

Re: insvirmem on sort

Guillou,

Check http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=673457

This is what I was talking about. You change something and sort no longer works.

But if I remember correctly this is only a warning, your sort should have worked (but maybe slowly).

Wim
Wim
Jan van den Ende
Honored Contributor

Re: insvirmem on sort

John,

since, as you write, AUTOGEN now sets PQL_MWSEXTENT to equal WSMAX, and, since it is not a good idea to have WSEXTENT exceeding PGFLQUOTA, would it not have been in the normal line of consistency we have come to expect from Engeneering, to have AUTOGEN also setting PQL_MPGFLQUOTA to WSMAX?

just wondering...

Jan
Don't rust yours pelled jacker to fine doll missed aches.
John Gillings
Honored Contributor

Re: insvirmem on sort

Jan,

>would it not have been in the normal
>line of consistency we have come to
>expect from Engeneering, to have
>AUTOGEN also setting PQL_MPGFLQUOTA
>to WSMAX?

What you say makes sense. However, I suspect the reason it isn't done is that AUTOGEN knows WSMAX and the physical size of memory. Therefore it knows that PQL_MWSEXTENT=WSMAX is "safe" - worst case is a process gets INSVIRMEM. However, AUTOGEN doesn't necessarily know that PGFLQUOTA=WSMAX is safe. It could be the system has constrained page files, and increasing PGFLQUOTA might allow a non-privileged user to fill a pagefile and hang the system.

In general, I've always considered anything that depends on PQL parameters to be a bug, because it's "invisible". I've seen many cases where moving an application from system to system suddenly fails because the dependence was overlooked. Ideally, nothing will ever obtain a quota from PQL_D or PQL_M parameters.

However, the issue engineering was trying to fix here was that too many systems were being artificially constrained by "VAX 11-750 heritage SYSUAF values". Systems with huge amounts of RAM were performing poorly because system managers didn't seem to be comprehending that WSEXTENT=8192 is no longer a lot of memory.

The multi layered working set model was required when memory was a scarce resource, but these days it's a hindrence, so OpenVMS has removed the top layer.

About the only real problem we've seen is INSVIRMEM from SORT. These are really only warnings as the SORT has completed. It's also very easy to diagnose and correct. Think of it as a warning to system managers that their quotas need adjusting upwards.

A system manager may choose to do this by setting PQL_MPGFLQUOTA, provided they know they have sufficient physical page file space to support it.
A crucible of informative mistakes
Jan van den Ende
Honored Contributor

Re: insvirmem on sort

Thanks John.

So, still totally VMS: Very worried that Murphy's Law might apply, and trying to cope in advance. If need be, prefer higher occurrance of smaller accidents over any chance of disaster.

Good to learn.

Jan
Don't rust yours pelled jacker to fine doll missed aches.