insvirmem on sort

[Jan van den Ende]

Willem,

in almost all practical cases nowadays, you will not even HAVE anything in the pagefile (apart from something like a reference to your process header), but you WILL have exhausted your RIGHTS for pagefile use.
If your system is not configured with a MUCH too small podified page list, ALL your pages in excess of WSEXTENT are still in memory; only NOT on your WS list, but on the Modified list.
And since the ModifiedPageWriter MUST be able to write them to the pagefile IMMEDIATELY if needs arises, they are already accounted against your pagefilequota.
Perhaps easiest to visualise if you consider the ModifiedPageList as some kind of LevelOne pagefile, or pagefile cache.

.. to get some idea of the extend of this feature, look at $ SHOW MEM/FIL, and watch for PageFile Used vs. Reservable. The latter may (on bigger systems with many processes)well get (very) negative, signifying that IF ALL currently running processes would simultaniously use ALL allowed pagefilequota, the pagefile would be too small. But that condition would be rare indeed.


Jan

[Bojan Nemec]

Steph,

The response was already given by Ian and Jan. I suggest you that you also look at other things that they mentioned.

And take my appologies for my "telegraphic" response. I received a telephon call yust after opening the reply to yours question and I went out. Now I see that other guys give you all the responses.

Bojan

[Hein van den Heuvel]

Guillou>> On the 2100 we didn't have this kind of problem.

Nah, probably nothing to do with 2100 vs DS10, more with 6.2 vs 7.3

Two potential reasons:
1) "LRL" as memtioned in the wizard article. This can cause explosive virtual memory needs. Can be fixed on a per file basis with SET FILE/ATT=LRL=

2) System/account tuning. Sort is very aware of memory and quota settings trying to use memory sort if it can. Of the new system setup 'lied' about those, then it can draw the wrong conclusion. Check DOC and Google and wizard for earlier descriptions.
Note: It can handle the error.

You report "lib-f-insvirmem, insufficient virtual memory error", but that is the 'second line' is is not? What was the real sort status? If it is just 'Warning' then the sort succeeded and it is just complaing about being mis-informed.
If is is fatal, then it is not this qutao mismatch I refer to.

General recommandation for sort: Make sure page file quota is at least 3x working set extent.

3) Is the file to be sorted per chance a VFC file? There is a 7.3-1 bug in this space. FInd yourself a more recent sort (V07-006)! (Btw sort from 7.3-2 could work on 7.3-1, or a patch kit).



Ian> often this sort of thing is due to higher resource requirements for processes in newer versions of VMS.

Like you say, 'often'. But I am pretty sure this is not the case this time.

Jan> If you don't, you will probably run into many more of those errors, not only on SORT but on a lot of functions, in all kinds of programs.


Often yes, but not this time. SORT is special.

Willem> ALL above on pagefilequota is true - but one thing should not be forgotten: the actual size of the pagefile.

Nah, Sort is special. Sort should NOT actually need the pagefile ever. It tries to avoid that. It knows how to use less memory by switching to workfiles. Now if you lied to it claiming there is more (virtual) memory then there really is.... Plus, like Jan writes, it is not the file, but the reservations for the files.



Guillou,

If you need further help or explanations, and to satisfyl our curiosity, please show us DIR/FULL for a failing file, and accoutn information for the two systems: SHOW WORK
and/or:

$ write sys$output "WSEXTENT =''F$GETJPI("","WSEXTENT")'"
$ write sys$output "PGFLQUOTA=''F$GETJPI("","PGFLQUOTA")'"


hth,
Hein.

[John Gillings]

The commonest reason for INSVIRMEM from SORT is that it's being executed by a process with WSEXTENT > PGFLQUOTA.

This is telling the system that you can have more real memory than you'd be able to fit in your page file quota - not a good idea!

SORT tries to make maximum use of memory by sizing it's virtual address space according to WSEXTENT. If other processes need memory, your process may need to relinquish some, and when it attempts to write the excess to the page file discovers there isn't enough room -> INSVIRMEM.

How can this happen? The most likely reason is the change in AUTOGEN which now sets PQL_MWSEXTENT = WSMAX, which effectively means your WSEXTENT values in SYSUAF are meaningless (overridden by PQL_MWSEXTENT).

So, just make sure PGFLQUOTAs are higher than WSMAX and all should be OK. Setting PQL_MPGFLQUOTA is a good idea. Also make sure you have plenty of page file available.

In general you should look at ALL your quotas - remember that there's lots more memory around now, and it's much cheaper. You paid for the resources, you may as well make sure your processes make maximum use of them.

[Martin P.J. Zinser]

Hi Steph,

for diagnosing quota problems a small DCL from
one of the older Freeware CDs can be pretty
handy, since it shows the quota usage for an executing process.

The DCL can be found at

http://zinser.no-ip.info/vms/sw/show_quota.htmlx

Greetings, Martin

[Wim Van den Wyngaert]

Guillou,

Check http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=673457

This is what I was talking about. You change something and sort no longer works.

But if I remember correctly this is only a warning, your sort should have worked (but maybe slowly).

Wim

[Jan van den Ende]

John,

since, as you write, AUTOGEN now sets PQL_MWSEXTENT to equal WSMAX, and, since it is not a good idea to have WSEXTENT exceeding PGFLQUOTA, would it not have been in the normal line of consistency we have come to expect from Engeneering, to have AUTOGEN also setting PQL_MPGFLQUOTA to WSMAX?

just wondering...

Jan

[John Gillings]

Jan,

>would it not have been in the normal
>line of consistency we have come to
>expect from Engeneering, to have
>AUTOGEN also setting PQL_MPGFLQUOTA
>to WSMAX?

What you say makes sense. However, I suspect the reason it isn't done is that AUTOGEN knows WSMAX and the physical size of memory. Therefore it knows that PQL_MWSEXTENT=WSMAX is "safe" - worst case is a process gets INSVIRMEM. However, AUTOGEN doesn't necessarily know that PGFLQUOTA=WSMAX is safe. It could be the system has constrained page files, and increasing PGFLQUOTA might allow a non-privileged user to fill a pagefile and hang the system.

In general, I've always considered anything that depends on PQL parameters to be a bug, because it's "invisible". I've seen many cases where moving an application from system to system suddenly fails because the dependence was overlooked. Ideally, nothing will ever obtain a quota from PQL_D or PQL_M parameters.

However, the issue engineering was trying to fix here was that too many systems were being artificially constrained by "VAX 11-750 heritage SYSUAF values". Systems with huge amounts of RAM were performing poorly because system managers didn't seem to be comprehending that WSEXTENT=8192 is no longer a lot of memory.

The multi layered working set model was required when memory was a scarce resource, but these days it's a hindrence, so OpenVMS has removed the top layer.

About the only real problem we've seen is INSVIRMEM from SORT. These are really only warnings as the SORT has completed. It's also very easy to diagnose and correct. Think of it as a warning to system managers that their quotas need adjusting upwards.

A system manager may choose to do this by setting PQL_MPGFLQUOTA, provided they know they have sufficient physical page file space to support it.

[Jan van den Ende]

Thanks John.

So, still totally VMS: Very worried that Murphy's Law might apply, and trying to cope in advance. If need be, prefer higher occurrance of smaller accidents over any chance of disaster.

Good to learn.

Jan