topic Re: Virtual Address Space is Full in Operating System - OpenVMS

Virtual Address Space is Full

dabaco — Fri, 20 Feb 2009 16:33:52 GMT

Hello,

I have encountered a problem on VMS V7.3-2 where we are connecting through java to JNI to retrieve global section information. After a while (7 or more hours usually) we get an error message 580:

%SYSTEM-F-VASFULL, virtual address space is full

We have checked the quota information on the user who owns the processes, and on the processes themselves and it doesn't seem that any quotas are within 80% being reached.

We have increased the PGFLQUOTA, and it seemed to have no effect.

Also referencing another wizard forum (http://h71000.www7.hp.com/wizard/wiz_0914.html), I looked at the VIRTUALPAGECNT attribute, but it had a datatype of 'obsolete'.

This is reproducible on nearly any vms node we have in house, and at client sites.

I'm looking for some new ideas on how to go about figuring out what the problem is, and how to go about resolving it. Currently we are investigating quota information, and the native code itself.

I'm not sure what information can best help in addressing the problem. Please let me know what I can provide and I will gladly assist.

Thanks,
Jeff

Re: Virtual Address Space is Full

dabaco — Fri, 20 Feb 2009 16:43:21 GMT

This is an alpha system.

AlphaServer DS25 SYSTEM NYMC03 (VMS V7.3-2)

Re: Virtual Address Space is Full

Robert Gezelter — Fri, 20 Feb 2009 16:44:43 GMT

Jeff,

Indeed, VIRTUALPAGECNT is obsolete effective Version 7.0 on Alpha (paraphrased from the HELP text within SYSGEN).

Please check ALL parameters relating to paging and the page files.

In simple terms, it would appear that your application (from the description, probably the C/C++ code accessed through JNI) has a memory leak. This is the underlying problem that needs to be corrected.

Which parameter is actually causing the VASFULL error is actually fairly irrelevant. The problem will recur, albeit at a larger elapsed time.

- Bob Gezelter, http://www.rlgsc.com

Re: Virtual Address Space is Full

Steven Schweda — Fri, 20 Feb 2009 16:46:49 GMT

You looked at "HELP /MESS VASFULL", right?
(I didn't see a magic bullet there, however.)

Re: Virtual Address Space is Full

Hein van den Heuvel — Fri, 20 Feb 2009 16:49:22 GMT

Most likely the process is truly running out of its 1GB P0 virtual address space.

Check with SYS$GETJPI or DCL form an other process using:

$ write sys$output f$getjpi(pid,"FREP0VA ")

The HARD WALL is at 1GB = 3FFF.FFFF

Might as well check FREP1VA.

The typical cause is 'address space laddering'.
Repeatedlty Mapping, unmapping, and re-mapping global section, but failing to re-use the prior addressing windows, using higher and higher (EXPREG!) address until it hurts. The total pages used my be reasonable (50 mmb, 100MB, whatever), tehre is just no place to put it.

There are techniques to limit this (LIFO!), but a simple application restart is most effective.

Best regards,
Hein van den Heuvel ( at gmail dot com )
HvdH Performance Consulting

Re: Virtual Address Space is Full

Hein van den Heuvel — Fri, 20 Feb 2009 16:55:06 GMT

Oh, and in case you wondered...
there is NO QUOTA that can get you beyond 1GB.
But please check with FREP0VA whether that is indeed the problem.

Avoidance techniques include
- Keep you global sections mapped
- reviewing RMS Settings
- review RMS global buffer and STATS usage
- review LINK option IMGIOCNT
- review Java memory settings.. pick a HIGH minimum to avoid nickle and diming the process VM usage up and up

Hein.

Re: Virtual Address Space is Full

John Gillings — Sun, 22 Feb 2009 21:03:20 GMT

Jeff,
I also suspect this is a memory leak. With Java you also have the added complication of the garbage collector, which can confuse things.

First thing I'd recommend is to REDUCE the maximum heap size for the application. Depending on application behaviour and garbage collector policy, this can paradoxically make Java run faster, and may fix memory full conditions!

Second, as a diagnostic, I'd also recommend REDUCING PGFLQUOTA to a very small value. Hopefully this will reduce your 7 hour lifetime to a few minutes, which should make it easier to debug.

Monitor the process PGFLQUOTA and, as Hein has suggested, FREP0VA.

Re: Virtual Address Space is Full

John Gillings — Sun, 22 Feb 2009 21:24:47 GMT

I've attached a procedure to gather F$GETJPI statistics for a process, or group of processes in T4 format.

At its simplest execute from another process as:

$ @T4JPI

See comments for complete instructions.

Re: Virtual Address Space is Full

Hein van den Heuvel — Sun, 22 Feb 2009 22:57:16 GMT

Nice script John!
Good template for such work.

Jeff, in case you had not used T4, check out TLVIZ to visualize the CSV file.
Start here: http://h71000.www7.hp.com/openvms/products/t4

Attached a self-resubmitting batch job to watch top-10 FREP0VA every 10 minutes (or whatever you choose) and send an Email when a treshold is crossed, hopefully before the process dies an uncontrolled death.

Cheers,
Hein.

Re: Virtual Address Space is Full

dabaco — Tue, 24 Feb 2009 15:24:18 GMT

Thank you all for your assistance. You've gotten us to a point where we can continue moving forward in resolving the issue.

Jeff