Increase Working Set

[Jakes Bruwer]

I have a Batch process which runs under the BACKUPS user account.

This process does backups to a TSM server.

The process starts off running very nicely; lots of I/O and lots of CPU

Then as some stage, the process just slows down to a snail’s pace or less...

I noticed that this happens every time the process’s working set reaches 131456.

                          Process BATCH_19                    

State               HIB                 Working set             131456

Cur/base priority   6/4                 Virtual pages           312471

Current PC FFFFFFFF.805FE5B0            CPU time         0 00:31:35.57

This corresponds to the WSMAX sysgen parameter:

SYSGEN>  SHOW WSMAX

Parameter Name            Current    Default     Min.       Max.   Unit  D

--------------            -------    -------   -------    -------  ----  -

WSMAX                     2103296     131072     16384  134217728 Pagelets

internal value            131456       8192      1024    8388608 Pages  

The parameters in UAF is as follows for BACKUPS account:

WSdef:       600000

WSquo:       600000

WSextent:     49152

Pgflquo:    3000000

My problem is that this backup process just never completes successfully.

It doesn’t crash, it just carries on for weeks.

Will increasing the working set for this process resolve this problem, and how can I increase the working set?

Thanks

Jakes

[Duncan Morris]

Hi Jakes,

some more information would prove useful - version of VMS, architecture VAX/Alpha/IA64 ?

When your process wants to increase working set is there plenty of memory available ($ SHOW MEM)?

You also need to read up on the meaning of WSDEF/WSQUOTA/WSEXTENT as this account has a lower WSEXTENT than either WDEF/WSQUOTA.

http://h71000.www7.hp.com/doc/73final/6491/6491pro_006.html#understand_memory

See this help from Authorize:

ADD

  /WSEXTENT

        /WSEXTENT=value

     Specifies the working set maximum. This represents the maximum
     amount of physical memory allowed to the process. The system
     provides memory to a process beyond its working set quota only
     when it has excess free pages. The additional memory is recalled
     by the system if needed.

     The value is an integer equal to or greater than WSQUOTA. By
     default, the value is 16384 pagelets on Alpha and Integrity
     server systems. The value cannot be greater than WSMAX. This
     quota value replaces smaller values of
     PQL_MWSEXTENT.

Are your backups using the native OpenVMS BACKUP command?

 If so, then you can find plenty of good references to recommended working set values and other quotas for the backup account. Read through this article from a recent OpenVMS Technical Journal:

http://h20195.www2.hp.com/v2/GetPDF.aspx/4AA3-7822ENW.pdf

Duncan

[Jakes Bruwer]

Thanks Duncan,

OpenVMS V8.3-1H1 for Integrity.

Physical Memory Usage (bytes):     Total        Free      In Use    Modified 

Main Memory             (GB)                 31.99       15.06       16.85        0.07

I'm not using the native OpenVMS Backup command, I'm using a 3rd party backups solution called Archive Backup Client for OpenVMS. (ABC)

Like you can see above the system has plenty of physical memory available.

I think that I might know what is wrong here, maybe you can verify;

The Sysgen parameter WSMAX is determined by multiplying the system memory with "4" f$getsyi("MEMSIZE")*4

In the system's modparams file I did not put a MIN_WSMAX = 16775164

WSMAX is currently set to 2103296, for some reason.

Calculated as follows:

EDIGTY>d = f$getsyi("MEMSIZE")

EDIGTY>sh sym d  D = 4193791   Hex = 003FFDFF  Octal = 00017776777

EDIGTY>d = f$getsyi("MEMSIZE") * 4

EDIGTY>sh sym d  D = 16775164   Hex = 00FFF7FC  Octal = 00077773774

Adding the MIN_WSMAX = 16775164 to my server's modparams file and running a autogen and reboot should increase the Working set maximum for all processes on the server, including the Backups.

Can this work?

[Duncan Morris]

Jakes,

the response from d = f$getsyi("MEMSIZE") is in pages.

For your IA64 system the page size is 8K, or 16 pagelets.

WSMAX is in pagelets, and so your default calculation is MEMSIZE/4 pages, which equates to MEMSIZE*4 pagelets.

However, AUTOGEN also considers other factors, such as reserved memory allocations, so will come up with a smaller WSMAX.

WSMAX sets the maximum number of pages on a systemwide basis for any working set. WSMAX is calculated as a quarter of the first 32 MB plus a sixteenth of the memory from 32 to 256 MB, plus a sixty-fourth of the memory (if any) above 256 MB.

Have you read about WSEXTENT for the account and amended it first? Changing WSMAX without dealing with the account quotas will not have the desired effect.

Have you read the backup performance PDF? The ABC solution does use the OpenVMS backup verb, so will be afftected by the factors discussed on Page 7 in particular.

Don't rush into forcing a larger WSMAX than AUTOGEN calculates, unless you know precisely what effect it will have on the overall system - sort out the backups account first!

Duncan

[Duncan Morris]

Jakes,

the performance drop may not have anything to do with the account quotas in the first place.

From past experience of the TSM server, performance is great whilst you are writing directly to the disk pool on the server, but when that pool runs out, you are limited by the rate at which the disk pool is being written to the tape storage. Is it possible that there is insufficient disk space on the TSM server to buffer all the output from you ABC backup? 

[Hein van den Heuvel · Accepted solution]

Hi Jakes,

This will have nothing to do with Working Set.

Don't bother investigating that route.

Dollars to donuts that you hit a very large directory at the time of the slowdown.

Thousands of block, tens of throusands of files?

The TSM product has a N-square algoritme to check for Alias files.

On system disks, with large aliased directories (VMS$COMMON) that can save hours of backup/restore by avoiding to backup multipel copies of what is the same file/directory.

On a normal data disk is it a total wast of time. 

Try the backup with /ALIAS=IGNORE

You should contact the vendor.

They are working on, or have a better implementation.

( Jim A @  storserver.com)

Hope this helps,

Hein.

[Duncan Morris]

Good call Hein (as usual!).
I've been on too many other products lately, and had forgotten about that feature.

Duncan

[John Gillings]

Hein's suggestion is far more likely to be fruitful than attempted tuning of working sets.

That said, on a modern OpenVMS system, the WSEXTENT settings in the UAF are usually ignored. Look at WSMAX and PQL_MWSEXTENT. They're usually equal, meaning any process on the system can expand to WSMAX, assuming there is enough memory available. The rather complex WSDEFAULT/WSQUOTA/WSEXTENT/WSMAX mechanism was inherited from a time when memory was much more scarce and needed to be tightly controlled. These days most systems have enough memory capacity to effectively remove the entire top layer.

Even if your process would like more memory than WSMAX, if there's enough memory available, it will tend to circulate between the process working set and the modified page list via soft faults. You only need worry if your process is generating huge numbers of pagefaults. In extreme cases, you might consider boosting WSMAX, setting PQL_MWSEXTENT lower and selectively allowing problem processes to access the higher WSMAX by setting their WSEXTENTs higher in the UAF, but this would be a VERY unusual requirement.