Recommended pagefile/swapfile size for IA64 with v8.3

[Dmitry V. Sinelnikov]

Hi all

I have an IA64 system (HP rx4640, 4Gb RAM ) running under OpenVMS V8.3-1H1. Would like to know if there are any recommended values for swap/page files? At the moment they have the following:

Directory SYS$SYSROOT:[SYSEXE]
PAGEFILE.SYS;1 439.45MB
SWAPFILE.SYS;1 57.22MB

More than sure it’s too small. AFAIK, for Win OS it’s recommended to set pagefile twice as large as RAM size. What about OpenVMS on IA64?

Any feedback is appreciated,

Best regards,
Dmitry Sinelnikov

[Volker Halle · Accepted solution]

Dmitry,

trust AUTOGEN.

Make sure there are no PAGEFILE=0 and SWAPFILE=0 statements in MODPARAMS.DAT and run

$ @SYS$UPDATE:AUTOGEN SAVPARAMS TESTFILES FEEDBACK

AUTOGEN will then tell you about the suggested size of your page and swap file.

Volker.

[Steven Schweda]

> [...] What about OpenVMS on IA64?

Pretty much the same as VMS on any other
hardware: Make them as big as they need to
be.

I'd expect the VMS installation procedure to
create files big enough to let the system
start. After that, everything depends on the
workload. SYS$UPDATE:AUTOGEN.COM can offer
advice on a system which has been running for
a while. INSVIRMEM errors suggest that you
need a bigger page file.

> [...] 4Gb RAM [...]

> PAGEFILE.SYS;1 439.45MB

440MB is not much of a boost over 4GB. But
if all your work fits into 4GB, then you
wouldn't need much.

HELP SHOW MEMORY /FILES

[The Brit]

Dmitry,

It obviously depends on your apps, users, physical memory, etc., however you can monitor your page/swap files in real time, using the command

$ show mem/files
System Memory Resources on 16-OCT-2009 09:53:44.57

Swap File Usage (8KB pages): Index Free Size
DISK$I64_V831H1:[SYS0.SYSEXE]SWAPFILE.SYS
1 6248 6248

Paging File Usage (8KB pages): Index Free Size
DISK$ND17:[SYS0.SYSEXE]PAGEFILE.SYS;1
249 624992 624992
DISK$ND16:[SYS0.SYSEXE]PAGEFILE.SYS;1
250 624992 624992

or use

$ monitor page

and watch the size of the modified list relative to the SYSGEN parameter MPW_HILIMIT

Dave

[Jon Pinkley]

Dmitry,

Having more RAM doesn't necessarily mean you need a bigger page/swap file. Given the same load, if you have more RAM you will actually need less pagefile space.

That said, it is better to have more pagefilee than needed than too little, as the system will eventually hang, if you memory demand exceeds your RAM plus your pagefile.

The pagefile is only used when the demand for memory exceeds what is available. In recent versions of VMS it is possible to run with no pagefile, but there are some old hard coded assumptions that there will be pagefiles around.

Here's an example: SYS$COMMON:[SYS$STARTUP]VMS$CONFIG-050_JOBCTL.COM computes the page file max (50% of total, minimum of 40960) on Alpha, I am not sure if IA64 does something different, I don't have one.

The point is that there is no meaningful correlation between RAM and Pagefile size. The pagefile size is much more dependent on the memory demand in excess of available RAM placed on the system.

As others have stated, let AUTOGEN look at what the demand has been, and let it make a recommendation.

Jon

[Andy Bustamante]

I'd like to remind you to check the size of your dumpfile as well. As Volker points out, set SWAPFILE, PAGEFILE and DUMPFILE to 0 in modparms.dat. This will prevent fragmenting these files from each run of AUTOGEN. See System Manager's Manual vol 2 at http://h71000.www7.hp.com/doc/82final/aa-pv5nj-tk/aa-pv5nj-tk.html for more information.

Andy

[Hoff]

Save FEEDBACK when you SHUTDOWN. Trust AUTOGEN.

Here's a general description of the implementation of virtual memory.

http://labs.hoffmanlabs.com/node/228

On OpenVMS, the pagefile and swapfile and backing section files are how the amount of physical memory that is (apparently) present in the box is increased. So if you find your particular application load needs more physical memory, you either need to increase its working set, or add more physical memory, or you need more pagefile, or some combination.

But as for recommendations, that is dependent on your particular load. Anyone that's performed typical OpenVMS system management operations has run an OpenVMS box with no pagefile, for instance.

[John Gillings]

Dmitry,

Disk space is cheap. Downtime is VERY expensive. Make sure you have sufficient pagefile space that downtime because of virtual address space depletion is effectively impossible. With disk space as cheap as it is consider investing (say) $10 in disk space for page file? Very cheap insurance. There are no simple rules of thumb, since it all depends on your workload.

With modern memory sizes, swap file is probably never used. SWAPFILE.SYS can only be used for swapping. PAGEFILE.SYS can be used for both (but with slightly higher overhead for swapping). Since swapping is likely to be a very rare event, you could go without a swapfile. On the other hand a few MB isn't going to break the bank.


re: Volker

>trust AUTOGEN.
>
>Make sure there are no PAGEFILE=0 and >SWAPFILE=0 statements in MODPARAMS.DAT and >run

Interesting! My advice is exactly the opposite of Volker. Maybe Hoff and Hein can come up with even further conflicting advice? ;-)

I never trust AUTOGEN to size system files, and recommend MODPARAMS.DAT set PAGEFILE=0, SWAPFILE=0 and DUMPFILE=0. This is mainly because AUTOGEN likes to change the file size, whereas I'd rather they kept their allocated size. In particular, I never want AUTOGEN to *reduce* a size I've specified.

Remember, AUTOGEN's job is to make a bootable system, not an optimal one, and it can't guess what your virtual memory requirements will be, and allocation algorithms haven't been updated in years (decades?), time over which the economic trade-offs have changed considerably! I'd guess that when AUTOGEN was last updated, the amount of RAM you have would have been considered a very large disk! The sizes you've quoted are a good example. They're a joke! Just compare the total cost of your system (order ~$K) against the investment in disk space for page and swap space (order cents).

Read the AUTOGEN recommendations, and make sure your allocation is higher (in most cases add at least one 0, if not two!).

All that said, it's not uncommon to see systems these days that don't touch their page file at all. I remember a case where it took a whole week to notice that a non-shadowed system disk had been popped out of its rack. The system ran from resident images and cache, never touching the page or swap files! (I'm still not sure if that's illustrates a good feature of OpenVMS or a very bad one!)

[Hoff]

Folks that are sufficiently familiar with and that are skilled with OpenVMS operations can and variously do lock AUTOGEN out of re-sizing the PAGEFILE and SWAPFILE and DUMPFILE files but - given the question here and given the likely requirement to quickly get the server into the right file size range? Yes, I trust AUTOGEN (particularly with FEEDBACK) to get a server into the right range; as a good first or second-order approximation, particularly as you (or AUTOGEN) learns about the load.

Or yes, if you have several "spare" (RAID-protected would be preferred) disks around, you could dedicate the whole of several of those disks to the PAGEFILE (and a runt swap file, and a dumpfile sized for a reasonable hunk of 4 GiB) and move on. This is the "throw hardware at the problem" approach, and that can be expeditious given disk prices these days.

[Volker Halle]

re: John & PAGEFILE=0

I also suggest to use PAGEFILE=0, SWAPFILE=0 and DUMPFILE=0 in MODAPRAMS.DAT, but to get the TESTFILES suggestions, one has to temporarily disable those lines.

So we are in synch with our suggestions and I should have added the word 'temporarily' to my response.

Volker.

[Dmitry V. Sinelnikov]

Thanks to everyone for the feedback.

TESTFILES FEEDBACK showed that pagefile must be enlarged to 4Gb (was corrected). However swap file is treated as too large and needs to be reduced:

SWAPFILE1_SIZE information (for SWAPFILE1_NAME):
Feedback information.
Old value was 117200, New value is 85500
Maximum observed usage: 0
SWAPFILE1_SIZE will be modified to hold 85500 blocks

Am I right in my understanding that ~60Mb swap file is large enough?

The reason I am asking is the following: this system is intended to be available 24/7 with ~120 services running, each service' image size is 50MB. There won't be 100% constant service usage thus some of the processes will be swapped ('swapped out'). Does that mean swapfile will have store these 'swapped' processes?

Best regards,
Dmitry Sinelnikov

[Volker Halle]

Dmitry,

AUTOGEN judged the swapfile size by current usage (from the saved feedback data) and the system never had to use the swapfile up to now, that's why AUTOGEN tends to decrease it's size.

As John already wrote, the pagefile can also be used for swapping, should that need arise. To estimate the swap file size in advance may be impossible, you can only go by experience of the behaviour of that workload. Process working sets will typically be trimmed, before being swapped out.

Do not bother to try to make your SWAPFILE.SYS file smaller. If there is enough space on your swapfile disk, just increase the size to make you feel better.

Volker.

[John Gillings]

Dmitry,

> Am I right in my understanding that ~60Mb
> swap file is large enough?

An arm waving, simplified explanation of the difference between SWAPFILE.SYS and PAGEFILE.SYS...

Paging is a very general mechanism for moving virtual memory out of RAM and onto disk. It must therefore deal with the allocation and management of varying sized chunks of memory. The internal structure of PAGEFILE.SYS is rather complex.

Swapping is specific to moving a whole process out of RAM. First step is to trim the process down to swap size by paging, then move the whole process into the swap file. Since the trimmed size is constant, the internal structure of the swap file is very simple, effectively just an array of swap slots. Very simple to allocate and manage.

If there is insufficient swap space in SWAPFILE.SYS to swap a process, it can be swapped to a page file, but it's a bit more work. If your system is doing lots of swapping, then it may make sense to allocate specific swap file space, but in reality, most systems these days don't swap at all, and even if they did, the additional overhead of using a pagefile rather than a swapfile is probably not measurable.

Remember, paging and swapping mechanisms were designed when the costs of storage were several orders of magnitude higher than they are today - the 256K (yes, *K*) of RAM in the first VAX 780's probably costs more than $10K (yes, *K* again), and the 50MB disk drives were the size of a washing machine, and also had prices in the $10K's. Back then it was vital to make sure the effort in moving data around was minimised and every last byte of memory and disk was utilised to the maximum. The basic mechanisms haven't changed much since then.

So, 60MB is most likely plenty of space for SWAPFILE.SYS. Chances are it will never be touched. On the other hand the cost of that 60MB is a fraction of a cent, even for the most expensive, multiply shadowed and controller mirrored storage, so please don't lose any sleep worrying about trivialities!

[Dmitry V. Sinelnikov]

Thanks to everyone spending your time here and explaining these things. I really appreciate it!

Frankly speaking this thread was started based on another issue that occurs frequently on a system. Server seems to work stable unless I execute too many services or any single process starts consuming physical memory fast (for example a telnet session which executes complicated database script). As a result I observe an exception
** Bugcheck code = 000001CC: INVEXCEPTN, Exception while above ASTDEL
with IA64 going to 'warm' reboot.

While telnet session is running I run command
SHOW MEM /PHYSICAL
periodically and can observe the following:

$ show mem /phy
System Memory Resources on 21-OCT-2009 18:05:02.87
Physical Memory Usage (pages): Total Free In Use Modified
Main Memory (3.99GB) 524240 70807 412964 40469

$ show mem /phy
System Memory Resources on 21-OCT-2009 18:11:58.56
Physical Memory Usage (pages): Total Free In Use Modified
Main Memory (3.99GB) 524240 23461 455969 44810

So you can see that 'Free' degrades significantly during 5-minute interval. That's just an example when it goes down slowly. If I execute more than 55 services (each has a 50Mb size) - system will work for 2 minutes only - then exception will occur and IA64 will go down.

Firstly I suspected pagefile / swapfile of a low space but now both of them seem to have enough now (4Gb for paging, 500Mb for swapping) and the system still experience this issue.

Attached is the output of SYSGEN with current system params, together with CLUE CRASH. 'Guilty' process always differs from the previous crash (it can be even Oracle's RDM_RB72_31). I assume that there are some troubles with either paging or system quotes but can't detect the one that needs to be corrected...

Best regards,
Dmitry Sinelnikov

[Richard W Hunt]

While you have gotten a LOT of good advice above, I thought I might add a way to conserve memory, if you haven't already done this.

If you have a database program that everyone uses or an application that is commonly used, consider whether you can install it using the INSTALL verb, in which case you might want to have it installed /OPEN /SHARED /HEADER_RESIDENT. This would only benefit you if you have a lot of users touching the same image a lot of times and it isn't already installed this way. Then you won't swap anything but the user's "impure" data sets. At our site, this sort of sharing saves me 20 resident copies of our most popular middleware programs vs. 1 copy shared 21 ways.

I offer this thought only so you can reduce your needs rather than trying to boost resources to a difficult-to-predict level. If you've already done this, sorry to bother you.

[Hoff]

Dmitry, you have just disclosed very centrally relevant and very critical details here.

All advice provided in this thread will now need to be reconfirmed as being relevant to this case.

First, that crash.

You have a central flaw here with V8.3-1H1 itself or with whatever is running into a limit. Some details on what part of inner-mode code is crashing here ("INVEXCEPTN, Exception while above ASTDEL") Somebody will want to examine the dump, as a start.

Second, there's the physical memory usage. I'd likely increase the amount of memory here regardless. Whether that memory usage can be attributed to application usage, I/O caching or some sort of a problem, is not clear. Improving paging is not a reliable resolution for INVEXCEPTN errors, and memory paging is far slower than having sufficient memory; it's a system capacity ad performance trade-off for when the server is running past available physical memory.

A typical modern laptop now often has as much or more physical memory as this Integrity server. Even virtual systems need physical memory, both for applications and for caching.

[Volker Halle]

Dmitry,

you quiter certainly are hitting an OpenVMS bug with this crash.

Have a look at the crash history of this system with:

$ TYPE CLUE$HISTORY

More than one crash with the same bugcheck, module and offset ?

Could you attach the full CLUE file of the last crash - to be found in CLUE$COLLECT:CLUE$node_ddmmyyhhmm.LIS ?

Have you installed all current V8.3-1H1 patches ?

Volker.

[Dmitry V. Sinelnikov]

Attached. Captured the whole SYSERR folder + some more info.

[Volker Halle]

Dmitry,

lots of interesting things can be learned from the CLUE file of those 21 crashes !

- most of the crashes happened with an uptime of less than 23 hours, some of them even with an uptime of less than 20 minutes. What's this system being used for ? The crashes seem to have been started just after 14-OCT-2009. What has been changed ?

- some crashes show 'pgflfrag' or even 'pgflcrit' in EXE$GL_FLAGS. These are indications, that the page file is much too small or the system load is unreasonable for this configuration.

- there are 2 types of crashes:

1) INVEXCEPTN at SMP$RESTOREL_C+002D1: ld4 r24 = [r22] with an ACCVIO at VA=0000001C due to R22=1C. R22 had been calculated by adding 1C to R24, but the problem is: R24=0 ! This routine is handling (i.e. conditionally releasing) spinlocks. The spinlock structure address is in R8 and R30. If you still have access to such a crashdump, try:

SDA> READ SYSDEF
SDA> FORM @R8 ; format spinlock
or
SDA> SHOW SPIN/ADDR=@R8

2) INVEXCEPTN with R8=43C = %SYSTEM-F-OPCDEC and no access to the process kernel stack in P1 space. The dumpfile may be way too small, AUTOGEN has suggested to create a 4.4 million block dump file, the current one is only 1.1 million blocks (see AGEN$PARAMS.REPORT)

Following the AUTOGEN suggestions only makes snse, if the system has been up and running long enough to collect meaningful feedback data for AUTOGEN. With these frequent crashes, this may not have been the case !

Volker.

[Hoff]

Please patch (ECO) this OpenVMS I64 V8.3-1H1 box to current, if you're not already there.

If you're not already current on your patches, then if (when) you engage HP to have a look at these system crashes, then HP is going to ask you to apply the current patches.

[Volker Halle]

Dmitry,

for the INVEXCEPTN crashes at SMP$RESTOREL, it would be useful to also get the following information:

SDA> SHOW CALL/SUMM

as the CLUE CALL and CLUE STACK commands have not been implemented for OpenVMS I64 and this data therefore is not in the CLUE file.

The spinlock handling code itself (execlet SYSTEM_SYNCHRONIZATION_MIN) has not been patched for V8.3-1H1, but the problem may be in the caller's code.

If you have an OpenVMS support contract with HP, I would strongly suggest to escalate these problems (after patching the system to the current patch level - as Hoff has suggested). Consider to collect at least one crashdump of each type for escalation.

Volker.

[Dmitry V. Sinelnikov]

For those how are interested in or might be curious: the issue has a continuation and a resolution.

To summarize, the server
- is running under OpenVMS 8.3-1H1, carries 4Gb RAM on board and has rx4640 architecture
- was not working stably under the high load of a special application (single master application that was handling nearly 50 similar images; each image has ~50Mb size and can be launched several times)
- server's crash occurred only if this application was working. I wrote a simple application that allocates memory on demand and doesn't release it. Server survived even when this program consumed 4Gbs of physical memory and nearly all free space (7Gb + 8Gb) of two page files. Once this Master App is launched (even with 1 image loaded) - server crashed. This is where I started to suspect this Master App of being guilty in crashes.
- one strange thing about all this stuff was that crash occurred only at the very edge when physical memory is about to be totally consumed and page file needs to be involved... Also I never observed that page file is used when Master App is running. And it was always used when I ran my 'memory leak' program...

Not so long ago we got some new RAM and decided to replace the old one to see if this can be some kind of hardware issue with the memory. All eight 512Mb banks were replaced. I started the 'memory leak' program and launched Master App. Another crash occurred in exactly the same way as before (once 4Gbytes of new physical memory were over).
Then we decided to mix old memory with the new one - for the server to run with 8Gb on board. This was the point where issue disappeared. Both page files are now fully involved. Server is operating stably even if almost all physical memory (8Gbs) is consumed right before I start the Master App. I was not able to initiate a crash even when system was running low on both Physical Mem and two 7&8 Gb page files...

I have an assumption that this system was installed on a 8Gb server and then it was downgraded to 4Gb but some system settings were not tuned to this value. But probably I'm wrong. Anyway, the system is up & running and issue is eliminated. Thanks to everyone for assistance, I appreciate this.

[John Gillings]

Dmitry,

If your system crashed, there should be a dump file. Show that to HP or, even better, someone like Volker, and they will probably be able to tell you exactly why the system crashed.

Crashes aren't solved using voodoo mechanisms like shuffling RAM between slots, unless the dump tells you that is the cause.

[Dmitry V. Sinelnikov]

John,

As you can see from the thread above I was providing all requested info to you and Volker and other guys.

In my previous post I just wanted to describe how the issue appeared and behaved and how it was eliminated in the end - probably this will help someone else...

Some questions are still unclear (like 'why pagefiles were not involved with 4Gbs onboard and why did they start working with 8Gbs on board') but I'm not sure if crash dumps or crashes themselves have something to do with it... But if you wish - I can still capture any info.

Best regards,
Dmitry Sinelnikov

[abrsvc]

This may not be particularly relevant, but I find it interesting that the problems occur with physical memory capped at the 32bit addressing boundary...