Recommended pagefile/swapfile size for IA64 with v8.3

[Dmitry V. Sinelnikov]

Thanks to everyone for the feedback.

TESTFILES FEEDBACK showed that pagefile must be enlarged to 4Gb (was corrected). However swap file is treated as too large and needs to be reduced:

SWAPFILE1_SIZE information (for SWAPFILE1_NAME):
Feedback information.
Old value was 117200, New value is 85500
Maximum observed usage: 0
SWAPFILE1_SIZE will be modified to hold 85500 blocks

Am I right in my understanding that ~60Mb swap file is large enough?

The reason I am asking is the following: this system is intended to be available 24/7 with ~120 services running, each service' image size is 50MB. There won't be 100% constant service usage thus some of the processes will be swapped ('swapped out'). Does that mean swapfile will have store these 'swapped' processes?

Best regards,
Dmitry Sinelnikov

[Volker Halle]

Dmitry,

AUTOGEN judged the swapfile size by current usage (from the saved feedback data) and the system never had to use the swapfile up to now, that's why AUTOGEN tends to decrease it's size.

As John already wrote, the pagefile can also be used for swapping, should that need arise. To estimate the swap file size in advance may be impossible, you can only go by experience of the behaviour of that workload. Process working sets will typically be trimmed, before being swapped out.

Do not bother to try to make your SWAPFILE.SYS file smaller. If there is enough space on your swapfile disk, just increase the size to make you feel better.

Volker.

[John Gillings]

Dmitry,

> Am I right in my understanding that ~60Mb
> swap file is large enough?

An arm waving, simplified explanation of the difference between SWAPFILE.SYS and PAGEFILE.SYS...

Paging is a very general mechanism for moving virtual memory out of RAM and onto disk. It must therefore deal with the allocation and management of varying sized chunks of memory. The internal structure of PAGEFILE.SYS is rather complex.

Swapping is specific to moving a whole process out of RAM. First step is to trim the process down to swap size by paging, then move the whole process into the swap file. Since the trimmed size is constant, the internal structure of the swap file is very simple, effectively just an array of swap slots. Very simple to allocate and manage.

If there is insufficient swap space in SWAPFILE.SYS to swap a process, it can be swapped to a page file, but it's a bit more work. If your system is doing lots of swapping, then it may make sense to allocate specific swap file space, but in reality, most systems these days don't swap at all, and even if they did, the additional overhead of using a pagefile rather than a swapfile is probably not measurable.

Remember, paging and swapping mechanisms were designed when the costs of storage were several orders of magnitude higher than they are today - the 256K (yes, *K*) of RAM in the first VAX 780's probably costs more than $10K (yes, *K* again), and the 50MB disk drives were the size of a washing machine, and also had prices in the $10K's. Back then it was vital to make sure the effort in moving data around was minimised and every last byte of memory and disk was utilised to the maximum. The basic mechanisms haven't changed much since then.

So, 60MB is most likely plenty of space for SWAPFILE.SYS. Chances are it will never be touched. On the other hand the cost of that 60MB is a fraction of a cent, even for the most expensive, multiply shadowed and controller mirrored storage, so please don't lose any sleep worrying about trivialities!

[Dmitry V. Sinelnikov]

Thanks to everyone spending your time here and explaining these things. I really appreciate it!

Frankly speaking this thread was started based on another issue that occurs frequently on a system. Server seems to work stable unless I execute too many services or any single process starts consuming physical memory fast (for example a telnet session which executes complicated database script). As a result I observe an exception
** Bugcheck code = 000001CC: INVEXCEPTN, Exception while above ASTDEL
with IA64 going to 'warm' reboot.

While telnet session is running I run command
SHOW MEM /PHYSICAL
periodically and can observe the following:

$ show mem /phy
System Memory Resources on 21-OCT-2009 18:05:02.87
Physical Memory Usage (pages): Total Free In Use Modified
Main Memory (3.99GB) 524240 70807 412964 40469

$ show mem /phy
System Memory Resources on 21-OCT-2009 18:11:58.56
Physical Memory Usage (pages): Total Free In Use Modified
Main Memory (3.99GB) 524240 23461 455969 44810

So you can see that 'Free' degrades significantly during 5-minute interval. That's just an example when it goes down slowly. If I execute more than 55 services (each has a 50Mb size) - system will work for 2 minutes only - then exception will occur and IA64 will go down.

Firstly I suspected pagefile / swapfile of a low space but now both of them seem to have enough now (4Gb for paging, 500Mb for swapping) and the system still experience this issue.

Attached is the output of SYSGEN with current system params, together with CLUE CRASH. 'Guilty' process always differs from the previous crash (it can be even Oracle's RDM_RB72_31). I assume that there are some troubles with either paging or system quotes but can't detect the one that needs to be corrected...

Best regards,
Dmitry Sinelnikov

[Richard W Hunt]

While you have gotten a LOT of good advice above, I thought I might add a way to conserve memory, if you haven't already done this.

If you have a database program that everyone uses or an application that is commonly used, consider whether you can install it using the INSTALL verb, in which case you might want to have it installed /OPEN /SHARED /HEADER_RESIDENT. This would only benefit you if you have a lot of users touching the same image a lot of times and it isn't already installed this way. Then you won't swap anything but the user's "impure" data sets. At our site, this sort of sharing saves me 20 resident copies of our most popular middleware programs vs. 1 copy shared 21 ways.

I offer this thought only so you can reduce your needs rather than trying to boost resources to a difficult-to-predict level. If you've already done this, sorry to bother you.

[Hoff]

Dmitry, you have just disclosed very centrally relevant and very critical details here.

All advice provided in this thread will now need to be reconfirmed as being relevant to this case.

First, that crash.

You have a central flaw here with V8.3-1H1 itself or with whatever is running into a limit. Some details on what part of inner-mode code is crashing here ("INVEXCEPTN, Exception while above ASTDEL") Somebody will want to examine the dump, as a start.

Second, there's the physical memory usage. I'd likely increase the amount of memory here regardless. Whether that memory usage can be attributed to application usage, I/O caching or some sort of a problem, is not clear. Improving paging is not a reliable resolution for INVEXCEPTN errors, and memory paging is far slower than having sufficient memory; it's a system capacity ad performance trade-off for when the server is running past available physical memory.

A typical modern laptop now often has as much or more physical memory as this Integrity server. Even virtual systems need physical memory, both for applications and for caching.

[Volker Halle]

Dmitry,

you quiter certainly are hitting an OpenVMS bug with this crash.

Have a look at the crash history of this system with:

$ TYPE CLUE$HISTORY

More than one crash with the same bugcheck, module and offset ?

Could you attach the full CLUE file of the last crash - to be found in CLUE$COLLECT:CLUE$node_ddmmyyhhmm.LIS ?

Have you installed all current V8.3-1H1 patches ?

Volker.

[Dmitry V. Sinelnikov]

Attached. Captured the whole SYSERR folder + some more info.

[Volker Halle]

Dmitry,

lots of interesting things can be learned from the CLUE file of those 21 crashes !

- most of the crashes happened with an uptime of less than 23 hours, some of them even with an uptime of less than 20 minutes. What's this system being used for ? The crashes seem to have been started just after 14-OCT-2009. What has been changed ?

- some crashes show 'pgflfrag' or even 'pgflcrit' in EXE$GL_FLAGS. These are indications, that the page file is much too small or the system load is unreasonable for this configuration.

- there are 2 types of crashes:

1) INVEXCEPTN at SMP$RESTOREL_C+002D1: ld4 r24 = [r22] with an ACCVIO at VA=0000001C due to R22=1C. R22 had been calculated by adding 1C to R24, but the problem is: R24=0 ! This routine is handling (i.e. conditionally releasing) spinlocks. The spinlock structure address is in R8 and R30. If you still have access to such a crashdump, try:

SDA> READ SYSDEF
SDA> FORM @R8 ; format spinlock
or
SDA> SHOW SPIN/ADDR=@R8

2) INVEXCEPTN with R8=43C = %SYSTEM-F-OPCDEC and no access to the process kernel stack in P1 space. The dumpfile may be way too small, AUTOGEN has suggested to create a 4.4 million block dump file, the current one is only 1.1 million blocks (see AGEN$PARAMS.REPORT)

Following the AUTOGEN suggestions only makes snse, if the system has been up and running long enough to collect meaningful feedback data for AUTOGEN. With these frequent crashes, this may not have been the case !

Volker.

[Hoff]

Please patch (ECO) this OpenVMS I64 V8.3-1H1 box to current, if you're not already there.

If you're not already current on your patches, then if (when) you engage HP to have a look at these system crashes, then HP is going to ask you to apply the current patches.