heavy paging

[Dean McGorrill]

We have a remote production machine with large rdb job that
runs weekends that ran out of pgflquota. I upped the
pgflquota and wired in a secondary pagefile and we now
complete. The problem is the job runs 33 hours and we
need to shorten in. We have a test system here that I have
pagefiles on stripesets that took 9+ hours off the job.

o If one adds more and more pagefiles, does the paging IO for a
*single* process get distributed across them?

o would putting up decram (license cost?) and put a pagefile
on the ram disk work? (we have 2gig of memory.)

o Any other ideas on how to reduce the run time?

tx for any help Dean - data below

account quotas..

Maxjobs: 0 Fillm: 350 Bytlm: 300000
Maxacctjobs: 0 Shrfillm: 0 Pbytlm: 0
Maxdetach: 0 BIOlm: 1000 JTquota: 40960
Prclm: 40 DIOlm: 1000 WSdef: 16384
Prio: 4 ASTlm: 1000 WSquo: 32767
Queprio: 4 TQElm: 1000 WSextent: 64000
CPU: (none) Enqlm: 18000 Pgflquo: 990000

system pagefiles...

DISK$ALPHA_V72_1:[SYS10.SYSEXE]PAGEFILE.SYS
1056768 543712 1056768
DISK$DEAN_TESTPG:[ENGDS2_PAGE]PAGEFILE2.SYS;1
1056640 986144 1056640
some sysgen params...

WSMAX 786432 4096 1024 8388608 Pagelets
internal value 49152 256 64 524288 Pages
NPAGEDYN 9494528 1048576 163840 -1 Bytes
PAGEDYN 4980736 524288 65536 -1 Bytes
QUANTUM 20 20 2 32767 10Ms D
FILE_CACHE 0 0 0 100 Percent
S2_SIZE 0 0 0 -1 MBytes
PFRATL 0 0 0 -1 Flts/10Sec D
PFRATH 8 8 0 -1 Flts/10Sec D
WSINC 2400 2400 0 -1 Pagelets D
internal value 150 150 0 -1 Pages D
WSDEC 4000 4000 0 -1 Pagelets D
internal value 250 250 0 -1 Pages D
FREELIM 473 32 16 -1 Pages
FREEGOAL 1812 200 16 -1 Pages D

from accounting of job quotas...


Page faults: 36664 Direct IO: 16237388
Page fault reads: 561 Buffered IO: 2053
Peak working set: 535072 Volumes mounted: 0
Peak page file: 728176 Images executed: 31

[Andy Bustamante]

Raise the working sets, the working set default and quota. Use sys$examples:working_set.com or Availability Manager to monitor these.

The other question is what versions of VMS and hardware are you using? Memory and disk configuration? Other processing? I'd say account quotas are set very low for an Alphaserver, you probably have pql_ overriding these. What else is happening on the system? Compare the two systems.

Performance review really needs many details.

Andy

[Jim_McKinney]

You report heavy paging but show only 36k page faults from accounting - lots of DIOs though. I'd look at trying to speed up the IOs either with caching (don't know what's available with RDB) or by locating the data on faster disks if possible. You may be able to eliminate most of the page faults by increasing the process WSEXTENT to some value close to the 535072 page peak working set (presuming that you've got a big chunk of that 2GB available).

[Hoff]

Back up a kilometer or three here, and systematically evaluate the system and system activities based on the guidelines in the performance management manual; with the tuning information in the documentation set.

Look at the processor performance (we don't know what box and what version), look at the available physical memory usage, look at I/O caching and I/O rates, at details like when the last AUTOGEN was run such, and then and only then start to look at specific behaviors such as page faulting.

Get a baseline configuration and baseline performance. You'll need this information, and you'll need it system-wide. MONITOR recording or the T4 tools can be useful in gaining and maintaining this baseline.

Once you have a baseline and once AUTOGEN has been turned loose -- sans constraints left in MODPARAMS.DAT -- then start to look at parameter adjustments. Then you can use the baseline to determine if your changes have had the desired effect.

If this is the working set, then you can look at what is needed to either increase the available working set, or decrease competing requirements, or adding more memory, or replacing the box.

If this is I/O-bound, then there can be faster I/O widgets, faster I/O paths, or DECram or such. (But realize that memory limitations can hammer your I/O rates due to overly-constrained cache sizes -- this stuff is all interconnected.)

My traditional guesstimate is that hand-tuning will generally get you about a 10% improvement beyond what AUTOGEN and baseline sane process quota settings and baseline identifying and removing any massive bottlenecks will get you, and that tuning effort is often a waste of time. More than a couple of days of time tuning (in aggregate) usually means it's time to upgrade, or to replace.

Application tuning may or may not pay off. It really depends on what the application is doing, and the effort involved in addressing latent design-related bottlenecks. (And these can be subtle, too, such as contention among the various FC SAN clients or out at the EVA or MSA storage controller.)

Tuning, BTW, has little to do with lists of numbers. It's all rates and relationships and trends and balances; about looking at the whole, and working toward identifying the limits. One set of numbers tells little.

Throwing hardware at the problem is usually (always?) cheaper than extended tuning efforts, in my experience. Barring gross configuration errors and once AUTOGEN has been brought to bear with feedback over time, system tuning tends to be a delaying tactic. At best.

And as for hardware upgrades, my laptop has 2GB.

Stephen Hoffman
HoffmanLabs LLC

[Jan van den Ende]

Dean,

First of all:
Why use disk space if you have physical memory to spare?
Raise WSMAX to be equal to physical memory.
And if you did that, there is no reason for a ramdisk: you have allowed the active processes to use the full physical memory, so why employ e detour to use part of that via a "disk" driver?
If that still does not help enough, you must revert back to the old IBM cure:
"Buy more memory"

as an aside (from your profile)
>>>
I have assigned points to 0 of 20 responses to my questions.
<<<
Please review
http://forums1.itrc.hp.com/service/forums/helptips.do?#33
about the way to say thanks in these forums.

Proost.

Have one on me.

jpe

[Ian Miller.]

Yes a process can use multiple page files but why not just increase the WSDEF,WSQUOTA,WSEXTENT of that job. If you have the memory free for a DECram then use it for process working sets instead and avoid the paging overhead.

[Dean McGorrill]

Andy,

the remote and test system are configued
the same. AlphaServer DS20 500 MHz with
2gb memory. hsz70 controller. on weekend
the systems dedicated to this job. pql.s

PQL_DWSDEFAULT 5936
PQL_MWSDEFAULT 5936
PQL_DWSQUOTA 11872
PQL_MWSQUOTA 11872
PQL_DWSEXTENT 786432
PQL_MWSEXTENT 786432

the disk presented to the alpha is
mirrored COMPAQ BD009122C6 disks, 10k rpm
9.1 gig drives.

[Martin Hughes]

What is the RDB job you mention actually doing?. What's your settings for database global buffers?. Direct I/O looks more of an issue than paging.

[Martin Hughes]

Also, are these hard page faults? and is your process spending a lot of time in PFW state?.

[Dean McGorrill]

tx all,
I will start with ws, (I thought 64k
was the max) jpe, points granted tx. upgrade
is not possible, but i'm working on mgmt.

Dean

[Hein van den Heuvel]

>> o would putting up decram (license cost?) and put a pagefile
on the ram disk work? (we have 2gig of memory.)

That remark might just qualify for an WTF entry.
Yeah, WTF suffegst 'What The F*&^', but is spelled as:
http://worsethanfailure.com/Default.aspx

If you have the memory, just allow the process to use it directly! (WS quotas)

Anyway, I understand that through running out of pgflquota you learned that paging might be an issue, but it does NOT appear to be the defining number for the performance here.

The accountng data, which unfortunately did not contain the also elapsed/cpu time, strongly suggests that DIRECT IO defines the performance. 16 Million IO in 33 hours is about 136 IO/sec.... when evenly spread.
If those are truly random, with single stream (1 batch job) driver) then that's about all you will ge no matter how many disks there are behind it. This is only the case if they are truly random, new READ IOs, meaning no IO cache has the data and none of the many disks is any closer to the target then any other.
With you I expect your storage system to perform better, but this may be all there is worst case.

So now back the test system.
What do the accounting numbers look like there?

Does it have EXACTLY the same database?
If it is 'very close' does it have exactly the same indexes defined.
Do both systems have much the same buffer pool defined for RDB?

If you want to make a real impact on the performance of this job then yo probably need to look at RDB settings, and query tunings, not at OpenVMS tweaks.

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Robert Gezelter]

Dean,

I concur with Hoff. My standard recommendation to clients is that if paging is perceived to be the problem, something else is actually the problem.

Certainly, increasing the working set limits (and the corresponding page file quotas) to values more in line with physical memory is a good idea.

Running data collection using T4 is always a good idea. Detailed review of what this data shows is also a sound idea. (Disclosure: We do perform this service for clients).

- Bob Gezelter, http://www.rlgsc.com

[John Gillings]

Dean,

On a 2GB system WSEXTENT=64000 is *tiny*. That's only 31MB. Less than most people's wristwatches. Fortunately AUTOGEN has overriden WSEXTENT to something more reasonable.

Check the VIRTPEAK for the process to get an idea of how large it wants to be.

However, even with low WSEXTENTs, OpenVMS is quite good at handling large virtual address spaces efficiently. In a recent capacity test on a system with 4GB and WSMAX at 1.5GB, a process with 2.5GB of virtual address space, reading through it all several times, sustained a (soft) fault rate of >100,000 per SECOND for more than 5 minutes. Amazing! They were all modified list faults, in effect OpenVMS was using the modified page list as an extension of the working set of the process. In comparison with your job, we were experiencing double your TOTAL number of pagefaults for your entire 33 hour job every second!

Soft faults are cheap. Eliminating them is easy (just increase process working sets), but is unlikely to return a significant performance improvement.

>put a pagefile on the ram disk work?
>(we have 2gig of memory.)

Direct answer - NO! This doesn't make sense! The idea of a page file is a place to put stuff that doesn't fit in physical memory. Putting the pagefile itself in physical memory is like having too much stuff in your garage, so you build a shed INSIDE the garage to to put the excess in. Can you see why that won't help?

What MIGHT work would be to build a RAM disk and put the DATA FILES on it, to reduce the cost of all those direct I/Os.

The other consideration, you haven't said how much CPU time the job took. I can't see how those paging and I/O stats can account for 33 hours. The CPU usage will give you a lower limit to the run time. If it's high, you should be looking at the code to see if there are faster algorithms to achieve what you're trying to do.

[Dean McGorrill]

Hi John,
its been a dozen years since I worked with this stuff. tuned up our build system
and then it was coding thereafter..
the CPU was 5 hours and some change for
both the production and test system. From
accounting..

Peak working set: 535072

that looks like it blew past its wsextent
limit anyway. also from uaf, my account wsextent..

WSextent: 16384
$ sho work
Working Set (pagelets) /Limit=5936 /Quota=16384 /Extent=786432
Adjustment enabled Authorized Quota=16384 Authorized Extent=786432

it says I have what wsmax is set to. (?)

>What MIGHT work would be to build a RAM disk and put the DATA FILES on it

thats an idea! anyway i've upped the quotas
and raised wsinc for a test run.

[Colin Butcher]

Hello,

It would be well worth your while getting some data using T4 - then you can see what's really going on in terms of IO, paging, file opens, locking, lock migration between nodes, network traffic etc.

Can you provide some configuration data too - machine type, VMS version, disc subsystem etc. as well please?

It's also probably worth looking at XFC usage as well - 2Gbytes isn't that much memory, so with some RDB tuning and some VMS tuning, combined with extra memory - you may see a decent difference. It all depends on the workload and how the RDB job functions.

You may find that some small changes to the way the RDB job is written can provide some big changes. Sometimes relatively small changes to code can provide big wins. Data from something like T4 will give you a starting point for a thorough investigation.

Cheers, Colin (http://www.xdelta.co.uk).

[Volker Halle]

Dean,

please consider using T4 to collect performance data on both systems and start collecting the data NOW, before you change lots of parameters.

TLviz (the T4 data visualizer) contains marvellous features for comparing performance data in a before-after analysis.

If the accounting data shown is from the ONLY process involved in this 'large RDB job', then the direct IOs seems to be the major factor. T4 will also give you system-wide performacen data and you may be able to easily 'see' other factors influencing performance.

Volker.

[Dean McGorrill]

hi Colin, the cpu & disks are described a few posts ago.

Volker,
today is the first time I'm watching this running live, and its not what I thought it was, paging. though its using up pgflquota. using the normal tools, monitor sda etc. I've yet to see a pfw. its all direct i/o as most have said. we slowly
increase ws, even though I set wsinc to 48k (?)

they have about 20 files all on one
disk open. I've caught a few that are busy
with sda sho proc/chan. is there a tool
to show what files are getting hit the
most?

t4 sounds like a very useful tool, I found the kit but this old 7.2-1 system I don't think product can read a compressed pcsi file. is there a uncompressed one out there?
If I find out the hot files, then splitting
them out to other spindles should help.

[Jim_McKinney]

> is there a tool to show what files are
> getting hit the most?

You've got at least one - SDA.

$ analyze/system
SDA> read sysdef
SDA> show proc/chan/id=xxpidxxx
SDA> ! use the addresses in the window column
SDA> ! and note the wcb$l_reads and wcb$l_writes
SDA> format/type=wcb xxwcbxxx

[Hein van den Heuvel]

Dean,

You indicated it is an RDB application.

So, for now, forget about tuning OpenVMS! just ask RDB where it hurts!
It can tell you which files are busy,
which files are waited for most...

Poke around with RMU
Look as RMU> SHOW STAT/STALL

Check the "Rdb7 Guide to Database Performance and Tuning"

No point in speeding up an IO which should not be done in the first place!

Give some more memory to the RDB caches!?

Use SHOW MEM/CACH=(TOPQIO=20,VOLUME=...) for a simple, cheap, OS hotfile list.
... if you have XFC caching going.

Or like you did, do a MONI CLUS or MONI DISK/TOPQIO during the run and see the top busy disk(s).
Now SHOW DEV/FILE for a first impression... a file must be open to be busy!

Having said that, if like you seem to indicate all the open file for the job are on a single disk then you may want to address that first without even before learning more about the load.

Personally, I like the SAME approach. Stripe And Mirror Everything.
'Don't worry your pretty head' about which file is busy or how to exactly balance the IO. Just brute-force spread it out if you can! This is trivial (default) on the EVA specifically, but straighforward on the HSZ70 as well allthough you can not go as wide.

An other this to look at his that HSZ.
Run the display there and watch which disks are being hit. This will also nicely give a read-write ratios and HSZ cache information.

hmmm.... are the HSZ batteries on the production system working properly? If not, the HSZ will disable the write-back caching and WRITE performance will suggest tremendeously. That can easily explain several hours.

Cheers,

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Dean McGorrill]

tx Jim,
hot files identified.
tx Hein,
yes the cache batts are good and
I will chat with our dba about rdb tuning.

dean

[Volker Halle · Accepted solution]

Dean,

to more easily look at the process read/write counters for open files, use the PROCIO$SDA extension. I have provided an improved version at

http://eisner.encompasserve.org/~halle/

The process may only be increasing it's workingset slowly, because it's not generating that many pagefaults.

Did you consider, that the underlying problem of exceeding pagefile quota - after long runtime - may be a memory leak ?!

Volker.

[Dean McGorrill]

Volker,
That sda extension is great, tx. I'll
watch the production machine for a leak.
we've identified a single file that is
getting most of the io. now to look at what
to change. tx again for the procio

Dean