vps_ceiling parameter: What is the effect of change

[Tony Drake_2]

I am doing some kernel changes for oracle. In the latest metalink document, Oracle reccomends changing VPS_Ceiling to 64 from a default of 16.

I've read the documentation and searched the forums, but I can't figure out what this will actually do.

Is there a downside of changing this parameter? Is there a downside to leaving it as is?

The system is running oracle roots (8.1.7 32 bit, 8.1.7 64 bit) and has 5 instances total on it.

Thanks.

[Pete Randall]

Tony,

From SAM's Help on Configurable Parameters:

"This parameter is provided as a means to minimize lost cycle time caused by TLB (translation look-aside buffer) misses on systems using newer PA-RISC devices such as the PA-8000 that have smaller TLBs and no hardware TLB walker.

If a user application does not use the chatr command to specify a page size for program text and data segments, the kernel selects a
page size that, based on system configuration and object size, appears to be suitable. This is called transparent selection. The selected size is then compared to the default maximum page-size value defined by vps_ceiling that is configured at system-boot time. If the the value is larger than vps_ceiling, vps_ceiling is used.

The value is also compared with the default minimum page-size value defined by vps_pagesize that is configured at system-boot time. If the the value is smaller than vps_pagesize, vps_pagesize is used. "


Does that help?


Pete

[Pete Randall]

Tony,

I forgot to mention that my 11i systems have it set to 64 (by default, I assume).


Pete

[Zygmunt Krawczyk]

vps_ceiling parameter is the size of virtual page size in kB. Increasing the page size will improve Oracle performance.

Regards,
Zygmunt

[Tony Drake_2]

I guess where I am confused is what page size we are talking about here.

The oracle block size is 8K.
The JFS block size is 1K
(yes i know they should be the same, but that change is not a priority right now as it involves a lot of work for us to make new Filesystems).

So in effect, is this saying that Oracle can move 2 blocks to disk at a time with a 16K VPS_Ceiling, and now it can move 8 blocks at a time with a 64K ceiling?

Or is this saying the unix buffer cache is now going to fill up in larger chunks?

Or is this saying that the pages swapped in and out to disk are going to be larger.

THanks

[Don Morris_1]

The virtual page size referenced here is the RAM size used within the process -- it does not affect disk blocks (at least not in any way I can imagine... the process shouldn't know or care [other than in TLB miss impact] what the underlying page size is).

Short version -- HP-UX uses RAM in chunks called pages (physical or virtual). Right now, the system page size is 4k. The kernel and many applications often have virtual objects spanning many of these pages. To improve performance, HP-UX groups the underlying "base" pages into "large pages". For example -- if an application allocates 64kb, that's 16 base pages. If you read/write those pages, you would take a TLB fault on each page. If the OS groups the underlying 16 pages into a single 64kb large page -- then a single translation is sufficient for the entire range. Which means you take only 1 TLB fault. [There's obviously a lot of detail I'm leaving out here -- I did say this was the short version!]

This is the behavior Oracle wants... since it works with large data sets. It could control this itself for Text and Data (chatr +pi / chatr +pd and the vps_chatr_ceiling tunables control this), but Oracle also uses large shared virtual objects (SysV shmem or mmap with MAP_SHARED) which are not part of the process Data (hence chatr +pd don't affect these). For these objects, the kernel uses a heuristic based on the size of the allocation to generate a "best guess" large page size -- and that guess is capped by the vps_ceiling tunable. Oracle is requesting that tunable be raised to allow the heuristic to give it larger pages to improve performance.

Now -- the obvious following question is "Why isn't the tunable by default larger?" The cost is that the heuristic is exactly that -- a heuristic, not a precise match. The best example in my mind is that of a sparse data set (which happens more than some might expect). If an application makes a very large virtual allocation (just in case it might need it -- or to hold a matrix for which there are only a few pertinent values, etc.) then the heuristic favors giving that large allocation large pages to match. If the tunable is high, then the heuristic value is used. Until a page is touched, there's no cost -- but once a page is touched (and real RAM has to be allocated to back the virtual page), then the larger page size comes into play. For example -- take an application that allocates a 1Gb virtual object and then walks every 1Mb changing a single byte (yeah, this is a silly example [I hope!] but I'm hoping it is clear]. The heuristic gives the object 4Mb large pages -- but the vps_ceiling value is 16kb. The application ends up using (16 * 1024) physical RAM pages. If the vps_ceiling value were >=4Mb, then the heuristic value would be used... the application would take 25% of the TLB faults of the first case (since only 1 TLB fault per 4 strides would be needed), but would consume the entire 1Gb of RAM (barring system memory load, swap, etc.).

Out of curiosity -- did you read the vps_ceiling man page on docs.hp.com (tunables are under man(5)). The man page shipped on post-11.11 releases, but there are web versions for 11.11. If you did and it wasn't clear, I'd appreciate any feedback on improving it.

[Tony Drake_2]

OK, its getting clearer now..

It would seem that this setting affects the only the RAM -> CPU part of the chain and allows the CPU to reference larger portions of RAM at once so to speak...is this accurate?


1) Does this setting then affect only each process individually? (ie if Oracle uses a 64KB size, then say SAS can still use its own setting lower than 64)

2) As far as TLB faults, where does that actually show up as a metric? Or does it? (ie, can I see that this is a problem by looking in glance/perfview/sar/etc)? Or is this so low level that it would not actually be of measurable impact.

3) As far as oracle goes, are you saying that this allows the SGA and other shared areas to be read in in larger chunks? IN other words, would this say mean that something like a full table scan would work better? Using my previous numbers, oracle could now parse out the data read into SGA 8 blocks at a time (64K), instead of 2 blocks at a time (16K)?

4) Are you saying that the downside of this setting is that memory usage will possibly increase as larger chunks of Ram are used?

I am a computer engineer (but not totally familiar with the PA RISC architecture on a low level), so I can handle a deep-tech answer if its required here.

Thanks again!

[Don Morris_1 · Accepted solution]

>OK, its getting clearer now..

Glad to hear it... hope this keeps things moving in that direction.

>
>It would seem that this setting affects the >only the RAM -> CPU part of the chain and >allows the CPU to reference larger portions >of RAM at once so to speak...is this >accurate?

I'd say "more efficiently" rather than "at once" -- you still have to load the cache lines into the data cache... this saves translation faults, is all.

>1) Does this setting then affect only each >process individually? (ie if Oracle uses a >64KB size, then say SAS can still use its >own setting lower than 64)

This tunable sets the upper bound for the entire system. Individual processes can communicate what they want for Text/Data using chatr, and there's a different tunable for how high they're allowed to go on that (vps_chatr_ceiling).

>2) As far as TLB faults, where does that >actually show up as a metric? Or does it? >(ie, can I see that this is a problem by >looking in glance/perfview/sar/etc)? Or is >this so low level that it would not actually >be of measurable impact.

Caliper is the tool I know of to track this.
http://www.hp.com/go/hpcaliper

>3) As far as oracle goes, are you saying >that this allows the SGA and other shared >areas to be read in in larger chunks? IN >other words, would this say mean that >something like a full table scan would work >better? Using my previous numbers, oracle >could now parse out the data read into SGA 8 >blocks at a time (64K), instead of 2 blocks >at a time (16K)?

No... Oracle doesn't know the underlying page size the system is using (this is crucial since large pages can be resized on the fly)... so Oracle will only know that it has X amount of memory allocated. What this does is shorten the time it will take Oracle to read 64Kb (since only one TLB miss will be taken instead of 4).

>
>4) Are you saying that the downside of this >setting is that memory usage will possibly >increase as larger chunks of Ram are used?

Yes. If the kernel decides a given virtual allocation merits a larger page size, then any access to that page will result in the entire page being brought in. So any parts of that large page which are unused are really wasted physical memory.

Hope this helped,
Don

>
>I am a computer engineer (but not totally >familiar with the PA RISC architecture on a >low level), so I can handle a deep-tech >answer if its required here.
>
>Thanks again!

[Tony Drake_2]

Thanks once again, the fog is continuing to clear :>).

I just did a bunch of work on the box over the weekend, and did a vmstat -s and got the following output (see below). The box has been up less than 24 hours.

Do the Page Select Size Failure numbers relate to VPS ceiling? (I looked at another forum post on the topic and it didn't clear it up much). The reason I ask is that that would provide a concrete justification to the change, and a measure of its success, both of which are important in my environment. Memory usage is fairly low at this point (less than 60 percent) so page allocation failures seem odd...


vmstat -s
0 swap ins
0 swap outs
0 pages swapped in
0 pages swapped out
129537690 total address trans. faults taken
38907443 page ins
86552 page outs
317590 pages paged in
86608 pages paged out
46024997 reclaims from free list
38834202 total page reclaims
7367 intransit blocking page faults
90335710 zero fill pages created
58098569 zero fill page faults
22916474 executable fill pages created
317590 executable fill page faults
0 swap text pages found in free list
9301237 inode text pages found in free list
0 revolutions of the clock hand
314316 pages scanned for page out
13243683 pages freed by the clock daemon
225727317 cpu context switches
343115354 device interrupts
186920555 traps
14061813464 system calls
7952875 Page Select Size Successes for Page size 4K
7535441 Page Select Size Successes for Page size 16K
46217 Page Select Size Successes for Page size 64K
30505 Page Select Size Successes for Page size 256K
17278 Page Select Size Successes for Page size 1M
281 Page Select Size Successes for Page size 4M
10 Page Select Size Successes for Page size 16M
9 Page Select Size Successes for Page size 64M
7952875 Page Select Size Failures for Page size 16K
206506 Page Select Size Failures for Page size 64K
252723 Page Select Size Failures for Page size 256K
283228 Page Select Size Failures for Page size 1M
300506 Page Select Size Failures for Page size 4M
292373 Page Select Size Failures for Page size 16M
292359 Page Select Size Failures for Page size 64M
103132150 Page Allocate Successes for Page size 4K
23017683 Page Allocate Successes for Page size 16K
46298 Page Allocate Successes for Page size 64K
30548 Page Allocate Successes for Page size 256K
7278 Page Allocate Successes for Page size 1M
275 Page Allocate Successes for Page size 4M
8 Page Allocate Successes for Page size 16M
34 Page Allocate Successes for Page size 64M
17104 Page Demotions for Page size 16K
2 Page Demotions for Page size 256K
139 Page Demotions for Page size 1M
15 Page Demotions for Page size 64M