Strange performance issue after upgrading from 11.11 to 11.23

[Geoff Wild]

Kept the same kernel parameters as before (except some are gon and some new ones at default).

I now see memory page in rate activity.

Last Friday, the average:

GBL_MEM_PAGEIN_RATE = 71
GBL_MEM_PAGEOUT_RATE = 1

On Sep 26, it was 0 for both...

I have a call into HP as well - thought I'd post as this could be valuable to everyone.

On out QA system (only 4 cpu's) it had 1.1 GB of ram available - yet still had page in's.

On prod, with 6 cpu's - page in's were less...

Rgds...Geoff

[Sandman!]

Well pageouts is never a good thing and a high pageins indicates that the I/O got re-distributed (as opposed to on or before Sep 26) causing page faults and page loadins from disk. Look at the I/O activity...has it gone up significantly and was the data on the disks reorgd?

[Geoff Wild]

Server has EMC PowerPath - no I/O changes made.

No significant I/O differences are seen...

Rgds...Geoff

[A. Clay Stephenson]

You may be comparing Apples to Oranges in that unless the activity was extremely similar during the two sampling intervals, the comparisons aren't all that meaningful. Any new command that isn't already running is going to be paged in --- so the pagein's are normal and expected. A trivial example is vi. The first instance that is launched must be completely paged in and you would see a number of page faults. Subsequent instances of vi would trigger no such activity --- as long as at least one instance of vi was still active. Pageout's can also occur on a system with plenty of memory if memory mapped files are in use.

[Geoff Wild]

Well - the main issue is - I've never had page in's at all before - Oracle Grid was reporting up to 19,000 page in's per second!

Our Prod seems okay - but QA was suffering - probably cause it had 2 less cpu's - and couldn't handle that many page in's...


Rgds...Geoff

[Bill Hassell]

Very important definition:

The kernel starts all processes by paging in the parts of the program needed to start running. The kernel does not differentiate between paging in from the filesystem where the executable is stored or paging in pages that were swapped out previously. The metric PAGE_IN counts every page -- which limits its usefulness in defining memory requirements. It is absolutely normal to have PAGE_IN rates in the thousands if you are starting lots of small processes. I run a big rp8420 that runs dozens of scripts per second and the PAGE_IN rate runs 100's to many thousands. PAGE_IN rate directly follows the number of new processes that are starting. PAGE_IN will also count text pages coming from the executable's original location when the text area must be restored. But alas, there is no way to tell whether PAGE-INs are coming from the swap areas.


PAGE_OUT *only* refers to paging a deactivated process to the swap area as well as some activity with memory mapped files. When PAGE_OUT is single digits, RAM is fine. Double-digits (10-99) for a long time mean RAM is a bit short. Triple digits (and higher) means severe RAM limitations. Lots of memory mapped file activity can push PAGE_OUT into double digits and this is controlled by the applications.

[Steven E. Protter]

Shalom,

The OS itself does use more memory. Though its supposed to improve performance not decrease it.

Having benefited from the wisdom of all above me:

1) Recommend a longer sample period for new data. No way to deal with the past but give things a chance to balance out.
2) Oracle may require some kernel parameters different on this OS and its possible in an OS upgrade the oracle checks were bypassed.
3) Just for grins: http://www.hpux.ws/?p=6 performance data collection.

I assume when you use the term upgrade you meant cold install followed by application install. I remember having a drink with you in Houston and discussing the ills if upgrade-ux.

SEP

[Geoff Wild]

Been a long time between posts, but here's an update - one of our instances on an APP server running 11.23 parisc died - from SAP:

Extended analysis has detected that the instance svr113_PRD_00
went down due to signal 9 again.

dev_disp:
..
Mon Nov 26 13:26:37 2007
*** ERROR => W1 (pid 14653) died [dpxxdisp.c 14319]

dev_w1:
..
W PpioRecoverLocks() done
W PprcRecoverLocks, table: c00000004d7c1848
W PprcRecoverLocks: directory lock set to: 0000000000000000
W PprcRecoverLocks() done
W === ipl_Init() returns 0, ITSPE_OK: OK
W =================================================


trc file: "dev_w1", trc level: 1, release: "700"

*
* ACTIVE TRACE LEVEL 1
* ACTIVE TRACE COMPONENTS all, MJ
*
M sysno 00
M sid PRD
M systemid 273 (HP/9000 (PA-RISC) with HP-UX)
M relno 7000
M patchlevel 0
M patchno 100
M intno 20050900
M make: single threaded, ASCII, 64 bit, optimized
M pid 14938
M
M
M Mon Nov 26 13:26:37 2007
M kernel runs with dp version 224(ext=109) (@(#) DPLIB-INT-VERSION-224)M length of sys_adm_ext is 360 bytes
M ***LOG Q01=> ThInit, WPStart (Workproc 1 2 14938) [thxxhead.c 1267]M ThInit: running on host svr113
M calling db_connect ...

Kindly see SAP Note 426188.


That note mentions NFS as the culprit...

Well - we use autofs and automounts...same as what we have been doing for years...



Rgds...Geoff