Stat's from VMSTAT and GLANCE don't agree.

[Dan_173]

The PO (pages out per second averaged over 5 secs) value from VMSTAT and GLANCE's High Rate value of Pages Out (highest rate in pages per sec) don't appear to agree. I'm seeing upto 850 PO's from VMSTAT, but in a glance session that's been running for some days, the high-water mark of Page Outs is only 63.

Anyone know why this might be?

Thanks in advance.
Dan

[Anu Mathew]

Hi Dan,

Please elaborate on your OS version and the patch level. I had had problems with version of glance on HP-UX 11.00, stating wrong reports on Shared Memory params. We updated the Glance Version C.03.58.00 as suggested by HP Support.

A difference of 850 and 63 is way too high, 850 is a huge number, and do you think that it could be true as far as the system performance is concerned?

At my site, POs are 0 almost all the time.

Regards

Anu Mathew

[Steven E. Protter]

In your situation I'd make sure everything was current, especially the glance version number.

Then I'd measure statistics over a longer period of time to reduce the impact of spikes.

I've included a set of sar scripts that run background. They collect data into a file for a timer period you determine by modifying the script.

SEP

[Peg McMahon]

A factor in the discrepency might be the refresh rate. vmstat is looking at five second data and GLANCE is probably looking at 15 second data. (Check your config.)

The bigger question, though, is what is going on with your memory. The number of page outs is alarming from any point of view. I was taught that a good number for page outs is zero.

Is this a new situation? If so, what has changed?

In any event, the text book indications of memory bottlenecks are:
*memory saturation - low free memory
*virtual memory queues - metric available in gpm or glance
*resource starvation - CPU is being used by memory management process - look for vhand, for example
*user complaints about response times

Another forum recently referenced this document for memory tuning:
http://docs.hp.com/hpux/onlinedocs/os/11.0/tuningwp.html#lotsdes

Two more points: I know it is hard for an SA to use a GUI interface but we have found that starting GlancePlus with the gpm& command uses far fewer system resources than starting with the glance command - and produces better metrics.

You do have to export your display to an X windows client using this command:

export DISPLAY=your Xwindows ip address:0.0

I know that with gpm, you cannot export data directly to a file. However, you can frequently get more granular info doing an extract from MeasureWare data. I outline the procedure for that below.

In addition, if you use the gpm& interface and go to Configure, you can choose many additional metrics. GBL_MEM_WAIT_PCT and GBL_MEM_WAIT_TIME are two that I would examine in your case.

Also on the Process Report, you can choose memory metrics around each process: PROC_MEM_MAJOR_FAULT_CUM, PROC_MEM_RES_HIGH, PROC_MEM_VFAULT_COUNT and PROC_MEM_WAIT_PCT are metrics that might help.

In addition, look carefully at the RSS memory size of your processes. If it increases over time for a given process, you have a memory leak. You need to ask the application owners to tune their code.

Finally, you can look at your process behavior in detail by doing an extract of MeasureWare data.

Using the following command will get you to the Unix equivalent of a Microsoft Wizard. (Don't laugh at it. It doing its best.)

Just answer the prompts as they come up.
I generally use the "export" option so I can open the resulting file in Excel.

Use the default metric source: /var/opt/perf/datafiles/global. Just press Enter when the wizard asks which data sources you wish to use.

It will ask you if you wish to create a metric template. Say no and use the one attached.

I'm attaching a template that I use when doing an export. I use it to get infomation about process behavior, especially focused on wait states. FTP it over to the server on which you are doing the extract.

When the extract wizard asks what template you want to use, give the fully qualified path to the one in the attachment.

Be sure to minimize the time slice that you use to collect data. The template obtains five minute data so a lot of information is produced. Using data from one or two hours is plenty - if you intelligently pick the one or two hours.

The command to start the extract wizard:

/opt/perf/bin/extract

To exit the extract program, you have to enter quit twice. (Don't ask me.)

When you get done, ftp the file to your desk computer and open it in Excel. The file is comma separated. (If you are an advanced Excel user, the data makes excellent pivot tables. Even using ordinary Excel sorts, it produces a great deal of useful information.)

If you need more help on this, I will be glad to provide it.

Peg McMahon
Capacity and Performance Management

[James R. Ferguson]

Hi Dan:

Bill Hassell commented on this here:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x345b0ea029a2d711abdc0090277a778c,00.html

Regards!

...JRF...