Operating System - HP-UX
1825789 Members
2122 Online
109687 Solutions
New Discussion

Re: Paging issue/Performance Question.

 
SOLVED
Go to solution
joe_91
Super Advisor

Paging issue/Performance Question.

Team:

We have this issue on an Nclass server running 11.0 64bit with 20Gig memory. We run a tool called SRM (System Resource Manager), which would look into Processor utilization, run queue and Page>Limit/Sec (This figure indicates the amount of time that the number of paged out virtual memory pages was greater than the Limit per second threshold). On this box it shows that the page>Limit/Sec has reached high levels (yellow) while the same data also shows that it has 13.4 Gig of free memory at all times. With this kind of free memory, I would assume that the machine never has to swap memory pages. Obviously, the tool uses commands such as sar, vmstat, swapinfo and other popular Unix commands. This is a critical performance indicator because a great decline in performance occurs once the system passes this threshold. Please advice where to start and why this may happen and what to look into.

Thanks
Joe.
24 REPLIES 24
harry d brown jr
Honored Contributor

Re: Paging issue/Performance Question.

If you aren't using glance/measureware + perfview, then you aren't looking at anything worth noting. sar, vmstat, iostat, are good for quick looks at servers, but if you want more accurate results get glance/measureware + perfview.

live free or die
harry
Live Free or Die
Steve Steel
Honored Contributor

Re: Paging issue/Performance Question.

Hi


What is running on the box.

1)Look at swapinfo -at and the total line.

I do not think that the SRM sees reserved memory as used so it may be that you have less memory than you think.

What values are dbc_max_pct and bufpages on your kernel

2)Are there errors in syslog.log or dmesg when this happens.


3)Is your patch level up to date.

Regards

Steve Steel

Quote of the moment
-------------------
"We are drowning in information but starved for knowledge."
-- John Naisbitt
If you want truly to understand something, try to change it. (Kurt Lewin)
John Bolene
Honored Contributor

Re: Paging issue/Performance Question.

Make sure you also have pseudo swap kernel parm turned on. swapmem_on
It is always a good day when you are launching rockets! http://tripolioklahoma.org, Mostly Missiles http://mostlymissiles.com
Stefan Farrelly
Honored Contributor

Re: Paging issue/Performance Question.


Im not convinced you are seeing a problem. The most reliable way to see page in's and out's is to use vmstat. Run vmstat for a while and watch the pi and po values - they should always be 0 (with maybe an ocassional 1) and see if they concur with SRM. If they are high then you do have a problem. Please report your findings.

eg.
vmstat 1 100
Im from Palmerston North, New Zealand, but somehow ended up in London...
joe_91
Super Advisor

Re: Paging issue/Performance Question.

Hi:

My dbc_max is 3 and min is 1 with bufpages 0.
I have attached two vmstat outputs vmstat(5 20) and vm(1 50).
Steve:

I do not think that the SRM sees reserved memory as used so it may be
that you have less memory than you think. - Can you expand on this? I mean are you using SRM?

Stefan:
Thanks. Please have a look at the vmstat and let me know.


Please have a look and help me here.

Thanks
Joe.
joe_91
Super Advisor

Re: Paging issue/Performance Question.

Here is vmstat with 3 20.

Thanks
Joe.
Steve Steel
Honored Contributor

Re: Paging issue/Performance Question.

Hi

If the bufpages is 0 then i think it will still take a lot of memory.


Related Parameters and System Values
bufpages interacts with nbuf as follows:
bufpages = 0, nbuf = 0: Enables dynamic buffer cache.
bufpages not zero, nbuf = zero: Creates BufPages/2 buffer headers
and allocates (bufpages 4 Kbytes) of buffer pool space at system
boot time.
bufpages = 0, nbuf not zero: Allocates Nbuf*2 pages of buffer pool
space and creates Nbuf headers at boot time..
bufpages not 0, nbuf not zero: Allocates BufPages pages of buffer
pool space and creates Nbuf buffer headers at boot time. If the two
values conflict such that it is impossible to configure a system


Please post swapinfo -at as well

Will look at data

Regards

Steve Steel

Quote of the moment
-------------------
"We are drowning in information but starved for knowledge."
-- John Naisbitt
If you want truly to understand something, try to change it. (Kurt Lewin)
Steve Steel
Honored Contributor

Re: Paging issue/Performance Question.

Hi


What version of this patch do you have

Patch Name: PHCO_27578

Patch Description: s700_800 11.00 vmstat(1) cumulative patch

Creation Date: 02/09/30

Post Date: 02/11/12

Hardware Platforms - OS Releases:
s700: 11.00
s800: 11.00
Regards

Steve Steel

Quote of the moment
-------------------
"We are drowning in information but starved for knowledge."
-- John Naisbitt
If you want truly to understand something, try to change it. (Kurt Lewin)
joe_91
Super Advisor

Re: Paging issue/Performance Question.

Thanks.
The vmstat patch is PHCO_21894. Also i am attaching the swapinfo output.

Thanks
Joe.
Stefan Farrelly
Honored Contributor

Re: Paging issue/Performance Question.

Hi Joe,

examined your vmstat output. It looks fine. CPU usage is fine, tons of free mem, not many running processes, your server looks pretty quiet. Now, the po rate is constantly 0 which is great - this matches with you having tons of memory free and no memory pressue which is good. However, pi (page-in's) is periodically high - but in your case this is not because anything had been paged out (due to memory pressue) but I think due to your application programs not loading themselves 100% into memory at run time -thus periodically they have to page in more of themselves as and when needed. This is an application issue - nothing to do with the OS - in my opinion.

You now need to look into your applications. What are the main processes (non system) running and how much memory are they using, how much shared mem, how big are they, can they expand to, etc.
Im from Palmerston North, New Zealand, but somehow ended up in London...
joe_91
Super Advisor

Re: Paging issue/Performance Question.

Hello Stefan:

Thanks for your time. We are connecting users from PC thru xemulator. All applications are xmotif applications (about 6) running on nclass. Regarding your suggestion of finding how much shared memory they use(how do i find them) also can you please say how you are saying it is an application issue(i mean what info in vmstat makes you say so). Please advice.

Thanks
Joe.
Steve Steel
Honored Contributor

Re: Paging issue/Performance Question.

Hi

I cannot read the swap info but this looks clearer

This is a quick and dirty way to see the size

#!/usr/bin/ksh
#
# Show processes sorted by size of core image
#
# Usage:
# psram [ quantity ]
#
# where quantity is the top RAM processes to show (default is 20)
#
set -u
if [ $# -gt 0 ]
then
TOPPROCS=$1
else
TOPPROCS=20
fi

MYNAME=$(basename $0)
TEMPFILE=/var/tmp/$MYNAME.$$
trap `rm -f $TEMPFILE > /dev/null 2>&1` 0 1 2 3 15

UNIX95= ps -e -o ruser,vsz,pid,args > $TEMPFILE
head -1 $TEMPFILE
DASH5="-----"
DASH25="$DASH5$DASH5$DASH5$DASH5$DASH5"
echo "$DASH5---- $DASH5- $DASH5 $DASH25$DASH25"
grep -v "VSZ COMMAND" $TEMPFILE | cut -c -78 | sort -rn -k2 | head -${TOPPROCS}
rm $TEMPFILE > /dev/null 2>&1
#### END OF SCRIPT


You may have a memory leak but if you run ps as shown without filtering lines but grep on your process name you can see size and if you do it several times you can see change.

Still seems as though you are just using a lot of memory because the machine can use it as it is there.

run top like this will give you a file with an overview

top -d5 -n60 -f /tmp/toplis

Regards

Steve Steel

Quote of the moment
-------------------
"We are drowning in information but starved for knowledge."
-- John Naisbitt
If you want truly to understand something, try to change it. (Kurt Lewin)
Stefan Farrelly
Honored Contributor

Re: Paging issue/Performance Question.


To look at shared memory usage of a process you need glance/gpm. Use it, select a process of a user and go into memory report/detail and it will show you how much of all types of memory it is using (stack, data, instruction, shared mem).

The reason I dont think its an OS issue is the vmstat page-out rate is constantly 0 - which is what you would expect as you have Gigs of memory free. ie. no memory pressue, so the OS does not page any processes out to disk (which would later require a page in when needed).
So - if the OS is NOT paging any procsses out but the page-in (pi) rate is sometimes a lot higher than 0 then it must be the individual appication process requesting more memory or data from disk which requires a page in. If the process locked itself into memory (a common application precaution - even ive used it to ensure no paging) then it wouldnt ever need any page-ins.

Now, looking at your application in detail is going to be easy if you have access to the developers and the source, but as your using some sort of emulator I think your going to have difficulty tracking it further. It could be something as simple as a programming bug which causes a memory leak which means app processes continually keep growing - and thus generate page-in's. Even HP has been known to have this bug (namely in the mib2agent process which would eventually grow to hundreds and hundreds of MB until HP released a patch for it). So, watch your app processes. As soon as one starts how much memory does it use ? as time passes does it stay the same size or keep growing ? if growing it looks like a development/app problem. Possibly the authors of have a patch or a new version which stops this ?


Im from Palmerston North, New Zealand, but somehow ended up in London...
MANOJ SRIVASTAVA
Honored Contributor

Re: Paging issue/Performance Question.

Joe

The system has lots of memeory what I would start is with increasing teh dbc_max to a atleast 6 or 7 + and leace teh min to 1 only and then rate the perfomance for some time period . The simple way to do is to check teh mem utilization in glance and see what is teh current avg and waht is the avg when the parameter was increased. there should be more mem utilization.


Manoj Srivastava
joe_91
Super Advisor

Re: Paging issue/Performance Question.

Thanks so much. I ran steve's script and i saw invariably 1 application process with a VSZ of 26020 and then 24000 and so on upto 17000 for the 20 process. Is it high steve.

Thanks
Joe.
Giri Sekar.
Trusted Contributor
Solution

Re: Paging issue/Performance Question.

Hi:

STOP. dbc_max to 7% of 20 gig. I would NOT recommend. Stefan's theory seems to good and i would not touch the kernel. Your po's and pi's are in single digit and that is good. You need not touch your kernel until you start to see two digits.

Thanks
Giri Sekar.
"USL" Unix as Second Language
Mladen Despic
Honored Contributor

Re: Paging issue/Performance Question.

Joe,

From what you describe, the system performance looks fine. You actually have a lot more physical memory than your application(s) need. Nevertheless, it is a good idea to check for processes leaking memory. Some suggestions are at the following link:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x6b0d72106351d5118fef0090279cd0f9,00.html

I think at least one of your applications probably uses memory-mapped files. Hence a small amount of pageouts may be there, but it's not a memory bottleneck. It is normal. (In Glance you could compare a "memory report" with a "syscall report." You would likely see that the small amount of pagouts is associated with the 'mmap' system call that your application(s) are making.)

To check for shared memory segments on your systems, run:

ipcs -mopb

You can get more information from Glance, of course. If you have Glance, you can try using the Glance adviser syntax posted as an attachement at the following link:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x61d4a848deccd61190050090279cd0f9,00.html

Mladen
joe_91
Super Advisor

Re: Paging issue/Performance Question.

Thanks. So i would not modify the dbc_max.

Thanks
Joe.
Mladen Despic
Honored Contributor

Re: Paging issue/Performance Question.

Joe,

I would not increase the current value dbc_max_pct=3 because 3% of 20 Gb is 600 Mb already. The usual guidelines for the maximum size of the buffer cache under HP-UX 11.00 bring a point about "system freezes" that may impact system performance if the buffer cache is too large. The recommendation is to keep the maximum size of the buffer cache around 300Mb-400 Mb. Now I don't think this is an absolute guideline. I think the current maximum size of 600 Mb will likely be fine.

That said, however, if you really want to tune the buffer cache for YOUR system, make sure to run something like:

sar -b 5 30

and check for %rcache and %wcache. If, ON AVERAGE, these values are at least 90% and 85% respectively, than your buffer cache need not be increased. You could then try decreasing the size of the buffer cache until you observe %rcache and %wcache dropping below the mentioned values. This will tell you how small your buffer cache can be without a significat impact on your system performance.

On the other hand, even though dbc_min_pct=1, the minimum size of the dynamic buffer cache can't effectively be smaller than 2% of the physical memory. In other words, if your buffer cache is configured as dynamic, there is no point in setting either dbc_max_pct or dbc_min_pct lower than 2.

In summary, you could abandon dynamic buffer cache completely, and set the fixed size of your buffer cache (next time you have a chance to do so), but I think the current dbc_max_pct and dbc_min_pct on your system are just fine. I would not schedule a system maintenance just for that!

I hope this helps a bit.

Mladen
MANOJ SRIVASTAVA
Honored Contributor

Re: Paging issue/Performance Question.

We have a system wiht 32 gb ram and we were recommended dbc max of 3 and min of 2 , this made us run into perfomance issues , the performace got better immediatelty after we bumped the dbc_max to 10 , the idea is too simple the way it works is that you do need to have more buffer and since u have more memory why not use it .



Manoj Srivastava
Mladen Despic
Honored Contributor

Re: Paging issue/Performance Question.

Manoj,

I think this really depends on a system. If the applications can make effective use of the large buffer cache, it may make sense to increase dbc_max_pct. In particular, if %rcache and %wcache (from 'sar -b') are below 90%, I would try increasing dbc_max_pct.

In Joe's case, no actual performance problem has been observed.

Mladen
Mladen Despic
Honored Contributor

Re: Paging issue/Performance Question.

Unless, of course, Joe can confirm otherwise ...

Joe,

Are you observing the actual performance degradation with your application, or was this mainly a concern about pageouts?

Mladen
joe_91
Super Advisor

Re: Paging issue/Performance Question.

Hi:

There is no performance degradation at all. It is only page outs. Occasionaly my vmstat shows po as 3. By the way i have opened one more discussion on the same subject.Thanks for all the ideas.

Thanks
Joe.
Mladen Despic
Honored Contributor

Re: Paging issue/Performance Question.

Joe,

Based on the information that you have provided, I don't see any reason for concern.

By the way, I could not find the glance report that you mention in your second post.

Regards,

Mladen