- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- Re: Paging issue/Performance Question.
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Forums
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 05:34 AM
12-03-2002 05:34 AM
We have this issue on an Nclass server running 11.0 64bit with 20Gig memory. We run a tool called SRM (System Resource Manager), which would look into Processor utilization, run queue and Page>Limit/Sec (This figure indicates the amount of time that the number of paged out virtual memory pages was greater than the Limit per second threshold). On this box it shows that the page>Limit/Sec has reached high levels (yellow) while the same data also shows that it has 13.4 Gig of free memory at all times. With this kind of free memory, I would assume that the machine never has to swap memory pages. Obviously, the tool uses commands such as sar, vmstat, swapinfo and other popular Unix commands. This is a critical performance indicator because a great decline in performance occurs once the system passes this threshold. Please advice where to start and why this may happen and what to look into.
Thanks
Joe.
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 05:47 AM
12-03-2002 05:47 AM
Re: Paging issue/Performance Question.
live free or die
harry
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 05:51 AM
12-03-2002 05:51 AM
Re: Paging issue/Performance Question.
What is running on the box.
1)Look at swapinfo -at and the total line.
I do not think that the SRM sees reserved memory as used so it may be that you have less memory than you think.
What values are dbc_max_pct and bufpages on your kernel
2)Are there errors in syslog.log or dmesg when this happens.
3)Is your patch level up to date.
Regards
Steve Steel
Quote of the moment
-------------------
"We are drowning in information but starved for knowledge."
-- John Naisbitt
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 05:56 AM
12-03-2002 05:56 AM
Re: Paging issue/Performance Question.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 05:57 AM
12-03-2002 05:57 AM
Re: Paging issue/Performance Question.
Im not convinced you are seeing a problem. The most reliable way to see page in's and out's is to use vmstat. Run vmstat for a while and watch the pi and po values - they should always be 0 (with maybe an ocassional 1) and see if they concur with SRM. If they are high then you do have a problem. Please report your findings.
eg.
vmstat 1 100
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 07:05 AM
12-03-2002 07:05 AM
Re: Paging issue/Performance Question.
My dbc_max is 3 and min is 1 with bufpages 0.
I have attached two vmstat outputs vmstat(5 20) and vm(1 50).
Steve:
I do not think that the SRM sees reserved memory as used so it may be
that you have less memory than you think. - Can you expand on this? I mean are you using SRM?
Stefan:
Thanks. Please have a look at the vmstat and let me know.
Please have a look and help me here.
Thanks
Joe.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 07:05 AM
12-03-2002 07:05 AM
Re: Paging issue/Performance Question.
Thanks
Joe.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 07:09 AM
12-03-2002 07:09 AM
Re: Paging issue/Performance Question.
If the bufpages is 0 then i think it will still take a lot of memory.
Related Parameters and System Values
bufpages interacts with nbuf as follows:
bufpages = 0, nbuf = 0: Enables dynamic buffer cache.
bufpages not zero, nbuf = zero: Creates BufPages/2 buffer headers
and allocates (bufpages 4 Kbytes) of buffer pool space at system
boot time.
bufpages = 0, nbuf not zero: Allocates Nbuf*2 pages of buffer pool
space and creates Nbuf headers at boot time..
bufpages not 0, nbuf not zero: Allocates BufPages pages of buffer
pool space and creates Nbuf buffer headers at boot time. If the two
values conflict such that it is impossible to configure a system
Please post swapinfo -at as well
Will look at data
Regards
Steve Steel
Quote of the moment
-------------------
"We are drowning in information but starved for knowledge."
-- John Naisbitt
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 07:15 AM
12-03-2002 07:15 AM
Re: Paging issue/Performance Question.
What version of this patch do you have
Patch Name: PHCO_27578
Patch Description: s700_800 11.00 vmstat(1) cumulative patch
Creation Date: 02/09/30
Post Date: 02/11/12
Hardware Platforms - OS Releases:
s700: 11.00
s800: 11.00
Regards
Steve Steel
Quote of the moment
-------------------
"We are drowning in information but starved for knowledge."
-- John Naisbitt
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 07:28 AM
12-03-2002 07:28 AM
Re: Paging issue/Performance Question.
The vmstat patch is PHCO_21894. Also i am attaching the swapinfo output.
Thanks
Joe.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 07:29 AM
12-03-2002 07:29 AM
Re: Paging issue/Performance Question.
examined your vmstat output. It looks fine. CPU usage is fine, tons of free mem, not many running processes, your server looks pretty quiet. Now, the po rate is constantly 0 which is great - this matches with you having tons of memory free and no memory pressue which is good. However, pi (page-in's) is periodically high - but in your case this is not because anything had been paged out (due to memory pressue) but I think due to your application programs not loading themselves 100% into memory at run time -thus periodically they have to page in more of themselves as and when needed. This is an application issue - nothing to do with the OS - in my opinion.
You now need to look into your applications. What are the main processes (non system) running and how much memory are they using, how much shared mem, how big are they, can they expand to, etc.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 07:39 AM
12-03-2002 07:39 AM
Re: Paging issue/Performance Question.
Thanks for your time. We are connecting users from PC thru xemulator. All applications are xmotif applications (about 6) running on nclass. Regarding your suggestion of finding how much shared memory they use(how do i find them) also can you please say how you are saying it is an application issue(i mean what info in vmstat makes you say so). Please advice.
Thanks
Joe.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 07:52 AM
12-03-2002 07:52 AM
Re: Paging issue/Performance Question.
I cannot read the swap info but this looks clearer
This is a quick and dirty way to see the size
#!/usr/bin/ksh
#
# Show processes sorted by size of core image
#
# Usage:
# psram [ quantity ]
#
# where quantity is the top RAM processes to show (default is 20)
#
set -u
if [ $# -gt 0 ]
then
TOPPROCS=$1
else
TOPPROCS=20
fi
MYNAME=$(basename $0)
TEMPFILE=/var/tmp/$MYNAME.$$
trap `rm -f $TEMPFILE > /dev/null 2>&1` 0 1 2 3 15
UNIX95= ps -e -o ruser,vsz,pid,args > $TEMPFILE
head -1 $TEMPFILE
DASH5="-----"
DASH25="$DASH5$DASH5$DASH5$DASH5$DASH5"
echo "$DASH5---- $DASH5- $DASH5 $DASH25$DASH25"
grep -v "VSZ COMMAND" $TEMPFILE | cut -c -78 | sort -rn -k2 | head -${TOPPROCS}
rm $TEMPFILE > /dev/null 2>&1
#### END OF SCRIPT
You may have a memory leak but if you run ps as shown without filtering lines but grep on your process name you can see size and if you do it several times you can see change.
Still seems as though you are just using a lot of memory because the machine can use it as it is there.
run top like this will give you a file with an overview
top -d5 -n60 -f /tmp/toplis
Regards
Steve Steel
Quote of the moment
-------------------
"We are drowning in information but starved for knowledge."
-- John Naisbitt
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 07:57 AM
12-03-2002 07:57 AM
Re: Paging issue/Performance Question.
To look at shared memory usage of a process you need glance/gpm. Use it, select a process of a user and go into memory report/detail and it will show you how much of all types of memory it is using (stack, data, instruction, shared mem).
The reason I dont think its an OS issue is the vmstat page-out rate is constantly 0 - which is what you would expect as you have Gigs of memory free. ie. no memory pressue, so the OS does not page any processes out to disk (which would later require a page in when needed).
So - if the OS is NOT paging any procsses out but the page-in (pi) rate is sometimes a lot higher than 0 then it must be the individual appication process requesting more memory or data from disk which requires a page in. If the process locked itself into memory (a common application precaution - even ive used it to ensure no paging) then it wouldnt ever need any page-ins.
Now, looking at your application in detail is going to be easy if you have access to the developers and the source, but as your using some sort of emulator I think your going to have difficulty tracking it further. It could be something as simple as a programming bug which causes a memory leak which means app processes continually keep growing - and thus generate page-in's. Even HP has been known to have this bug (namely in the mib2agent process which would eventually grow to hundreds and hundreds of MB until HP released a patch for it). So, watch your app processes. As soon as one starts how much memory does it use ? as time passes does it stay the same size or keep growing ? if growing it looks like a development/app problem. Possibly the authors of have a patch or a new version which stops this ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 08:04 AM
12-03-2002 08:04 AM
Re: Paging issue/Performance Question.
The system has lots of memeory what I would start is with increasing teh dbc_max to a atleast 6 or 7 + and leace teh min to 1 only and then rate the perfomance for some time period . The simple way to do is to check teh mem utilization in glance and see what is teh current avg and waht is the avg when the parameter was increased. there should be more mem utilization.
Manoj Srivastava
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 08:09 AM
12-03-2002 08:09 AM
Re: Paging issue/Performance Question.
Thanks
Joe.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 08:28 AM
12-03-2002 08:28 AM
SolutionSTOP. dbc_max to 7% of 20 gig. I would NOT recommend. Stefan's theory seems to good and i would not touch the kernel. Your po's and pi's are in single digit and that is good. You need not touch your kernel until you start to see two digits.
Thanks
Giri Sekar.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 09:48 AM
12-03-2002 09:48 AM
Re: Paging issue/Performance Question.
From what you describe, the system performance looks fine. You actually have a lot more physical memory than your application(s) need. Nevertheless, it is a good idea to check for processes leaking memory. Some suggestions are at the following link:
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x6b0d72106351d5118fef0090279cd0f9,00.html
I think at least one of your applications probably uses memory-mapped files. Hence a small amount of pageouts may be there, but it's not a memory bottleneck. It is normal. (In Glance you could compare a "memory report" with a "syscall report." You would likely see that the small amount of pagouts is associated with the 'mmap' system call that your application(s) are making.)
To check for shared memory segments on your systems, run:
ipcs -mopb
You can get more information from Glance, of course. If you have Glance, you can try using the Glance adviser syntax posted as an attachement at the following link:
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x61d4a848deccd61190050090279cd0f9,00.html
Mladen
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 09:52 AM
12-03-2002 09:52 AM
Re: Paging issue/Performance Question.
Thanks
Joe.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 11:04 AM
12-03-2002 11:04 AM
Re: Paging issue/Performance Question.
I would not increase the current value dbc_max_pct=3 because 3% of 20 Gb is 600 Mb already. The usual guidelines for the maximum size of the buffer cache under HP-UX 11.00 bring a point about "system freezes" that may impact system performance if the buffer cache is too large. The recommendation is to keep the maximum size of the buffer cache around 300Mb-400 Mb. Now I don't think this is an absolute guideline. I think the current maximum size of 600 Mb will likely be fine.
That said, however, if you really want to tune the buffer cache for YOUR system, make sure to run something like:
sar -b 5 30
and check for %rcache and %wcache. If, ON AVERAGE, these values are at least 90% and 85% respectively, than your buffer cache need not be increased. You could then try decreasing the size of the buffer cache until you observe %rcache and %wcache dropping below the mentioned values. This will tell you how small your buffer cache can be without a significat impact on your system performance.
On the other hand, even though dbc_min_pct=1, the minimum size of the dynamic buffer cache can't effectively be smaller than 2% of the physical memory. In other words, if your buffer cache is configured as dynamic, there is no point in setting either dbc_max_pct or dbc_min_pct lower than 2.
In summary, you could abandon dynamic buffer cache completely, and set the fixed size of your buffer cache (next time you have a chance to do so), but I think the current dbc_max_pct and dbc_min_pct on your system are just fine. I would not schedule a system maintenance just for that!
I hope this helps a bit.
Mladen
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 11:24 AM
12-03-2002 11:24 AM
Re: Paging issue/Performance Question.
Manoj Srivastava
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 11:38 AM
12-03-2002 11:38 AM
Re: Paging issue/Performance Question.
I think this really depends on a system. If the applications can make effective use of the large buffer cache, it may make sense to increase dbc_max_pct. In particular, if %rcache and %wcache (from 'sar -b') are below 90%, I would try increasing dbc_max_pct.
In Joe's case, no actual performance problem has been observed.
Mladen
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 11:46 AM
12-03-2002 11:46 AM
Re: Paging issue/Performance Question.
Joe,
Are you observing the actual performance degradation with your application, or was this mainly a concern about pageouts?
Mladen
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 12:05 PM
12-03-2002 12:05 PM
Re: Paging issue/Performance Question.
There is no performance degradation at all. It is only page outs. Occasionaly my vmstat shows po as 3. By the way i have opened one more discussion on the same subject.Thanks for all the ideas.
Thanks
Joe.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2002 12:12 PM
12-03-2002 12:12 PM
Re: Paging issue/Performance Question.
Based on the information that you have provided, I don't see any reason for concern.
By the way, I could not find the glance report that you mention in your second post.
Regards,
Mladen