- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- Excessive page-outs, even after memory upgrade
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-31-2009 08:08 AM
тАО12-31-2009 08:08 AM
Originally, we were running with 12GB of RAM. We were seeing massive paging activity and were several gigabytes into swap.
We have since increased the system RAM to 32GB. We are no longer into physical disk-based swap, but swapinfo is still showing 1000-1500 page-outs a second, and total page-faults are running in the 8-10 thousand range.
Since we are using 0 blocks of physical swap, I'm guessing the page-outs are going to pseudo-swap.
Glance is showing 30.4GB of pseudo-swap available, and 7.2GB of it used on the system.
How can I get this sucker to stop paging out so much? We're only 24% utilized on physical memory at the current time. Our shmmax is 4GB and I have the buffer cache constrained to 10$ of available memory. What other kernel parameters do we need to check?
Also, I know that page-in counts include code loaded from disk to be run from memory. Do page-out counts include data written to disk by programs? The manpage for vmstat isn't clear on that.
Thanks!
Andrew
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-31-2009 08:27 AM
тАО12-31-2009 08:27 AM
Re: Excessive page-outs, even after memory upgrade
Lets take a look at application usage.
Check the oracle SGA's and such. Trim your buffer pool if possible, to free up memory.
Lets take a look at glance and see what processes are using how much memory. Lets identify the porcess actually using the memory (top can do this) and take action on that basis.
Remember, an application will reserve swap even if it never uses it.
SEP
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-31-2009 08:33 AM
тАО12-31-2009 08:33 AM
Re: Excessive page-outs, even after memory upgrade
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-31-2009 08:37 AM
тАО12-31-2009 08:37 AM
Re: Excessive page-outs, even after memory upgrade
swapinfo(1m) doesn't show that. (Please provide the "swapinfo -tam" and vmstat output.)
Did you mean vmstat(1)?
>Since we are using 0 blocks of physical swap, I'm guessing the page-outs are going to pseudo-swap.
I'm not sure how useful this is for the kernel to do this?
>Do page-out counts include data written to disk by programs?
It depends on whether you have mapped files.
I'm not sure if it applies to the file cache on 11.31?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-31-2009 10:14 AM
тАО12-31-2009 10:14 AM
Re: Excessive page-outs, even after memory upgrade
>swapinfo(1m) doesn't show that. (Please provide the "swapinfo -tam" and vmstat output.)
>Did you mean vmstat(1)?
Yes, I did. Sorry.
>>Since we are using 0 blocks of physical swap, I'm guessing the page-outs are going to pseudo-swap.
>I'm not sure how useful this is for the kernel to do this?
It's not useful at all, but that doesn't stop the system from doing it. The way I understand pseudo swap is the VM requires there to be a page of swap for every page of memory allocated, and they came up with pseudo swap so that they could create the pages in memory instead of on disk when available RAM allowed it.
And perhaps my page-outs are simply page creations in pseudo swap and not actual page outs? I don't know, and I don't know how to tell.
>>Do page-out counts include data written to disk by programs?
>It depends on whether you have mapped files.
I'm not sure if it applies to the file cache on 11.31?
I don't know, either.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-31-2009 10:14 AM
тАО12-31-2009 10:14 AM
Re: Excessive page-outs, even after memory upgrade
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-31-2009 11:08 AM
тАО12-31-2009 11:08 AM
Re: Excessive page-outs, even after memory upgrade
I would assume it isn't doing it and there is some other cause.
>they came up with pseudo swap so that they could create the pages in memory instead of on disk when available RAM allowed it.
Yes. The page is its own swap area.
>perhaps my page-outs are simply page creations in pseudo swap and not actual page outs?
I would assume they would suppress these as confusing. Or put them under some other statistic.
>I don't know, either.
Are you using 11.31? Do you have all the latest VM patches?
Your attachment says it is waiting for I/O.
It has FS Reads/Writes. No VM Reads/Writes. Some System Writes.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-31-2009 12:04 PM
тАО12-31-2009 12:04 PM
Re: Excessive page-outs, even after memory upgrade
As for the process being in IO Wait, yes. Everything is almost always in IO wait on this system. That's the core problem I'm trying to solve: why is everything in IO Wait all the time?
The system has two 2GB fiber channel cards and has its own EVA 8400 with 91 spindles (all in one disk group). We have CA eHealth watching the fabric and the server, and its telling me the fiber channel cards are under 10% utilized, but my disk is grinding away at 30-70%. My CPU loads are hovering around 70%, mostly due to IO wait.
I ran evaperf against the EVA and processed the results through TLVIZ and found a few nasty spikes, but on average we're not hurting for storage speed and my controller utilization is really low. My write latencies spike to 9 or 10ms during a few really heavy spots where multiple instances are applying transaction logs at the same time, but on average are hovering around 4ms.
I have a bottleneck somewhere, I just can't find it. These excessive page-outs seemed like a good place to start looking.
I checked with the DBAs, and the total SGA allocated for all of the database instances was just 4GB, as they had not yet increased them from before the memory upgrade.
Would having the SGA size set too low cause excessive paging?
Oh, and this system is ~84% WRITE on its data disks. So consider that in any parameter change recommendations.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-31-2009 04:01 PM
тАО12-31-2009 04:01 PM
Re: Excessive page-outs, even after memory upgrade
You might have pageins which are ok but not pageouts. You may be misunderstanding the output of the vmstat memory report. You will not get pageouts unless you are low on memory and all of it is used.
You might want to increase the amount of filecache_max
How is your oracle databases setup. JFS filesystem, RAC with CFS or Raw Lvs ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-31-2009 04:21 PM
тАО12-31-2009 04:21 PM
Re: Excessive page-outs, even after memory upgrade
>My CPU loads are hovering around 70%, mostly due to IO wait.
These are separate/opposite. You could be up to 30% IO wait.
>I have a bottleneck somewhere, I just can't find it. These excessive page-outs seemed like a good place to start looking.
How do you know the page-outs are for these processes?
>Would having the SGA size set too low cause excessive paging?
Or lots of I/O.