- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- Re: lsof cpu/memory utilization is high
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-20-2011 10:21 AM
07-20-2011 10:21 AM
HI Expert,
I have issue with CPU and memory Utilization is high and some process occured due to this. when i checked into the server i noticed that rds service is higher.
My objective is
to findout what is root cause why cpu/memory utilization is high and this server running Data Protector.
I have some question
1. did maxfiles related to this issue? so i need to extend more value on maxfiles_lim? or i need to put new request to upgrade memory? see below
server2b# lsof |wc -l
4284
server2b# kctune |grep -i maxfiles_lim
maxfiles_lim 4096 Default Immed
2. when i execute below command i found some process error, are this affect to cpu/mem utilization?
server2b#lsof -c rds
.
.
.
rds 27873 root 106u unix 0x2497b980 0t0 /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0x8247d100)
rds 27873 root 107u unix 0xdb292680 0t0 /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xc007b100)
rds 27873 root 109u unix 0xda92e700 0t0 /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xe7e3400)
rds 27873 root 110u unix 0xa4c0d00 0t0 /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0x819da380)
rds 27873 root 111u unix 0xcc634380 0t0 /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xd2c62380)
rds 27873 root 117u unix 0xaad47680 0t0 /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xa8a64a00)
rds 27873 root 120u unix 0x8435680 0t0 /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xcc079100)
rds 27873 root 121u unix 0xbfa3fd00 0t0 /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0x8443700)
rds 27873 root 123u unix 0xa9970a00 0t0 /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xf933fd00)
rds 27873 root 125u unix 0xdb292c80 0t0 /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xd87f1080)
^C
3. Total file running under rds is 145, are this normal?
server2b#lsof -c rds |wc -l
145
4. Top out showing user is high load
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 1.39 53.5% 0.0% 37.2% 9.3% 0.0% 0.0% 0.0% 0.0%
2 1.44 73.0% 0.0% 20.7% 6.4% 0.0% 0.0% 0.0% 0.0%
4 1.40 60.2% 0.0% 31.8% 8.0% 0.0% 0.0% 0.0% 0.0%
6 1.43 57.7% 0.0% 33.4% 8.9% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 1.42 60.9% 0.0% 30.8% 8.3% 0.0% 0.0% 0.0% 0.0%
Please help me on this
Thanks
BR
Naj
____________________________________________
:: Really appreciate if you could assign some points.
:: Don't know how to assign point? Click the KUDOS! star!
Solved! Go to Solution.
- Tags:
- lsof
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-20-2011 11:27 AM
07-20-2011 11:27 AM
Re: lsof cpu/memory utilization is high
>1. Is maxfiles related to this issue? so I need to extend more value on maxfiles_lim?
maxfiles_lim is a policeman. It prevents processes from going crazy. But some programs need to have larger values. The sysadmin needs to judge whether a useful application is more important that this rogue processes.
I'm not sure I see the connection between the default lsof output and maxfiles_lim?
>2. when I execute below command I found some process error, are this affect to cpu/mem utilization?
This is typically sloppiness in the code. It may be trying to tell you that there is something wrong and may be just a warning.
>4. Top out showing user is high load
It looks like a lot of time is spent in the kernel. Are the data protector processes the top ones?
Why are you using lsof to debug this performance problem? Any particular reasons?
Do you have glance installed? There is a trial version.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-20-2011 06:35 PM
07-20-2011 06:35 PM
Re: lsof cpu/memory utilization is high
HI Handy,
> Are the data protector processes the top ones?
Yes, rds are the top ones
>Why are you using lsof to debug this performance problem? Any particular reasons?
Do you have glance installed? There is a trial version.
I've glance installed in server, i used that but it just giving basic info. Im looking more details, what the process running on that time causes processing high and is this because of any error trying to attempt. From that, i'll be able to further troubleshot
I've little knowledge in data protector, but if you guys have some info please share.
1. How to manage DP process in HPUX, i mean to reduce rds CPU processing in one time. if CPU/memory high load, it would be affect to backup activity and many process would be occur.
Thanks
BR
Naj
____________________________________________
:: Really appreciate if you could assign some points.
:: Don't know how to assign point? Click the KUDOS! star!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2011 01:19 AM
07-21-2011 01:19 AM
Re: lsof cpu/memory utilization is high
Any updates?
____________________________________________
:: Really appreciate if you could assign some points.
:: Don't know how to assign point? Click the KUDOS! star!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2011 04:36 AM
07-21-2011 04:36 AM
Re: lsof cpu/memory utilization is high
>I've glance installed in server, I used that but it just giving basic info.
glance gives you all the details you would ever want. I don't see lsof helping here.
>what the process running on that time causes processing high
glance will give that.
>I mean to reduce rds CPU processing in one time. If CPU/memory high load, it would be affect to backup activity and many process would be occur.
I assume you want high CPU processing by rds so you can get faster backups.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2011 04:58 AM
07-21-2011 04:58 AM
Re: lsof cpu/memory utilization is high
If Omniback/DataProtector is interfering with production then you either need to schedule your backups for off hours or move your OB/DP cell manager to another box.
Pete
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2011 06:01 AM
07-21-2011 06:01 AM
Re: lsof cpu/memory utilization is high
when DP is running , in most case the "rds" utilization is normal. this issue is new OR happens now and then ?
did you try a restart omni;
/opt/omni/sbin/omnisv -stop
/opt/omni/sbin/omnisv -start
sometimes it works.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2011 06:11 AM
07-21-2011 06:11 AM
SolutionHI
Below is current reading on glance
Glance C.04.70.001 20:54:34+ ia64 Current Avg High
-----------------------------------------------------------------------------------------------------------------------------------
CPU Util S SAU U | 68% 68% 68%
Disk Util F FV V | 7% 7% 7%
Mem Util S SU U | 85% 85% 85%
Swap Util U UR R | 33% 33% 33%
-----------------------------------------------------------------------------------------------------------------------------------
PROCESS LIST Users= 3
User CPU % Thrd Disk Memory Block
Process Name PID Name ( 400% max) Cnt IOrate RSS/VSS On
--------------------------------------------------------------------------------
rds 27873 root 143.9 31 0.0 434.1mb 466.5mb SLEEP
glance 28714 root 18.2 1 29.2 696kb 940kb SLEEP
crs 27895 root 8.1 1 0.0 16.8mb 18.0mb SLEEP
ksh 19508 root 6.1 1 0.0 612kb 656kb PIPE
ovconfd 2510 root 1.4 11 0.0 8.9mb 19.9mb SLEEP
midaemon 12128 root 0.7 4 0.0 41.1mb 42.5mb SLEEP
bsm 10039 root 0.0 1 0.0 35.2mb 36.6mb SLEEP
cmcld 28072 root 0.0 10 0.0 33.3mb 33.4mb SLEEP
coda 12132 root 0.0 9 0.0 596.3mb 661.3mb SLEEP
vxfsd 70 root 0.0 15 8.5 24.4mb 27.4mb SLEEP
scopeux 12092 root 0.0 1 0.0 28.0mb 33.8mb SLEEP
httpd 9308 hpsmh 0.0 27 0.0 36.2mb 54.4mb SLEEP
kcmond 2667 root 0.0 1 0.0 41.3mb 41.8mb SLEEP
ovcd 2212 root 0.0 28 0.0 21.2mb 48.1mb SLEEP
cimprovagt 1866 root 0.0 26 0.0 145.6mb 188.1mb SLEEP
cimprovagt 1867 root 0.0 4 0.0 24.2mb 35.6mb SLEEP
bsm 10090 root 0.0 1 0.0 24.5mb 26.4mb SOCKT
Based on glance, rds consume 143.9 % CPU and current load for CPU is 68%, are this normal? sometime it hit till 80% constantly.
Highlight
-This server running on cluster, any possible cluster has any affect (in sense of processing) to DP activity?
- When i execute lsof, i found died path/error path, are this will be affected to CPU load, my logic is, DP trying to trigger the path for sometime and this will be hang up and consume CPU load, are this true?
- If DP need to be tuning back, what the item need to emphasis ?
Why i ask that, because same configuration implement on other cluster server and their DP processing looks more stable rather than this server. So i would suspected, it must be *something* that must investigate. Since, i don't have knowledge in Data Protector, so i need your kind advice on this to improve CPU load to be more stable.
Thanks
BR
Naj
____________________________________________
:: Really appreciate if you could assign some points.
:: Don't know how to assign point? Click the KUDOS! star!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2011 06:13 AM
07-21-2011 06:13 AM
Re: lsof cpu/memory utilization is high
Restarting is only works for temporary, CPU load back to high after 2/3 hours.
____________________________________________
:: Really appreciate if you could assign some points.
:: Don't know how to assign point? Click the KUDOS! star!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2011 06:17 AM
07-21-2011 06:17 AM
Re: lsof cpu/memory utilization is high
Well, you could always stop running backups. That will certainly reduce your CPU load. :smileywink:
Pete