lsof cpu/memory utilization is high

[Naj]

HI Expert,

I have issue with  CPU and memory Utilization is high and some process occured due to this. when i checked into the server i noticed that rds service is higher.

My objective is

to findout what is root cause why cpu/memory utilization is high and this server running Data Protector.

I have some question

1.  did maxfiles related to this issue? so i need to extend more value on maxfiles_lim? or i need to put new request to upgrade memory? see below

server2b# lsof |wc -l
4284
server2b# kctune |grep -i maxfiles_lim
maxfiles_lim                       4096  Default     Immed

2. when i execute below command i found some process error, are this affect to cpu/mem utilization?

server2b#lsof -c rds

.

.

.

rds     27873 root  106u  unix         0x2497b980        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0x8247d100)
rds     27873 root  107u  unix         0xdb292680        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xc007b100)
rds     27873 root  109u  unix         0xda92e700        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xe7e3400)
rds     27873 root  110u  unix          0xa4c0d00        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0x819da380)
rds     27873 root  111u  unix         0xcc634380        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xd2c62380)
rds     27873 root  117u  unix         0xaad47680        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xa8a64a00)
rds     27873 root  120u  unix          0x8435680        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xcc079100)
rds     27873 root  121u  unix         0xbfa3fd00        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0x8443700)
rds     27873 root  123u  unix         0xa9970a00        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xf933fd00)
rds     27873 root  125u  unix         0xdb292c80        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xd87f1080)
^C

3. Total file running under rds is 145, are this normal?

server2b#lsof -c rds |wc -l
145

4. Top out showing user is high load

CPU   LOAD   USER   NICE    SYS   IDLE  BLOCK  SWAIT   INTR   SSYS
 0    1.39  53.5%   0.0%  37.2%   9.3%   0.0%   0.0%   0.0%   0.0%
 2    1.44  73.0%   0.0%  20.7%   6.4%   0.0%   0.0%   0.0%   0.0%
 4    1.40  60.2%   0.0%  31.8%   8.0%   0.0%   0.0%   0.0%   0.0%
 6    1.43  57.7%   0.0%  33.4%   8.9%   0.0%   0.0%   0.0%   0.0%
---   ----  -----  -----  -----  -----  -----  -----  -----  -----
avg   1.42  60.9%   0.0%  30.8%   8.3%   0.0%   0.0%   0.0%   0.0%

Please help me on this

Thanks

BR

Naj

[Dennis Handly]

>1.  Is maxfiles related to this issue? so I need to extend more value on maxfiles_lim?

maxfiles_lim is a policeman.  It prevents processes from going crazy.  But some programs need to have larger values.  The sysadmin needs to judge whether a useful application is more important that this rogue processes.

I'm not sure I see the connection between the default lsof output and maxfiles_lim?

>2. when I execute below command I found some process error, are this affect to cpu/mem utilization?

This is typically sloppiness in the code.  It may be trying to tell you that there is something wrong and may be just a warning.

>4. Top out showing user is high load

It looks like a lot of time is spent in the kernel.  Are the data protector processes the top ones?

Why are you using lsof to debug this performance problem?  Any particular reasons?

Do you have glance installed?  There is a trial version.

[Naj]

HI Handy,

> Are the data protector processes the top ones?

Yes, rds are the top ones

>Why are you using lsof to debug this performance problem?  Any particular reasons?

Do you have glance installed?  There is a trial version.

I've glance installed in server, i used that but it just giving basic info. Im looking more details, what the process running on that time causes processing high and is this because of any error trying to attempt. From that, i'll be able to further troubleshot

I've little knowledge in data protector, but if you guys have some info please share.

1. How to manage DP process in HPUX, i mean to reduce rds CPU processing in one time. if CPU/memory high load, it would be affect to backup activity and many process would be occur.

Thanks

BR

Naj

[Naj]

Any updates?

[Dennis Handly]

>I've glance installed in server, I used that but it just giving basic info.

glance gives you all the details you would ever want.  I don't see lsof helping here.

>what the process running on that time causes processing high

glance will give that.

>I mean to reduce rds CPU processing in one time. If CPU/memory high load, it would be affect to backup activity and many process would be occur.

I assume you want high CPU processing by rds so you can get faster backups.

[Pete Randall]

If Omniback/DataProtector is interfering with production then you either need to schedule your backups for off hours or move your OB/DP cell manager to another box.

[Hakki Aydin Ucar]

when DP is running , in most case the "rds" utilization is normal. this issue is new OR happens now and then ?

did you try a restart omni;

/opt/omni/sbin/omnisv -stop

/opt/omni/sbin/omnisv -start

sometimes it works.

[Naj · Accepted solution]

HI

Below is current reading on glance

 Glance C.04.70.001             20:54:34+     ia64                                                  Current  Avg  High
-----------------------------------------------------------------------------------------------------------------------------------
CPU  Util   S                     SAU                                          U                                  | 68%   68%   68%
Disk Util   F FV   V                                                                                              |  7%    7%    7%
Mem  Util   S                         SU        U                                                                 | 85%   85%   85%
Swap Util   U                   UR           R                                                                    | 33%   33%   33%
-----------------------------------------------------------------------------------------------------------------------------------
                                                           PROCESS LIST                                                Users=    3
                         User      CPU %    Thrd Disk        Memory       Block
Process Name     PID     Name   ( 400% max) Cnt  IOrate      RSS/VSS      On
--------------------------------------------------------------------------------
rds                27873 root        143.9    31   0.0   434.1mb  466.5mb SLEEP
glance             28714 root         18.2     1  29.2     696kb    940kb SLEEP
crs                27895 root          8.1     1   0.0    16.8mb   18.0mb SLEEP
ksh                19508 root          6.1     1   0.0     612kb    656kb  PIPE
ovconfd             2510 root          1.4    11   0.0     8.9mb   19.9mb SLEEP
midaemon           12128 root          0.7     4   0.0    41.1mb   42.5mb SLEEP
bsm                10039 root          0.0     1   0.0    35.2mb   36.6mb SLEEP
cmcld              28072 root          0.0    10   0.0    33.3mb   33.4mb SLEEP
coda               12132 root          0.0     9   0.0   596.3mb  661.3mb SLEEP
vxfsd                 70 root          0.0    15   8.5    24.4mb   27.4mb SLEEP
scopeux            12092 root          0.0     1   0.0    28.0mb   33.8mb SLEEP
httpd               9308 hpsmh         0.0    27   0.0    36.2mb   54.4mb SLEEP
kcmond              2667 root          0.0     1   0.0    41.3mb   41.8mb SLEEP
ovcd                2212 root          0.0    28   0.0    21.2mb   48.1mb SLEEP
cimprovagt          1866 root          0.0    26   0.0   145.6mb  188.1mb SLEEP
cimprovagt          1867 root          0.0     4   0.0    24.2mb   35.6mb SLEEP
bsm                10090 root          0.0     1   0.0    24.5mb   26.4mb SOCKT

Based on glance, rds consume 143.9 % CPU and current load for CPU is 68%, are this normal? sometime it hit till 80% constantly. 

Highlight

-This server running on cluster, any possible cluster has any affect (in sense of processing) to DP activity?

- When i execute lsof, i found died path/error path, are this will be affected to CPU load, my logic is, DP trying to trigger the path for sometime and this will be hang up and consume CPU load, are this true?

- If DP need to be tuning back, what the item need to emphasis ?

Why i ask that, because same configuration implement on other cluster server and their DP processing looks more stable rather than this server. So i would suspected, it must be *something* that must investigate. Since, i don't have knowledge in Data Protector, so  i need your kind advice on this to improve CPU load to be more stable.

Thanks

BR

Naj

[Naj]

Hi Hakki,

Restarting is only works for temporary, CPU load back to high after 2/3 hours.

[Pete Randall]

Well, you could always stop running backups.  That will certainly reduce your CPU load. :smileywink: