topic Re: lsof cpu/memory utilization is high in Operating System - HP-UX

lsof cpu/memory utilization is high

Naj — Wed, 20 Jul 2011 17:21:25 GMT

HI Expert,

I have issue with CPU and memory Utilization is high and some process occured due to this. when i checked into the server i noticed that rds service is higher.

My objective is

to findout what is root cause why cpu/memory utilization is high and this server running Data Protector.

I have some question

1. did maxfiles related to this issue? so i need to extend more value on maxfiles_lim? or i need to put new request to upgrade memory? see below

server2b# lsof |wc -l
4284
server2b# kctune |grep -i maxfiles_lim
maxfiles_lim 4096 Default Immed

2. when i execute below command i found some process error, are this affect to cpu/mem utilization?

server2b#lsof -c rds

rds     27873 root 106u unix         0x2497b980        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0x8247d100)
rds     27873 root 107u unix         0xdb292680        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xc007b100)
rds     27873 root 109u unix         0xda92e700        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xe7e3400)
rds     27873 root 110u unix          0xa4c0d00        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0x819da380)
rds     27873 root 111u unix         0xcc634380        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xd2c62380)
rds     27873 root 117u unix         0xaad47680        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xa8a64a00)
rds     27873 root 120u unix          0x8435680        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xcc079100)
rds     27873 root 121u unix         0xbfa3fd00        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0x8443700)
rds     27873 root 123u unix         0xa9970a00        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xf933fd00)
rds     27873 root 125u unix         0xdb292c80        0t0       /var/opt/omni/tmp/ud_ob2_40 (Error: Invalid argument) (->0xd87f1080)
^C

3. Total file running under rds is 145, are this normal?

server2b#lsof -c rds |wc -l
145

4. Top out showing user is high load

CPU   LOAD   USER   NICE    SYS   IDLE BLOCK SWAIT   INTR   SSYS
0    1.39 53.5%   0.0% 37.2%   9.3%   0.0%   0.0%   0.0%   0.0%
2    1.44 73.0%   0.0% 20.7%   6.4%   0.0%   0.0%   0.0%   0.0%
4    1.40 60.2%   0.0% 31.8%   8.0%   0.0%   0.0%   0.0%   0.0%
6    1.43 57.7%   0.0% 33.4%   8.9%   0.0%   0.0%   0.0%   0.0%
---   ---- ----- ----- ----- ----- ----- ----- ----- -----
avg   1.42 60.9%   0.0% 30.8%   8.3%   0.0%   0.0%   0.0%   0.0%

Please help me on this

Thanks

Naj

Re: lsof cpu/memory utilization is high

Dennis Handly — Wed, 20 Jul 2011 18:27:21 GMT

>1. Is maxfiles related to this issue? so I need to extend more value on maxfiles_lim?

maxfiles_lim is a policeman. It prevents processes from going crazy. But some programs need to have larger values. The sysadmin needs to judge whether a useful application is more important that this rogue processes.

I'm not sure I see the connection between the default lsof output and maxfiles_lim?

>2. when I execute below command I found some process error, are this affect to cpu/mem utilization?

This is typically sloppiness in the code. It may be trying to tell you that there is something wrong and may be just a warning.

>4. Top out showing user is high load

It looks like a lot of time is spent in the kernel. Are the data protector processes the top ones?

Why are you using lsof to debug this performance problem? Any particular reasons?

Do you have glance installed? There is a trial version.

Re: lsof cpu/memory utilization is high

Naj — Thu, 21 Jul 2011 01:35:22 GMT

HI Handy,

> Are the data protector processes the top ones?

Yes, rds are the top ones

>Why are you using lsof to debug this performance problem? Any particular reasons?

Do you have glance installed? There is a trial version.

I've glance installed in server, i used that but it just giving basic info. Im looking more details, what the process running on that time causes processing high and is this because of any error trying to attempt. From that, i'll be able to further troubleshot

I've little knowledge in data protector, but if you guys have some info please share.

1. How to manage DP process in HPUX, i mean to reduce rds CPU processing in one time. if CPU/memory high load, it would be affect to backup activity and many process would be occur.

Thanks

Naj

Re: lsof cpu/memory utilization is high

Naj — Thu, 21 Jul 2011 08:19:57 GMT

Any updates?

Re: lsof cpu/memory utilization is high

Dennis Handly — Thu, 21 Jul 2011 11:36:12 GMT

>I've glance installed in server, I used that but it just giving basic info.

glance gives you all the details you would ever want. I don't see lsof helping here.

>what the process running on that time causes processing high

glance will give that.

>I mean to reduce rds CPU processing in one time. If CPU/memory high load, it would be affect to backup activity and many process would be occur.

I assume you want high CPU processing by rds so you can get faster backups.

Re: lsof cpu/memory utilization is high

Pete Randall — Thu, 21 Jul 2011 11:58:08 GMT

If Omniback/DataProtector is interfering with production then you either need to schedule your backups for off hours or move your OB/DP cell manager to another box.

Re: lsof cpu/memory utilization is high

Hakki Aydin Ucar — Thu, 21 Jul 2011 13:01:52 GMT

when DP is running , in most case the "rds" utilization is normal. this issue is new OR happens now and then ?

did you try a restart omni;

/opt/omni/sbin/omnisv -stop

/opt/omni/sbin/omnisv -start

sometimes it works.

Re: lsof cpu/memory utilization is high

Naj — Thu, 21 Jul 2011 13:11:48 GMT

Below is current reading on glance

Glance C.04.70.001             20:54:34+     ia64                                                  Current Avg High
-----------------------------------------------------------------------------------------------------------------------------------
CPU Util   S                     SAU                                          U                                  | 68%   68%   68%
Disk Util   F FV   V                                                                                              | 7%    7%    7%
Mem Util   S                         SU        U                                                                 | 85%   85%   85%
Swap Util   U                   UR           R                                                                    | 33%   33%   33%
-----------------------------------------------------------------------------------------------------------------------------------
                                                           PROCESS LIST                                                Users=    3
                         User      CPU %    Thrd Disk        Memory       Block
Process Name     PID     Name   ( 400% max) Cnt IOrate      RSS/VSS      On
--------------------------------------------------------------------------------
rds                27873 root        143.9    31   0.0   434.1mb 466.5mb SLEEP
glance             28714 root         18.2     1 29.2     696kb    940kb SLEEP
crs                27895 root          8.1     1   0.0    16.8mb   18.0mb SLEEP
ksh                19508 root          6.1     1   0.0     612kb    656kb PIPE
ovconfd             2510 root          1.4    11   0.0     8.9mb   19.9mb SLEEP
midaemon           12128 root          0.7     4   0.0    41.1mb   42.5mb SLEEP
bsm                10039 root          0.0     1   0.0    35.2mb   36.6mb SLEEP
cmcld              28072 root          0.0    10   0.0    33.3mb   33.4mb SLEEP
coda               12132 root          0.0     9   0.0   596.3mb 661.3mb SLEEP
vxfsd                 70 root          0.0    15   8.5    24.4mb   27.4mb SLEEP
scopeux            12092 root          0.0     1   0.0    28.0mb   33.8mb SLEEP
httpd               9308 hpsmh         0.0    27   0.0    36.2mb   54.4mb SLEEP
kcmond              2667 root          0.0     1   0.0    41.3mb   41.8mb SLEEP
ovcd                2212 root          0.0    28   0.0    21.2mb   48.1mb SLEEP
cimprovagt          1866 root          0.0    26   0.0   145.6mb 188.1mb SLEEP
cimprovagt          1867 root          0.0     4   0.0    24.2mb   35.6mb SLEEP
bsm                10090 root          0.0     1   0.0    24.5mb   26.4mb SOCKT

Based on glance, rds consume 143.9 % CPU and current load for CPU is 68%, are this normal? sometime it hit till 80% constantly.

Highlight

-This server running on cluster, any possible cluster has any affect (in sense of processing) to DP activity?

- When i execute lsof, i found died path/error path, are this will be affected to CPU load, my logic is, DP trying to trigger the path for sometime and this will be hang up and consume CPU load, are this true?

- If DP need to be tuning back, what the item need to emphasis ?

Why i ask that, because same configuration implement on other cluster server and their DP processing looks more stable rather than this server. So i would suspected, it must be *something* that must investigate. Since, i don't have knowledge in Data Protector, so i need your kind advice on this to improve CPU load to be more stable.

Thanks

Naj

Re: lsof cpu/memory utilization is high

Naj — Thu, 21 Jul 2011 13:13:21 GMT

Hi Hakki,

Restarting is only works for temporary, CPU load back to high after 2/3 hours.

Re: lsof cpu/memory utilization is high

Pete Randall — Thu, 21 Jul 2011 13:17:00 GMT

Well, you could always stop running backups. That will certainly reduce your CPU load. :smileywink:

Re: lsof cpu/memory utilization is high

Hakki Aydin Ucar — Thu, 21 Jul 2011 14:02:59 GMT

I cannot say normal due to it depends on your system perfomance and capacity , and conditions like backup is running OR another performance issue effect this. But I remember more than 100% utily of rds in some customer. According to your CPU usage no cpu bottleneck yet, but 144% seems a bit high.

it reminds of something I faced in past, I had solved similar issue with a missing linker patch ,but my hw was PA-RISC not Integrity..

Re: lsof cpu/memory utilization is high

Naj — Thu, 21 Jul 2011 14:11:40 GMT

Hi Hakki,

>I had solved similar issue with a missing linker patch
could you please more details on this,means after patch the process avg below 100%?

Re: lsof cpu/memory utilization is high

Hakki Aydin Ucar — Fri, 22 Jul 2011 06:00:25 GMT

>could you please more details on this,means after patch the process avg below 100%?
All I remember about similar issue (that is not same because my omni start got stuck with high rds not started and reached above 100%), but I saw %90 around in most of healthy systems. So , somehow 100% not indicates a problem directly as I said before.

My solution was to put patch PHSS_41214 , watch out for this PA-RISC only !

I hope it igives some clue and my explanation makes sense.