Resource Utilization is very high

[Bunti]

Hello,

We have one production box which is UP n running from last 910days.

Output of glance is as below --

Most of the resources are being used by one of the application called medpap

any alternative how can we reduce load on this system?

B3692A GlancePlus C.03.86.00    01:03:38 vfauh427 9000/800                        Current  Avg  High

----------------------------------------------------------------------------------------------------

Cpu  Util   S                         SN                                  NU    U  | 99%   99%   99%

Disk Util   F                                                                 F    | 96%   91%  100%

Mem  Util   S                       SU                                UB     B     | 93%   93%   93%

Swap Util   U                    UR                    R                           | 64%   61%   64%

----------------------------------------------------------------------------------------------------

                                            PROCESS LIST                                Users=    5

                              User      CPU Util   **bleep**     Disk             Thd

Process Name  PID   PPID  Pri Name    (1000% max)  CPU    IO Rate    RSS    Cnt

--------------------------------------------------------------------------------

arm_CV       18223  18146 253 medPap   67.2/45.9     4.7  0.0/ 0.5   6.2mb    1

arm_CV       18233  18166 253 medPap   57.6/38.6     4.0  0.0/ 0.1   7.4mb    1

arm_CV       18395  18350 253 medPap   54.2/54.2     2.8  0.3/ 0.3   6.2mb    1

arm_CV       16625  16564 253 medPap   52.5/50.6    12.9 29.4/26.7   6.3mb    1

arm_CV       16852  16802 253 medPap   50.4/47.9    12.2 53.8/39.6   6.1mb    1

arm_CV       17510  17467 253 medPap   50.0/54.7     8.6 20.9/21.0   6.1mb    1

arm_CV       17896  17856 253 medPap   48.1/52.2     5.4 32.1/31.1   6.0mb    1

oracleMED05   1069      1 149 medP3db  41.2/28.7    64.5 1581/ 762  22.8mb    1

arm_CV       18019  17968 253 medPap   33.6/43.0     4.4  0.0/ 0.3   7.5mb    1

ksh          28410   4682 158 medPap   33.2/34.8  3800.1  0.0/ 0.1 392.2mb    1

ksh          28402   4682 158 medPap   23.7/26.0  2834.1  0.0/ 0.1 372.8mb    1

oracleMED05  16094      1 239 medP3db  22.9/26.4     9.5  251/ 280  22.9mb    1

oracleMED05   6545      1 239 medP3db  22.5/28.9    48.8  274/ 418  22.8mb    1

arm_CV       18536  18488 223 medPap   22.3/22.3     1.2  1.3/ 1.3   7.5mb    1

ftm           4808      1 154 medPap   17.2/17.8  1943.2 24.4/16.4  21.6mb    1

arm_fr       18232  18166 154 medPap   13.5/ 8.8     0.9 17.1/10.9   2.3mb    1

ftcd          4682      1 239 medPap   12.6/14.8  1611.1  0.0/ 0.0  11.4mb    1

arm_fr       18222  18146 154 medPap   11.6/ 8.0     0.8 15.1/10.0   2.4mb    1

arm_fr       18394  18350 154 medPap    9.4/ 9.4     0.5 11.3/11.3   2.9mb    1

sqlplus      18284  28410 149 medPap    7.4/ 7.3     0.4  0.5/ 0.5   3.4mb    1

arm_fr       18018  17968 218 medPap    6.7/ 9.2     1.0  8.4/11.5   2.4mb    1

arm_fr       16624  16564 240 medPap    5.9/ 5.6     1.4  7.3/ 6.9   3.7mb    1

vfauh427[/home/yg09958]$ more /etc/passwd|grep medPap

medPap:*:6130:2130:Mediation application owner:/med/p31100/comptel/medPap:/usr/bin/ksh

vfauh427[/home/yg09958]$ more /etc/passwd| grep medP3db

medP3db:*:7701:2889:Oracle Mediation database owner:/usr/users/medP3db:/usr/bin/ksh

vfauh427[/home/yg09958]$ ps -eaf|grep arm_CV

  medPap   432   384 254 00:59:38 ?         0:04 /med/p31100/comptel/medPap/amd/arm/bin/arm_CV -e ICC3 -i 182935

  medPap   520   472 255 00:59:38 ?         0:04 /med/p31100/comptel/medPap/amd/arm/bin/arm_CV -e ICC1 -i 182935

 yg09958   799 21624  0 00:59:43 pts/8     0:00 grep arm_CV

  medPap   349   296 255 00:59:37 ?         0:04 /med/p31100/comptel/medPap/amd/arm/bin/arm_CV -e ICC0 -i 182935

  medPap   375   327 252 00:59:37 ?         0:05 /med/p31100/comptel/medPap/amd/arm/bin/arm_CV -e ICC_ADJ1 -i 18

  medPap 29211 29177 253 00:59:19 ?         0:19 /med/p31100/comptel/medPap/amd/arm/bin/arm_CV -e ICC2 -i 182935

vfauh427[/home/yg09958]$

vfauh427[/home/yg09958]$ ps -eaf|grep oracleMED05P

 yg09958  4484 21624  0 01:00:24 pts/8     0:00 grep oracleMED05P

  medPap  4757     1 98  Oct  2  ?        1536:47 oracleMED05P (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

 medP3db 19745     1  0  Nov 15  ?         1:14 oracleMED05P (LOCAL=NO)

  medPap  4700     1  0  Oct  2  ?         7:10 oracleMED05P (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

 medP3db  3187     1  0 14:01:51 ?         0:38 oracleMED05P (LOCAL=NO)

 medP3db 19986     1 254 00:55:31 ?         1:36 oracleMED05P (LOCAL=NO)

 medP3db 26599     1 253 00:58:12 ?         0:42 oracleMED05P (LOCAL=NO)

 medP3db 24274     1  0 14:08:10 ?         0:29 oracleMED05P (LOCAL=NO)

 medP3db 23570     1  0 16:29:18 ?         0:04 oracleMED05P (LOCAL=NO)

  daemon  2278  1869 254 01:00:05 ?         0:03 oracleMED05P (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

 medP3db 18214     1 252 00:55:10 ?         1:41 oracleMED05P (LOCAL=NO)

 medP3db  1069     1 254 00:59:53 ?         0:07 oracleMED05P (LOCAL=NO)

 medP3db  6416     1  0  Nov 14  ?         1:19 oracleMED05P (LOCAL=NO)

vfauh427[/home/yg09958]$

[Dennis Handly]

>any alternative how can we reduce load on this system?

Why?  What kind of complaints are you getting?

Replace the system by more and faster CPUs/memory.  Port to an Integrity server.

Reduce the number of medPap's processes.

[Prakash Pitroda]

Get the high Cpu, Memory utilization processes & informed users accordinglly.

For list 5 Highest CPU utilized process.

 
UNIX95= ps -eo user,pid,pcpu,comm | sort -nrbk3 | head -5
 
For list 5 highest Memory utilized process.
 
UNIX95= ps -eo user,pid,vsz,comm | sort -nrbk3 | head -5

[Bill Hassell]

You have used all your CPUs and memory, and your disks are at their limit, based on the snapshot from Glance.

The simple answer is to stop running the medPap application. Then your resources will return to normal.

I assume that this is not an option. You need a lot more processors, and a lot more memory, and probably a faster set of disks. Your machine has reached its limit. I know that sysadmins are often asked to fix overloaded machines with some sort of "go-faster" button but without a major redesign of the application, or reducing the application load, you'll need a much bigger machine.