Problem Performance

[Jairo Campana]

Hello , I have a problem of performance , that I can do

Load averages: 6.06, 6.95, 6.14
144 processes: 119 sleeping, 25 running
Cpu states:
LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
6.06 50.7% 0.0% 48.7% 0.6% 0.0% 0.0% 0.0% 0.0%

Memory: 145932K (60680K) real, 241096K (144504K) virtual, 21512K free Page# 1/6

that they make the processes
statdaemon , swapper, vhand, syncer
UID PID PPID C STIME TTY TIME COMMAND
root 0 0 0 Jun 18 ? 419:40 swapper
root 1 0 0 Jun 18 ? 39:34 init
root 2 0 0 Jun 18 ? 389:59 vhand
root 3 0 0 Jun 18 ? 258:21 statdaemon
root 4 0 0 Jun 18 ? 3:53 unhashdaemon
root 8 0 0 Jun 18 ? 0:00 supsched
root 9 0 0 Jun 18 ? 0:00 strmem
root 10 0 0 Jun 18 ? 0:00 strweld
root 11 0 0 Jun 18 ? 0:00 strfreebd
root 12 0 0 Jun 18 ? 0:49 ttisr
root 18 0 0 Jun 18 ? 20:47 lvmkd
root 19 0 0 Jun 18 ? 23:00 lvmkd
root 20 0 0 Jun 18 ? 20:54 lvmkd
root 21 0 0 Jun 18 ? 23:04 lvmkd
root 22 0 0 Jun 18 ? 21:38 lvmkd
root 23 0 0 Jun 18 ? 19:40 lvmkd
root 24 0 0 Jun 18 ? 0:00 lvmschedd
root 9424 1 0 Oct 26 console 0:00 /usr/sbin/getty console console
root 394 1 0 Jun 18 ? 281:32 /usr/sbin/syncer


# sar -u 3 5

HP-UX proc B.11.00 U 9000/800 11/09/04

12:29:09 %usr %sys %wio %idle
12:29:12 61 33 7 0
12:29:15 63 33 4 0
12:29:18 57 38 5 0
12:29:21 67 33 0 0
12:29:24 45 55 0 0

Average 59 38 3 0


# sar -q 3 5

HP-UX B.11.00 U 9000/800 11/09/04

12:30:10 runq-sz %runocc swpq-sz %swpocc
12:30:13 3.7 100 0.0 0
12:30:16 5.3 100 0.0 0
12:30:19 3.0 100 0.0 0
12:30:22 5.0 100 0.0 0
12:30:25 7.0 100 0.0 0

Average 4.8 100 0.0

# sar -v 3 5
HP-UX B.11.00 U 9000/800 11/09/04

12:30:44 text-sz ov proc-sz ov inod-sz ov file-sz ov
12:30:47 N/A N/A 144/700 0 1900/1900 0 1435/3010 0
12:30:50 N/A N/A 146/700 0 1900/1900 0 1441/3010 0
12:30:53 N/A N/A 144/700 0 1889/1900 0 1438/3010 0
12:30:56 N/A N/A 144/700 0 1873/1900 0 1438/3010 0
12:30:59 N/A N/A 144/700 0 1841/1900 0 1438/3010 0


iostat 3 5

device bps sps msps

c1t2d0 0 0.0 1.0
c2t2d0 0 0.0 1.0

c1t2d0 4618 134.2 1.0
c2t2d0 5958 160.4 1.0

c1t2d0 4862 216.3 1.0
c2t2d0 4678 224.3 1.0

c1t2d0 3232 170.5 1.0
c2t2d0 3494 191.1 1.0

c1t2d0 4468 200.0 1.0
c2t2d0 4961 241.2 1.0

thanks ,

[Alzhy]

From the looks of it you've a really loaded system, your memory is nearly exhausted. And you're CPU is heavily loaded.

What are you running on this server?

[Steven E. Protter]

swapinfo -tam

vmstat 1 30

Are we paging.

More advanced perf data collection attached.

If you are paging into swap, its memory buy time.

The good news is you don't seem to be waiting on disk io. That may be because there is too much stuff running on the box to have time to do a lot of disk io

Answering Nelson's question can give us all information that can be useful in giving you further assistance.

SEP

[Geoff Wild]

For paging:

vmstat -S 5 20

Look at SI and SO columns...

Rule of thumb for performance - Check Memory first, then disk, then CPU...

Your disks and CPU may be loaded because of memory pressure - fix the memory - and your CPU and disks may be fine.

Decrease the size of your buffer cache may help - what is it set to? DBC_MAX_PCT and DBC_MIN_PCT...

More memory may be your solution...

Rgds...Geoff

[Jairo Campana]

model
9000/800/L1000-36
in server run:
Openview
oracle
NNM
ISEE/EMS/STM
NETWORKER (installed today)

before of the problem performance install agent of networker

swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 51 973 5% 0 - 1 /dev/vg00/lvol2
dev 1024 51 973 5% 0 - 1 /dev/vg00/lvswap
reserve - 327 -327
total 2048 429 1619 21% - 0 -

nms1 # vmstat 1 30
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 8 0 32872 4232 72 25 0 0 26 0 0 166 47259 153 48 15 37
2 7 0 33161 4181 4 0 1 0 0 0 0 1250 65555 392 71 17 12
2 7 0 33161 4181 3 0 1 0 0 0 0 1256 65309 388 77 13 10
2 7 0 33161 4181 2 0 1 0 0 0 0 1272 64560 393 67 18 14
2 7 0 33161 4181 1 0 1 0 0 0 0 1271 64693 387 71 17 12
2 7 0 33161 4257 0 0 1 0 0 0 0 1300 62334 444 63 26 11
2 3 0 37402 4222 2 0 2 0 0 0 0 1145 60629 540 73 18 8
2 3 0 37402 4184 3 0 2 0 0 0 0 1199 58770 553 52 33 15
2 3 0 37402 4049 18 5 2 0 4 0 0 1182 57702 544 52 28 20
2 3 0 37402 4253 14 4 2 0 4 0 0 1052 54742 674 54 27 19
2 3 0 37402 4099 39 16 2 0 15 0 0 941 52238 681 54 32 13
4 4 0 38277 4124 149 53 2 0 53 0 0 908 48086 738 45 43 12
4 4 0 38277 4200 161 57 2 0 57 0 0 988 48216 681 57 28 15
4 4 0 38277 4200 171 60 1 0 60 0 0 1006 49505 613 55 34 11
4 4 0 38277 4343 182 64 0 0 69 0 0 1041 49860 546 50 39 11
4 4 0 38277 4326 219 75 0 0 78 0 0 1030 48741 518 41 47 12
4 1 0 39875 4200 224 79 0 0 77 0 0 1102 49238 468 46 41 13
4 1 0 39875 4167 233 85 0 0 83 0 0 1147 50192 442 18 70 12
4 1 0 39875 4200 215 76 0 0 74 0 0 1138 50040 425 69 22 9
4 1 0 39875 4343 175 62 0 0 65 0 0 1181 53857 410 78 14 9
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
4 1 0 39875 4343 140 49 0 0 52 0 0 1205 57211 408 25 62 13
14 4 0 41354 4091 127 47 0 0 44 0 0 1201 54680 389 39 52 9
14 4 0 41354 4200 131 45 0 0 43 0 0 1204 52768 348 63 26 11
14 4 0 41354 4200 104 36 0 0 34 0 0 1305 56389 338 66 23 11
14 4 0 41354 4200 84 28 0 0 27 0 0 1309 59802 344 75 15 10
14 4 0 41354 4200 67 22 0 0 21 0 0 1304 62649 357 78 13 9
1 7 0 35837 4200 54 17 0 0 16 0 0 1295 64949 351 59 28 13
1 7 0 35837 4200 43 13 0 0 12 0 0 1286 66900 360 65 13 21
1 7 0 35837 4200 27 8 0 0 7 0 0 1180 66447 354 54 37 9
1 7 0 35837 4220 54 15 0 0 19 0 0 1212 62756 336 52 39 9

[Sridhar Bhaskarla]

Hi Jairo,

Openview tools can be both memory and CPU extensive. I suggest you buy little bit more memory and some CPUs. Try installing latest Openview/NNM patches to see if they make any difference. More nodes/CIs you add and more maps you add/create, more will be memory and CPU usages.

This time we can't really ask you to go back to your developers. Can we? ;-). Openview is HP's tool. Even before installing Networker, you system looked busy. Stop running some unnecessary processes such as sendmail, prm, httpd etc., if you don't need them.

-Sri