Sluggish System

[Charles Egbon]

Hello Guys,

Our users are complaing of sluggish system. Here is a snapshot of system resource utilisation.
System: SCRBORA1 Fri Aug 22 16:02:09 2008
Load averages: 0.31, 0.23, 0.20
277 processes: 258 sleeping, 18 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.32 41.3% 0.0% 2.2% 56.5% 0.0% 0.0% 0.0% 0.0%
1 0.30 35.1% 0.0% 3.8% 61.1% 0.0% 0.0% 0.0% 0.0%
--- ---- - - - - - - - -----
avg 0.31 38.2% 0.0% 3.0% 58.8% 0.0% 0.0% 0.0% 0.0%

Memory: 2385272K (890224K) real, 2765540K (1144152K) virtual, 4022812K free Page# 1/26

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 13392 oracle 192 20 1524M 7456K run 168:53 15.04 15.01 oracleEXPRESS
1 ? 15406 oracle 148 20 1538M 15968K sleep 0:19 12.77 12.75 oracleEXPRESS
0 ? 26639 oracle 154 20 1522M 7072K sleep 0:02 1.11 1.10 oracleEXPRESS
1 ? 95729028le 156 20 41524M 82M 7200ep 0:00 3.1 1.03 1.03 oracleEXPRESS
1 ? 10578897 oracle 154 20 1534M 19424K sleep 0:53 0.82 0.82 oracleEXPRESS
0 ? 3076 root -27 20 17800K 13724K run 2:12 0.71 0.71 cmcld
1 pts/ta 13390 express 152 20 258M 72644K run 4:29 0.62 0.62 java
1 ? 13403 oracle 154 20 1524M 8032K sleep 9:25 0.55 0.55 oracleEXPRESS
0 ? 37 root 152 20 7424K 7424K run 0:56 0.53 0.53 vxfsd
1 ? 14159 oracle 154 20 1524M 8096K sleep 0:08 0.44 0.44 oracleEXPRESS

What could be the problem?

Charles.

[Pete Randall]

Charles,

Try running "vmstat 1 5" and take a look at the "po" numbers. Zeroes are great, single digits are good, double digits are bad. That would indicate you're swapping and you need to free some memory or provide some more.


Pete

[Tim Nelson]

"sluggish" in my mind, just like Pete's, is either 100% CPU or a swap issue. Your current snap with top rules out CPU (note the word current).


Also post swapinfo -tam with the vmstat .

Next idea would be IO.

If all of those are great then possible application issue.

Last but not least. sluggish from remote but not local ? Then WAN issue.

[Charles Egbon]

Thanks guys fpr your generous gesture. I will reciprocate your effort. Meanwhile:

#swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
dev 6000 0 6000 0% 0 - 1 /dev/vg00/lvsecswap
reserve - 2190 -2190
memory 6305 1108 5197 18%
total 16401 3298 13103 20% - 0 -



#vmstat 1 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy
id
1 0 0 209334 1049494 211 19 0 0 0 0 36 1004 40814 1081 14 3
83
1 0 0 209334 1049438 63 7 0 0 0 0 0 475 3503 280 0 0
100
1 0 0 209334 1049438 56 5 0 0 0 0 0 484 3719 285 7 1
92
1 0 0 209334 1049439 45 4 0 0 0 0 0 505 3264 281 0 0
100
1 0 0 209334 1049438 36 3 0 0 0 0 0 536 3555 292 10 0
90

[Tim Nelson]

With thost snapshots I see no memory pressure.

On to the next..

Disk ?

sar -d 5 5

[Charles Egbon]

#sar -d 5 5

HP-UX HOSTNAME01 B.11.11 U 9000/800 08/25/08

10:50:09 device %busy avque r+w/s blks/s avwait avserv
10:50:14 c2t1d0 3.79 0.50 8 54 0.00 7.04
c2t0d0 2.59 0.50 6 47 0.00 5.25
c12t1d0 0.20 0.50 13 117 0.00 0.17
c17t1d0 0.40 0.50 13 116 0.00 0.21
c6t0d5 0.20 0.50 4 73 0.00 0.26
c15t0d5 0.20 0.50 3 63 0.00 0.30
c8t0d6 0.40 0.50 5 67 0.00 0.72
c12t0d6 0.20 0.50 0 0 0.00 9.87
c10t0d6 0.20 0.50 0 0 0.00 10.11
c6t0d6 0.20 0.50 0 0 0.00 10.06
c17t0d6 0.20 0.50 4 56 0.00 0.24
10:50:19 c2t1d0 5.01 0.50 10 83 0.00 8.54
c2t0d0 3.21 0.50 7 72 0.00 5.42
c8t0d2 0.20 0.50 0 6 0.00 3.20
c6t0d5 0.60 0.50 2 33 0.00 3.07
c8t0d6 0.40 0.50 3 28 0.00 1.16
10:50:24 c2t1d0 8.40 0.50 20 139 0.00 7.20
c2t0d0 6.00 0.50 16 126 0.00 4.53
c12t1d0 0.20 0.50 15 86 0.00 0.17
c17t1d0 0.40 0.50 13 82 0.00 0.20
c6t1d2 0.40 0.50 2 230 0.00 4.74
c6t0d1 0.40 0.50 5 62 0.00 0.89
c8t0d6 0.20 0.50 3 42 0.00 0.55
c12t0d6 0.20 0.50 0 0 0.00 8.89
c10t0d6 0.20 0.50 0 0 0.00 9.25
c6t0d6 0.20 0.50 0 0 0.00 9.05
c15t0d6 0.20 0.50 0 0 0.00 0.46
10:50:29 c2t1d0 5.20 0.50 12 93 0.00 5.30
c2t0d0 3.20 0.50 10 83 0.00 3.54
c12t1d0 0.20 0.50 11 51 0.00 0.16
c6t1d2 0.20 0.50 2 231 0.00 4.23
c6t0d1 0.20 0.50 3 49 0.00 0.28
c15t0d1 0.20 0.50 2 45 0.00 0.32
c8t0d2 18.20 0.50 615 9844 0.00 0.30
c15t0d5 0.20 0.50 2 35 0.00 0.32
c6t0d6 0.20 0.50 0 0 0.00 1.49
c17t0d6 0.20 0.50 3 28 0.00 0.23
10:50:34 c2t1d0 7.80 0.50 15 94 0.00 8.30
c2t0d0 4.60 0.50 13 84 0.00 5.27
c12t1d0 0.20 0.50 2 22 0.00 0.25
c17t1d0 0.20 0.50 2 22 0.00 0.23
c6t0d1 0.60 0.50 7 89 0.00 1.04
c8t0d2 81.60 0.50 2753 44042 0.00 0.30
c21t0d2 0.80 0.50 1 19 0.00 6.42
c6t0d5 0.20 0.50 2 39 0.00 0.88
c8t0d6 0.60 0.50 3 40 0.00 2.38
c12t0d6 0.20 0.50 0 0 0.00 6.34
c10t0d6 0.20 0.50 0 0 0.00 6.57

Average c2t1d0 6.04 0.50 13 93 0.00 7.28
Average c2t0d0 3.92 0.50 10 82 0.00 4.73
Average c12t1d0 0.16 0.50 8 55 0.00 0.17
Average c17t1d0 0.20 0.50 6 44 0.00 0.20
Average c6t0d5 0.20 0.50 1 29 0.00 1.09
Average c15t0d5 0.08 0.50 1 20 0.00 0.31
Average c8t0d6 0.32 0.50 3 36 0.00 1.11
Average c12t0d6 0.12 0.50 0 0 0.00 8.37
Average c10t0d6 0.12 0.50 0 0 0.00 8.65
Average c6t0d6 0.12 0.50 0 0 0.00 6.87
Average c17t0d6 0.08 0.50 1 17 0.00 0.24
Average c8t0d2 20.00 0.50 674 10779 0.00 0.30
Average c6t1d2 0.12 0.50 1 92 0.00 4.45
Average c6t0d1 0.24 0.50 3 40 0.00 0.86
Average c15t0d6 0.04 0.50 0 0 0.00 0.46
Average c15t0d1 0.04 0.50 0 9 0.00 0.32
Average c21t0d2 0.16 0.50 0 4 0.00 6.42

[Prashanth Waugh]

Hi

Check
#ipcs -ma

Regards
Atul

[Rasheed Tamton]

Hi,

There is no disk bottle neck either as per your sar -d report. avwait is always 0.0 and avque is also normal.

You need to monitor continuously and use metric to evaluate the performance.

rgds.

[Charles Egbon]

#ipcs -ma
IPC status from /dev/kmem as of Mon Aug 25 13:54:56 2008
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID
LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x411c29ca --rw-rw-rw- root root root root 0 348 674
674 15:02:47 15:02:47 15:02:41
m 1 0x4e0c0002 --rw-rw-rw- root root root root 1 61760 674
674 15:02:43 15:02:47 15:02:41
m 2 0x4120161e --rw-rw-rw- root root root root 1 8192 674
686 15:02:43 15:02:41 15:02:41
m 6147 0x0c6629c9 --rw-r----- root root root root 2 18994744 1652
1638 15:03:25 no-entry 15:03:25
m 4 0x06347849 --rw-rw-rw- root root root root 1 65626 1652
1728 15:03:29 15:03:25 15:03:25
m 517 0x4918003a --rw-r--r-- root root root root 0 22908 1638
1728 13:54:00 13:54:00 15:03:26
m 10758 0x5e1007ab --rw------- root root root root 1 512 2345
2345 15:03:43 no-entry 15:03:43
m 88071 0x642392bc --rw-r----- oracle dba oracle dba 71 1431584768 12839
773 13:54:55 13:54:55 19:46:14

[Steven E. Protter]

Shalom Charles,

1 zombie

Always a concern. Zombie processes can tie up resources.

http://www.hpux.ws/?p=6
A toolset intended to make performance management easier.

Because you are running Oracle and java, I would say they are the most logical suspects.

Look for memory leaks
http://www.hpux.ws/?p=8

SEP

[Dennis Handly]

>SEP: 1 zombie. Zombie processes can tie up resources.

They only take up a process table slot.

[Charles Egbon]

What can I do to the zombie process?

[Tim Nelson]

Reboot is the only option to rid of zombies.


All of the information you have provided shows that your system is performing optimally. CPU, MEM and DISK all look ok.

So, one of a couple more things:

1) the stats provided have NOT been gathered during the "sluggishness"

2) Your issue lies with a network problem

3) Your issue lies with an application issue.

Is this experience constant ?
During certain times of the day ?
Intermittant ?
Experienced by all users, local and remote ?
Experienced at both the OS level and application level ?

Continue to gather stats on a regular basis, make note of when users complain, use those stats during that time to identify if a resource issue or not. If not then push it back to either application group or network group.

[Dennis Handly]

>What can I do to the zombie process?

Why are you worrying about it?
First you have to find out which it is. (There are some hardware monitoring processes that always has a zombie due to a known bug.)

Read my reply about wanted posters: :-)
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1145648

[Dennis Handly]

>Tim: Reboot is the only option to rid of zombies.

This isn't correct. Read my URL above about zombie masters.

[Patrick Wallek]

>>Our users are complaing of sluggish system.

The information you have provided above all looks pretty good.

The key here is what do the users mean by "sluggish?" Is it taking a long time to run some SQL commands? Is it taking a long time to return data from the DB?

I think we need a better definition of "sluggish" before we go much further.

If they are complaining about Oracle performance then you may want to check and see what SQL they are running. Nothing will make getting data slow like badly written SQL.

There could also be issues with your Oracle SGA. How much RAM do you have? How large is the SGA?

You may need to get your DBA's involved and have them look at the performance of the DB itself.