All the values are high in "sar -c 5 5" output on HPUX 11.23

[Senthil_N]

Hi All,

There is one HPUX servers. which became very slow suddenly. 

Output of Top command:

#top

System: dxp22hxa Fri Nov 22 17:34:25 2013
Load averages: 0.54, 0.61, 0.62
688 processes: 564 sleeping, 123 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.54 10.5% 0.0% 21.2% 68.3% 0.0% 0.0% 0.0% 0.0%
1 0.52 17.1% 0.2% 18.1% 64.7% 0.0% 0.0% 0.0% 0.0%
2 0.50 12.1% 0.0% 20.8% 67.1% 0.0% 0.0% 0.0% 0.0%
3 0.50 12.7% 0.0% 22.2% 65.1% 0.0% 0.0% 0.0% 0.0%
4 0.51 13.7% 0.0% 18.8% 67.5% 0.0% 0.0% 0.0% 0.0%
5 0.50 16.7% 0.0% 20.8% 62.5% 0.0% 0.0% 0.0% 0.0%
6 0.53 7.7% 0.0% 23.0% 69.2% 0.0% 0.0% 0.0% 0.0%
7 0.53 13.5% 0.0% 18.7% 67.9% 0.0% 0.0% 0.0% 0.0%
8 0.54 10.7% 0.0% 24.2% 65.1% 0.0% 0.0% 0.0% 0.0%
9 0.58 10.5% 0.0% 21.4% 68.1% 0.0% 0.0% 0.0% 0.0%
10 0.54 13.9% 0.0% 21.8% 64.3% 0.0% 0.0% 0.0% 0.0%
11 0.55 10.7% 0.0% 20.8% 68.5% 0.0% 0.0% 0.0% 0.0%
12 0.55 10.9% 0.0% 21.2% 67.9% 0.0% 0.0% 0.0% 0.0%
13 0.56 10.1% 0.0% 20.2% 69.6% 0.0% 0.0% 0.0% 0.0%
14 0.57 16.3% 0.0% 23.6% 60.1% 0.0% 0.0% 0.0% 0.0%
15 0.55 9.3% 0.0% 22.0% 68.7% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.54 12.1% 0.0% 21.2% 66.7% 0.0% 0.0% 0.0% 0.0%

Memory: 6079840K (5292156K) real, 13526352K (12452376K) virtual, 48713892K free Page# 1/41

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
6 ? 26271 root 152 20 21252K 6416K run 28:31 62.33 62.22 ldapclientd
2 ? 2528 root 152 20 4332K 2308K run 345:35 26.45 26.40 utmpd
4 ? 1132 oracle 152 20 1834M 345M run 5:32 25.17 25.13 java
5 ? 3602 root 238 20 2032K 344K run 136:13 20.99 20.95 syslogd
15 ? 27007 rtkbatch 158 20 2928K 456K sleep 2673:24 12.97 12.95 ksh
8 ? 15663 retek 158 20 2928K 432K sleep 108:05 12.14 12.12 ksh
1 ? 12068 rtkbatch 154 20 36684K 7324K sleep 3:18 10.68 10.67 bbypresexcpaip
6 ? 6647 root -16 20 101M 65996K run 1013:36 5.97 5.96 midaemon
1 ? 12957 mqm 152 20 384M 227M run 762:29 5.79 5.78 amqzlaa0
2 ? 14000 rtkbatch 154 20 84792K 16512K sleep 1:53 4.87 4.86 prmext
6 ? 18437 rtkbatch 154 20 84792K 16512K sleep 1:51 4.56 4.55 prmext
10 ? 15901 mqm 152 20 40096K 1996K run 174:10 4.55 4.54 amqpcsea
4 ? 12902 rtkbatch 154 20 84792K 16512K sleep 1:49 4.51 4.50 prmext
7 ? 17422 rtkbatch 154 20 84792K 16512K sleep 1:50 4.47 4.47 prmext
3 ? 15044 rtkbatch 154 20 84796K 16512K sleep 1:56 4.46 4.45 prmext
2 ? 16159 rtkbatch 154 20 84796K 16512K sleep 1:47 4.36 4.35 prmext

Output of "sar -c 5 5":

#sar -c 5 5

HP-UX dxp22hxa B.11.23 U 9000/800 11/22/13

17:36:16 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s
17:36:21 239354 20497 14592 310.18 309.98 351181728 2778699
17:36:26 227214 19289 14614 331.08 328.09 395601408 2495133
17:36:31 250815 19786 13199 396.79 383.17 404300032 2826158
17:36:36 248851 20993 14888 352.89 343.91 386840416 2543240
17:36:41 252975 22849 14387 292.38 286.77 335464256 2498437

My Questions:

1)Why the values in the output of "sar -c 5 5" is high, I verified on other HPUX servers, where it is normal?

2)How to dig down further and find the root cause?

[Dennis Handly]

26271 root 152 20 21252K 6416K run 28:31 62.33 62.22 ldapclientd
2528 root 152 20 4332K 2308K run 345:35 26.45 26.40 utmpd
3602 root 238 20 2032K 344K run 136:13 20.99 20.95 syslogd

Do you know why you have a lot of these?  It seems you are getting lots of logins.

[Senthil_N]

Hi Dennis,

The server slowness issue got resolved after restarting syslogd. Now the server performance is fine. But still I am seeing following output. Please let me know why all the values are high.

#sar -c 5 5
HP-UX dxp22hxa B.11.23 U 9000/800 11/22/13
17:36:16 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s
17:36:21 239354 20497 14592 310.18 309.98 351181728 2778699
17:36:26 227214 19289 14614 331.08 328.09 395601408 2495133
17:36:31 250815 19786 13199 396.79 383.17 404300032 2826158
17:36:36 248851 20993 14888 352.89 343.91 386840416 2543240
17:36:41 252975 22849 14387 292.38 286.77 335464256 2498437

[Dennis Handly]

>Please let me know why all the values are high.

You need to use a tool like glance to find out which process is doing all of those syscalls.

And if your CPU or I/O isn't maxed out, why are you worried?