%sys in sar is very high

[Evelyn Daroga]

Ever since rebooting the system a couple weeks ago, the %sys in sar runs very high during business hours (it appears normal after hours). Any answer other than we're just overloading our system? I've noticed some sys processes (psmctd, pwgrd, midaemon) showing up as among the higher cpu users --is that normal? I've stopped/started the psmctd and pwgrd processes, but that doesn't seem to help. Disk I/O (glance) has at times been 100%, but not during the time I took these stats. People are complaining -- any suggestions would be appreciated!

OUTPUT OF SAR IS:
sar 5 20
HP-UX visib B.11.00 U 9000/800 01/10/08
09:23:02 %usr %sys %wio %idle
09:23:07 27 72 0 0
09:23:12 39 60 1 0
09:23:17 35 65 0 0
09:23:22 32 67 0 0
09:23:27 32 67 1 0
09:23:32 43 55 1 0
09:23:37 43 56 0 0
09:23:42 26 74 0 0
09:23:47 41 59 0 0
09:23:52 35 65 0 0
09:23:57 34 66 0 0
09:24:02 35 65 0 0
09:24:07 36 64 0 0
09:24:12 40 60 0 0
09:24:17 51 48 0 0
09:24:22 48 52 0 0
09:24:27 38 61 0 0
09:24:32 31 69 0 0
09:24:37 52 48 0 0
09:24:42 42 58 0 0

Average 38 62 0 0

TOP CPU USERS:
UNIX95= ps -ef -o "pcpu pid user ruser stime time args" | sort -rn | head -10
22.11 15590 lp lp 08:51:44 08:05 quiz auto=/fh_home/jervis/v63yoln/quiz/pa110rrjw.qzs NOLIST
16.99 5887 dmsarka dmsarka 07:47:46 01:16 quick subdict=search auto=/fh_home/jervis/v63yoln/MENUGO.qkg
9.53 18764 clstale clstale 07:54:45 01:38 quick subdict=search auto=/fh_home/jervis/v63yoln/MENUGO.qkg
9.36 17340 oracle oracle 09:21:11 00:15 oracleWEBB (LOCAL=NO)
7.82 19647 root root Jan 4 05:10:10 psmctd
7.60 19150 hikostr hikostr 09:23:30 00:01 quiz auto=/canada_home/jervis/v63yoln/quiz/pa130bld5.qzs NOL
6.37 26983 ccparke ccparke 07:18:29 03:14 quick subdict=search auto=/fh_home/jervis/v63yoln/MENUGO.qkg
6.36 14094 dmmcdon dmmcdon 08:26:35 00:19 quick subdict=search auto=/fh_home/jervis/v63yoln/MENUGO.qkg
6.04 17307 lp lp 09:21:08 00:07 qtp cc=(JERVIS,UNIX,US,ORACLE,LSTRANS) subdict=search auto=/
5.51 29190 root root Jan 4 01:58:06 /usr/sbin/pwgrd

And again:
UNIX95= ps -ef -o "pcpu pid user ruser stime time args" | sort -rn | head -10
15.68 18764 clstale clstale 07:54:45 01:43 quick subdict=search auto=/fh_home/jervis/v63yoln/MENUGO.qkg
14.91 5887 dmsarka dmsarka 07:47:46 01:20 quick subdict=search auto=/fh_home/jervis/v63yoln/MENUGO.qkg
12.94 14094 dmmcdon dmmcdon 08:26:35 00:24 quick subdict=search auto=/fh_home/jervis/v63yoln/MENUGO.qkg
12.84 15590 lp lp 08:51:44 08:07 quiz auto=/fh_home/jervis/v63yoln/quiz/pa110rrjw.qzs NOLIST
11.92 18592 oracle oracle 09:23:04 00:05 oracleWEBB (LOCAL=NO)
11.50 19150 hikostr hikostr 09:23:30 00:06 quiz auto=/canada_home/jervis/v63yoln/quiz/pa130bld5.qzs NOL
10.47 19265 oracle hikostr 09:23:32 00:04 oracleCAN (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
9.45 17340 oracle oracle 09:21:11 00:18 oracleWEBB (LOCAL=NO)
7.42 19647 root root Jan 4 05:10:13 psmctd
6.25 26983 ccparke ccparke 07:18:29 03:17 quick subdict=search auto=/fh_home/jervis/v63yoln/MENUGO.qkg

[Steven E. Protter]

Shalom,

Performance monitor
http://www.hpux.ws/?p=6

Memory leak detector:
http://www.hpux.ws/?p=8

Looks like Oracle and/or autofs is using a lot of resources.

There may be a lot of writes going, creating i/o which the system handles.

You need more data to determine the source of the problem.

SEP

[Rita C Workman]

You probably shouldn't shutdown the psmctd, cause that daemon passes information to another monitoring daemon (psmmon) and then you starting getting other errors. Could even affect MC/SG.

If your midaemon is running high and staying there, then you have a something running that is choking the box.
The fact that your disk is hitting 100%, and I'm guessing by your concern, more often than it rightly should also indicates some process that has run amuck.

It would be too hard to answer based on just this info. Could be tuning of parms is in order; could be poorly written syntax on some job or querry; and so on.

Start digging around. See if your DBA can do some checks on the high hitter processing to narrow something down from the Oracle side. If you have some utilities that can grab some packettes of your high users (like "tusc") that you could look down, you might be able to find something.
Depending on your O/S version you might have other utilities you can use, like pstack....

Just a couple thoughts,
Rgrds,
Rita

[Laurent Menase]

Hi
First remark, sar cpu stats doesn't give precise data as it is a tick time sample.
, prefer "glance" to have accurate data.

[Evelyn Daroga]

Thanks for the feedback Steven, Rita and Laurent.
I have run the scripts provided by Steven -- they didn't reveal anything real obvious. No apparent memory leaks, although mem usage is very high. Thanks, Steve, I'll keep them around for future use.

As for disk i/o, it is not unusual for that to be very high â even 100%. It has actually been lower throughout the morning, than it generally is. What IS unusual is for the SYSTEM CPU usage to be so high. I have been watching closely via sar, top, and glance â all indicate the CPU max'd out (which, also, has not been that unusual on this system), and all indicate the SYSTEM using lots of CPU â this is the part that is unusual. If the CPU is max'd out, then so be it. My questions is not "why is cpu usage so high", but rather "why is the SYSTEM cpu usage so high?"

If the users are running more reports than usual, then that would increase the I/O. I understand that I/O requests are "system" requests, and thus add to the system's cpu requirements.

Memory is also an issue on this system â has peaked at 100% more than I like. This has caused some paging -- again, increasing the system's cpu requirements.

If these are viable explanations, then ok. I can understand that the system's cpu usage might go up a bit, but it seems excessive to me. I guess that's where my dilemma is.

Thanks for all the input â I appreciate it!

[Laurent Menase]

use glance and look at the time passed in the different syscalls,

Forget sar if you want to make an accurate analyse on CPU usage.

[Rita C Workman]

Evelyn,

What is your dbc_max% & dbc_min% ?
Can you give us copy of your kernel parms?
Can you give us a copy of your swapinfo -tam?
Can you run for me sar -v 1 20 (my favorite sar command)....?
What is total physical memory on this box?
How many CPU's?
I see your at 11.0 - so we know the version level.

Let's just look down a couple basic parms and see if there is anything that will help.

Also - Oracle - what disks exactly hit 100% utilization? What is hitting those disks? Your disk contention may be resolved by moving things around. You may have too much hitting one disk...like all your oracle logging going to the same disk for example.

Hopefully someone might see something that might help.
Rgrds,
Rita

[Evelyn Daroga]

Thanks, again, for the reply Rita.
I have attached a txt document with the information you requested. I appreciate your help!

[Evelyn Daroga]

Thanks also for your reply, Laruent. I have been looking at glance along with sar. The items that jump out with the highest cpu usage are: OPEN, STAT, and LSTAT64.

[Evelyn Daroga]

Thanks also for your reply, Laruent. I have been looking at glance along with sar. The system calls that jump out with the highest cpu usage are: OPEN, STAT, and LSTAT64.