topic Re: Possible CPU Bottleneck in Operating System - HP-UX

Possible CPU Bottleneck

Charles McCary — Wed, 17 Apr 2002 14:22:05 GMT

Group,

Hi - here's the scenario:

We have a vendor product running on an HP N-class with 8 processors. When their process runs, it spawns parallel processes (8) and it basically takes up 100% of all processors during the time of it's running.

The runq is occupied about 85% of the time and is always aroun 1.5-2.5 in length.

Of course, the h/w they recommended was much LESS than the N-class and was supposed to handle the processing it needs to do in less time.

It is now running way too long and I want to make sure that I haven't missed something critical to help speed this up from an admin perspective. Can you guys think of anything I should check? I'll supply more info. if you need it to offer an opinion.

tx,
c

Re: Possible CPU Bottleneck

Victor BERRIDGE — Wed, 17 Apr 2002 14:31:48 GMT

Hi,
Difficult to give an advice like that...
If you have glance then use it to see whats happening, if you dont have it - Get an trial version (a time limited I think 60 days).
What is the product supposed to do?

All the best
Victor

Re: Possible CPU Bottleneck

Helen French — Wed, 17 Apr 2002 14:33:29 GMT

Hi Charles:

Some points to check:

1) Kernel parameters. Check all parameters and increase/reduce if needed. Reduce the dbc_max_pct value, if it is set to 50 ( default ). You can try 15 or 20.
2) System patch level.
3) Check system usage with GlancePlus and find out if anything wrong.

HTH,
Shiju

Re: Possible CPU Bottleneck

Paula J Frazer-Campbell — Wed, 17 Apr 2002 14:36:03 GMT

Hi
Do you have glance installed ?

If not there is a 60 day trial version on the apps cd.

Glance will allow you to track what is going on.
If it is slow running then check memory/swap usage and also disk activity by disk.

Sar is also a useful tool see man sar.

HTH

Paula

Re: Possible CPU Bottleneck

Trond Haugen — Wed, 17 Apr 2002 14:42:20 GMT

what is the HP-UX version? I know there is a bug in 11i showing a large load average on even a idle system. It's fixed with PHKL_24551.
Are the users reporting bad performance? Is most CPU time spendt in user or system mode?

Regards,
Trond

Re: Possible CPU Bottleneck

S.K. Chan — Wed, 17 Apr 2002 14:53:38 GMT

You need glance to do this. At the surface you can quickly check if you CPU is used more by the "system" or "user". If more of the CPU time is used by the user (ie outside the kernel), then it's generally a good thing, still you have to determine that it is running the correct user program.
Run glance and at the main process list window, enter SHIFT-?, that gives you a list of things you want to look at. Enter "c" for CPU report.
Now in that window, look at "User","System" and "Idle", or better still post the snapshot of this glance, so we could look at it. What you want to see is the "user" utilization should be higher than that of "system" which simply could mean the CPU is well utilized. If you got more % of CPU used by "system", we'll have to drill further.

Re: Possible CPU Bottleneck

Charles McCary — Wed, 17 Apr 2002 16:23:24 GMT

Group,

Hi - I do have glance and the system utilization is way under user utilization:

Not sure how this is going to post, but here's a snapshot:
State Current Average High Time Cum Time
--------------------------------------------------------------------------------
User 86.8 82.3 88.9 6.96 6011.97
Nice 0.0 0.0 1.6 0.00 2.98
Negative Nice 0.1 0.1 1.3 0.01 5.44
RealTime 1.0 1.0 1.6 0.08 69.86
System 9.7 9.2 16.2 0.78 672.97
Interrupt 0.9 0.6 3.1 0.07 40.21
ContextSwitch 1.0 0.9 1.3 0.08 68.13
Traps 0.1 0.2 0.4 0.01 11.95
Vfaults 0.0 0.1 4.0 0.00 5.44
Idle 0.4 5.7 98.7 0.03 417.24

The main problem is that they're program is running much longer than they promised, there's not really a performance lag during the time that it's running, or at least the user's don't seem to complain.

tx,

Charlie

Re: Possible CPU Bottleneck

Charles McCary — Wed, 17 Apr 2002 16:26:28 GMT

Also, running HPUX 11.0

Re: Possible CPU Bottleneck

Bill McNAMARA_1 — Wed, 17 Apr 2002 16:30:43 GMT

I would say yes, but, you are buying the CPU to use 100% of it aren't you?

Although it's from the hp java site, this analysis flow is probably what'll get to the nitty gritty of it:

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1618,00.html

Later,
Bill

Re: Possible CPU Bottleneck

Jeff Schussele — Wed, 17 Apr 2002 16:43:53 GMT

Hi Charles,

Please note that patch PHKL_24551 (11i only) has been superseded by PHKL_25389(11i only) - see:

http://us-support.external.hp.com/wpsl/bin/doc.pl/screen=wpslDisplayPatch/sid=46a2f2481b40fd883f?PACH_NAM=PHKL_25389&HW=s800&OS=11.11

Although looking at your glance output I would:
1) Question whether the code may be running away
2) Question what kernel parameter settings and/or patches the developer need or recommend.

because it's almost all wrapped up in user time - >10 % in system. You might want to start looking deeper at what's happening by using the ipcs command or the lsof utility. And don't discount the possibility af corrupted or even "crap" code or libraries.

Rgds,
Jeff

Re: Possible CPU Bottleneck

Sandip Ghosh — Wed, 17 Apr 2002 16:52:47 GMT

Can you provide some more data about the system?
Amount of total memory
swapinfo -tm output
Kernel Parameters
sar -b 5 10
top 5 process from top

Sandip

Re: Possible CPU Bottleneck

Charles McCary — Wed, 17 Apr 2002 16:58:34 GMT

Re: Possible CPU Bottleneck

Charles McCary — Wed, 17 Apr 2002 17:02:06 GMT

Also, here's the glance syscall info for one of the processes:

System Calls PID: 24642, accumat2 PPID: 24628 euid: 501 User: mibas
Elapsed Elapsed
System Call Name ID Count Rate Time Cum Ct CumRate CumTime
--------------------------------------------------------------------------------
read 3 2064 430.0 0.22196 1150407 696.8 208.19893
write 4 370 77.0 0.00641 251975 152.6 5.04829
open 5 1693 352.7 0.07188 907305 549.6 28.53180
close 6 1693 352.7 0.02098 907304 549.6 12.71636
time 13 1693 352.7 0.00572 907305 549.6 3.51548
lseek 19 1693 352.7 0.01586 908630 550.4 6.13265
stat 38 1694 352.9 0.03282 907305 549.6 18.53448
ioctl 54 1694 352.9 0.02062 907305 549.6 7.81397
ulimit 63 28506 5938.7 0.13937 14394366 8719.6 55.12364
gettimeofday 116 1120 233.3 0.00691 548530 332.2 5.48679

Re: Possible CPU Bottleneck

Hai Nguyen_1 — Wed, 17 Apr 2002 18:07:02 GMT

Charles,

I think now it is time to turn to the vendor's application. Does it have any tunable configuration files which, for example, specify how often a program runs, how many processes should be spawn...

Also, a good application should generate some meaningful log(s) which you can look into to find out what it is doing...

That's my 2 cents.

Hai

Re: Possible CPU Bottleneck

Charles McCary — Wed, 17 Apr 2002 18:09:53 GMT

Hai,

thanks, yes as an aside, this vendors product can run in several different configurations. It can run with 1 process all the way up to 64 processes. We have run with many different configurations and finally decided that 16 was giving us the best performance. I'm beginning to suspect you are correct though, I don't see any obvious system problems (and no one else does either), so it is time for the vendor to speed up their program.

thanks,

Charlie

Re: Possible CPU Bottleneck

Charles McCary — Wed, 17 Apr 2002 18:14:04 GMT

Someone above requested the following info:

8 550 mhz processors
4 gb memory

Re: Possible CPU Bottleneck

Jeff Schussele — Wed, 17 Apr 2002 18:29:22 GMT

Hi Charles,

Well, nothing stands out to me on your tunables & overall setup.
You're using some swap space but I suspect that it's being reserved by all the instances of accumat2. And my, does that app know how to really "take over" a system?
I would hesitate to run anything else on this system w/o throttling it back some - either less processes or higher nice values.

So I guess my final response would be - unless it's response times start tanking OR you run out of memory or swap - let it rip. Unused CPU cycles are just that - unused. You paid for them - might as well use them.

Rgds,
Jeff

Re: Possible CPU Bottleneck

Charles McCary — Wed, 17 Apr 2002 18:33:06 GMT

Jeff,

yeah - the only problem with letting them rip is that their product is taking too long to complete. When you factor in the runtime of their product, then another application that runs after it, then backups, we have basically started running out of hours in the day. I was hoping someone would look at this stuff and say "whoa, you really missed the boat on that one", tell me something obvious to change and make the runtime of the app go from 9 hours to 6 hours. Oh well, wishful thinking I guess.

thanks,

Charlie

Re: Possible CPU Bottleneck

Sandip Ghosh — Wed, 17 Apr 2002 18:42:55 GMT

Hi ,

Everything looks pretty normal, I mean to say the configuration. It is not swapping, the %rcache for the system is 100% (too good), there is no runaway process. It's now really difficult to comment.

Can you just once change the bufpages and nbuf to 0 and see the performance? It seems the nbuf parameter is set too high. Normally it should be bufpages/2 but in your case it is more than the bufpage.

Re: Possible CPU Bottleneck

Charles McCary — Wed, 17 Apr 2002 18:49:21 GMT

Sandip,

Hmmm - didn't notice that.

sysdef reports nbuf as 90443, and SAM (kernel parms) shows it set to 0.