Performance issue with a particuar job(it is taking 2 hours instaed on 50 mon)

[skt_skt]

OS HP-UX 11.11;9000/800/S16K-A; 8CPU(750MHz;45GB RAM0; server is already patched to standard level as per yearly pathcing policy.

There were no system bottlenecks (IO, Memory, or CPU) that slowed the performance. Even at the peak mem usage for this program, there was another gigabyte of free memory on the system -- if the program wanted to use it, it could have

How to identify if a particular job is spending time in CPU(The majority of the job's time was spent in CPU ;greater than 90%).

How to use the CPU time more efficiently.

[Hein van den Heuvel]

Interesting.

Relatively good starting info. Thank you.

IMHO It is very likely that the process is doing what it needs to do, and that very little can be done other then upgrading to faster CPUs. The application might just be very suitable for Itanium ?!?!

Anyway, the missing information is

A) How many units of work are being processed now, versus the actual business work done when the (elapsed?) time was just 50 minutes. This could be counted in order, customers, messages or whatever seems appropriate. If twice as many gizmos are processed, then using a little over twice as many resources might not be unreasonable.


B) How much time is spent in SYSTEM mode versus USER mode. Is there is a lot of system time (doesn't sound like it), then kernel tunign might help. If it is more than 90% user time, then kernel tuning can at most help 10% no?

C) Did the total amount of data to be processed perhaps change? Can the application be told to expunge some old records? If it has to wade through records from 2004, 2005, and 2006 before reaching current data, then that can possibly be improved, data retention rules permitting.
Ditto for 'stale' products / customers / ...

D) Is there an (Oracle?) Database behind this application which can be asked about performance details and possibly be tuned?

E) File system / Log files filling up?

F) Is the application just using 1 CPU?
Does the application have an opportinuty to process work in parallel. Maybe you can divide and conquer! May favourit method for this is a number (4? 8? 16?) worker streams which keep on selecting chunks of work ordered by descending size untill all done. Perhaps customers starting with xxx or zip-codes or phone area numbers, anything which allows for easy dividing up.

Finally one technical tip...
Did you ever try 'chatr' on the executable?
Run it first with no option to list the current settings. Well, first 'man chatr' of course. next try chatr -pi / -pd

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[A. Clay Stephenson · Accepted solution]

I have no idea how to answer this:

How to identify if a particular job is spending time in CPU(The majority of the job's time was spent in CPU ;greater than 90%).

If the job was spending more than 90% of it's time in the CPU then it's spending time in the CPU.

Bear in mind that there may be nothing wrong and a particular job is running longer than expected because of the data --- which might be simply more data or different data. Some kinds of computation scale exponentially rather than linearly with the size of the data.

There is no "speed up" command for the system. One thing you should determine is the fraction of user time and system time for your process. If the majority of the time is user then nothing you can do is going to fix this because it's working as hard as it can. To fix these problems, you have to change the algorithms. If you do have access to the source then you can profile the code and see where it is spending most of its time. If the application is multi-threaded then adding more processor could help.

It would help if you described a little about this application. Have you examine the context-switch rate.

[skt_skt]

How much time is spent in SYSTEM mode versus USER mode?How to find this?


How much time is spent in SYSTEM mode versus USER mode?How to find this?

[James R. Ferguson]

Hi:

> How much time is spent in SYSTEM mode versus USER mode?How to find this?

You can use 'glance', toggle the CPU report and select the process id in question. In fact, 'glance' can be used to evaluate many aspects of your job.

Regards!

...JRF...

[skt_skt]

i see the context switch rate of (Forced CSwitch=700) on this 8 CPU system.

is that high?

As oberved more time of this process is in user mode)(30% on system mode;60%on user mode;a few with A and R)

what is this A and R on whihc i see after toggling to CPU report.?

[skt_skt]

"If it is more than 90% user time, then kernel tuning can at most help 10% no?"

do we need to thik about the kernel param if 60% is in user mode ?whihc parms?

[Hein van den Heuvel]

>> see the context switch rate of (Forced CSwitch=700) on this 8 CPU system. is that high?

That does not strike me as too high. It would not seem a major cause of concern. As you monitorred processes, did they mostly stay on the same cpu (goodness) or bounce around (bad, and has a relation with too much context switches).

> As oberved more time of this process is in user mode)(30% on system mode;60%on user mode;a few with A and R)

30% system is not too good.
It could indicate options to tune stuff, like IO stack, scheduling, memory management.

It also suggest a good opportunity to do some system call tracing as a relatively easy method of profilling the application.
Check out tools like truss/trace/tusc:

http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/tusc-7.8/

Santosh,
You did not yet address many / any of the questions we raised. Notably... Is the process simply asked to do 3 times more work? Sometimes that can be due to gradual increase, sometimes due to a surprise / wrong new selection of data to work on.

That brings up an other question classic performance question: "What changed"? Did the run time jump from 50 minutes to 2 hours 'over night' or gradually over a year or more? If it was sudden, then what really changed at that time? OS version? Patches? Hardware changes? Bought a new business? Added encryption? Did a disk in a raid-5 set go bad?

The other unanswerred question is some indication of what the process might be doing, and whether a database is used at all.

Regards,
Hein.

[skt_skt]

The major change happened is three more databases moved to this server ; As per analysis the average MEN utilization became 90%(from 75%) and CPU utilization became 80%(from 60%).

But MEM usage not reached 100%(reached 98%) when we run this program last time. Also the GLOBAL_PRI_QUEUE was below one as usual when this program was running. PV report for CPU does not say it is touching threashold(like 95-100%).

[skt_skt]

tusc was always reporting running in user mode..