Any (constructive) comments on this...

[Carlo Henrico_1]

/ on server1 (root) $ sar 5 5

HP-UX server1 B.11.00 U 9000/800 06/18/08

19:46:31 %usr %sys %wio %idle
19:46:36 81 19 0 0
19:46:41 82 18 0 0
19:46:46 81 18 1 0
19:46:51 79 21 0 0
19:46:56 81 18 0 1

Average 81 19 0 0

[Patrick Wallek · Accepted solution]

As it relates to what?

This doesn't tell us a whole lot other than that you currently have a very busy system. The ratio of USR to SYS percentages indicates this is mostly user, probably application, processes that are running.

Any other comments will require more information. Output from top or glance showing what processes are running would be a good start.

[TTr]

Your server has 100% CPU utilization. Is that bad? Maybe, maybe not. It would be bad if you have processes that are CPU bound (ie. wasting CPU cycles for no reason) or processes that don't need to be running and you have valid processes waiting to take their turn in the CPU. It is perfect if you don't have any processes waiting for CPU. That means you get very good usage of your CPU. In general you want the CPU utilization not to be 100% so that you have some room for growth and accomodate those peak times. You did not provide any other details...

[Hein van den Heuvel]

Looks like the system is nicely busy.
Is it doing all the work it is supposed to do?
Is it the kind of environment where there is an infinte supply of work, so 100% busy is expected?

What is the system doing, versus what is it supposed to do? Must of us are not psycics here, so you have to help us with this!

Typically the next logical step to run it : top
This will show which process(es) are responsible for the CPU consumption, and you then have to know the application to know whether that is good or bad.

You may bee serious tools liek caliper and freinds.

a simple ps -ef may help you show where the 'burn' is.

Good luck,
Hein van den Heuvel
HvdH Performance Consulting

[Carlo Henrico_1]

Hi guys

Thanks for the comments so far.

Top shows our apps (3 of them) consuming most of the CPU cycles then Oracle and Websphere MQ.

We do have a problem that at certain times of the day it seems to the user that the machine has "frozen", I think this is when it gets seriously busy. The processes are very busy most of the day, but twice a day there are certain runs which "push it over the top".

Basically the working day for the users are getting longer and longer... :(

Overall, possibly time for an upgrade on CPU's?

Cheers

Carlo

[Patrick Wallek]

Possibly.

What model server is this?

# model

Also, as I'm sure you are aware, HP-UX 11.0 is now out of support.

If possible, it may be time to look into a new server running a supported version of HP-UX.

[Carlo Henrico_1]

Hope the formatting works...here is glance. The model is 9000/800/L1000-5X
B3692A GlancePlus C.03.55.00 20:43:26 zwdproda 9000/800 Current Avg High
--------------------------------------------------------------------------------
Cpu Util S SRU U | 99% 99% 100%
Disk Util F F | 28% 20% 44%
Mem Util S SU UB B |100% 100% 100%
Swap Util U UR R | 41% 41% 41%
--------------------------------------------------------------------------------
PROCESS LIST Users= 3
User CPU Util Cum Disk Thd
Process Name PID PPID Pri Name ( 200% max) CPU IO Rate RSS Cnt
--------------------------------------------------------------------------------
java 5277 1 239 operator 64.6/63.7 3173.8 24.9/28.2 330.2mb 16
myap 5881 1 240 apps 61.3/62.9 2681.4 0.0/ 0.0 19.8mb 9
oraclezwdcr 5893 1 154 oracle 15.7/14.7 627.7 2.1/28.6 21.7mb 1
oraclezwdcr 5913 1 156 oracle 15.7/14.7 624.6 0.9/ 2.4 20.8mb 1
amqzlaa0 5438 4959 154 operator 7.2/ 5.7 280.2 3.1/ 3.0 6.4mb 31
oraclezwdcr 5904 1 154 oracle 6.8/ 4.7 200.2 0.0/ 0.0 20.6mb 1
amqzlaa0 4966 4959 154 operator 3.9/ 3.9 194.7 0.3/ 1.9 6.5mb 33
amqhasmx 4961 4959 148 operator 2.5/ 1.8 90.7 93.5/67.9 3.1mb 2
midaemon 1226 1 -16 root 2.3/ 2.0 114.5 0.0/ 0.0 8.4mb 3
amqcrsta 7577 708 154 mqm 1.4/ 0.9 21.8 0.0/ 0.0 1.2mb 3
oraclezwdcr 5403 1 154 oracle 0.6/ 1.4 71.2 0.0/ 0.4 22.1mb 1
YELLOW ALERT CPU Bottleneck probability= 75.00% Page 1 of 4

[Steven E. Protter]

Shalom,

Your system is being used.

That means you are getting your moneys worth, at least for the 25 seconds covered on this report.

If there are complaints like slow response time, then you have a performance problem you might wish to address. To do that you need better tools.

http://www.hpux.ws/?p=6

SEP

[Carlo Henrico_1]

PS: All our HP-UX 11.0 customers are all due for upgrades soon but priorities may be decided by this type of investigation!!!

Carlo

[TTr]

So you did have glance after all...

You not only have CPU problems but more importantly memory problems as well. The problem that you have at certain times when the machine "has frozen" as you say, is probably caused by disk swapping more than CPU. Hit "w" in the glance screen to see the disk swap utilization. You could limit buffer cache to conserve memory and see if you can turn off any other processes that are not needed. Ask the DBAs to do the same with tuning their database and apps and not run anything that is not needed. It would help some but don't expect a lot.

[Carlo Henrico_1]

Steve - I have run that script, can I upload the results for you to look at perhaps?

TTr - Here is the Glance -> w

Unfortunately the system has slowed down now, its after hours but the mem utilisation still a concern I think:

B3692A GlancePlus C.03.55.00 21:16:36 zwdproda 9000/800 Current Avg High
--------------------------------------------------------------------------------
CPU Util SU | 2% 3% 20%
Disk Util | 0% 3% 23%
Mem Util S SU UB B | 99% 99% 99%
Swap Util U UR R | 41% 41% 41%
--------------------------------------------------------------------------------
SWAP SPACE Users= 2

Swap Device Type Avail Used Priority
--------------------------------------------------------------------------------
/dev/vg00/lvol2 device 2.0gb 0mb 1
/dev/vg00/swap device 1.1gb 12mb 0
pseudo-swap memory 1.5gb 411mb na







Swap Available: 4719mb Swap Used: 422mb Swap Util (%): 41 Reserved: 1939mb
Page 1 of 1

[Bill Hassell]

100% CPU is not meaningful What is meaningful is uptime or top, and report the load value. This load value indicates the average number of processes that are running and ready to run at 1,5,15 minutes. You have a two CPU system (based on Glance: (200%)) and that means that 2 processes can run at the same time for a load value of 2. But the kernel will put into the run queue as many programs as are ready to run. So a load value of 5 or 10 means there simply aren't enough CPUs to get all the work done. The ideal load value is the same as the number of CPUs. For 8 CPUs, a load value of 8 perhaps a bit more, is good.

As far as memory, using all your RAM is a good thing but using more than you have can slow the system. To see how slow, look at the output of vmstat, specifically the po column (page out). Single digits = OK, 2 digits = caution, 3 digits or more = serious paging (swapping) and very slow operations. Only two fixes:

1. Stop running some of your programs

2. Buy more RAM (double or triple)

[TTr]

So you have 423MB swapped out to disk. And this is during a "slow down" period. I can imagine what this would be in busy periods. Most likely this swapping is very active when your system becomes frozen. You need to do major cleanup on your server and consider some serious hardware upgrades.

[Hein van den Heuvel]

TTr,
With all due respect, if a system runs @100% cpu, mostly user mode, then is does not have a swap/memory problem. period.

fwiw,
Hein

[TTr]

> Hein: if a system runs @100% cpu, mostly user mode, then is does not have a swap/memory problem. period.

I don't understand what you are trying to say. I do realize if you were to add CPUs to a server like this, the user processes would enter the system, run and exit faster and would not stay in the server long enough to occupy all that memory space. However I don't think it would alleviate the memory problem enough to eliminate swapping and "freezing up" conditions. He is running databases/apps in this server and database and app processes tend to stay in-core longer (think transaction processes with sustained I/O) than typical user processes.

[Carlo Henrico_1]

Hi Bill

Here is vmstat:

/ on zwdproda (root) $ vmstat
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
0 0 0 18279 6006 17 6 4 0 10 0 77 518 8163 1670 27 8 65


Formatting is a problem. po = 0.

TTr - Swap used always grows and gets to 100% sometimes...Machine has just been boot thats why its just over 40%.

[TTr]

I think you need to add memory and CPUs to this server. You may be able to clean up somewhat but first you need to identify what needs to be cleaned up and that's not an easy task. You need to go through each process and identify what it is and decide if it is needed or not. There are areas in the database to optimize resources as well. You or your DBA/app person should look into that as well.

[Carlo Henrico_1]

Thanks for all the comments guys - would have liked to hear about the swap-creep still but this has given me plenty food-for-thought! Points awarded.

Carlo

[Jeeshan]

may be some kernel parameters can help you to tune this. but if your application is hungry than, nothing but upgrade will give you relief. ;-)