topic Re: system performance in Operating System - HP-UX

system performance

Rosli Ahmad — Tue, 17 Jun 2003 08:15:38 GMT

Hi,
How can we check system's performance other than top? My user is complaining that the system is slow but the load avg from top is less than 10. I noticed that the system is 60% while user roughly 40% from top.
What other resources should I look at and what command to use. Pls. advise.

thank you,
-rosli-

Re: system performance

BFA6 — Tue, 17 Jun 2003 08:20:26 GMT

Hi Rosli,

I tend to use glance to monitor a system where users are complaining about performance.

You could also use sar to get disks stats, and other info.

Regards,

Hilary

Re: system performance

Massimo Bianchi — Tue, 17 Jun 2003 08:21:00 GMT

Hi,
usefull tools for checking performance:

free
sar, vmstat : io for disks and memory

sar -d 3 1000 : check for disks
if %busy it too high you are reaching disk maximum throughput.

sar 3 1000 : general check

uptime :gives you the run queue, should be 3 or less

swapinfo -mt : gives you swap information, usually the lesser the better.

see man for details.

payment tool (but it deserve):

glance

HTH,
Massimo

Re: system performance

Stefan Farrelly — Tue, 17 Jun 2003 08:36:28 GMT

Your load average is < 10, so what is it ?

> 0 or 1 ? The normal guideline is a load average of 1 per cpu, if you have 10 cpu's then a load average of 10 is ok. If you have 1 cpu then a load average > 1 is not ok.

Re: system performance

Ian Lochray — Tue, 17 Jun 2003 08:53:44 GMT

A system CPU figure of 60% seems way too high. Which processes does top report are using the most CPU? A load average of 10 is also very high (unless as already mentioned you have a lot of CPUs) but I am not surprised it is so hihg if your CPU is 100% used.

Re: system performance

Rosli Ahmad — Tue, 17 Jun 2003 09:24:43 GMT

Hi,
My load average is 7 with only 1 CPU. So I guess the load is very heavy then. The swap utilization is at 70%. Any idea how can I drill down to the real culprit that's eating up my resources? I'll have to provide a report on this. Appreciate your reply.

thank you.
-rosli-

p/s: Glance is not yet 'affordable' (that's what they said)

Re: system performance

Gavin Clarke — Tue, 17 Jun 2003 09:26:32 GMT

Yes I'd agree with that too, isn't it supposed to be around 20% sys? I've got a document here (only in paper form I'm afraid) that suggests turning off system accounting or auditing, is this a possibility?

On a general note I tend to worry most about memory. I use vmstat 2 10 and look for po's. There is also an adb command to get the minfree desfree lotsfree figures which you can compare against the free column from vmstat. Still this doesn't seem to help your immediate concern.

Re: system performance

Stefan Farrelly — Tue, 17 Jun 2003 09:40:46 GMT

load of 7 with 1 cpu is terrible. The question is what is causing it, could be cpu, memory or disk.

If you have swap usage of 70% then it sounds like the cause is lack of memory, basically our physical ram size is 100% used, and an extra 70% used on top of that!

To find memory hoggers without glance use this;

UNIX95= ps -e -o vsz=Kbytes -o ruser -o pid,args=Command-Line | sort -rnk1 | more

It lists processes by memory usage (rough total only - not exact). Hopefully you can id the heavy memory users and get rid of some. But with swap 70% used your going to need to do something drastic to get it back down to 0%, you may have to purchase more ram.

Re: system performance

Ian Lochray — Tue, 17 Jun 2003 09:43:28 GMT

Rosli,
can you tell us what processes are using most of the CPU and most of the memory? Post the output from the following commands:

UNIX95= ps -eo "pid ruser pcpu vsz=Kbytes" -o args | sort -rn -k 3 | head -n 10

UNIX95= ps -eo "pid ruser pcpu vsz=Kbytes" -o args | sort -rn -k 4 | head -10

Re: system performance

Michael Steele_2 — Tue, 17 Jun 2003 09:59:11 GMT

Samplings during peak periods with this few commands tell volumes:

sar -u 5 5
sar -v 5 5
sar -b 5 5
sar -d 5 5
vmstat 5 5
swapinfo -tam
ipcs -mob

For memory leaks and other per process details then use 'glance advisor'.

Re: system performance

Michael Tully — Tue, 17 Jun 2003 10:00:22 GMT

You might also post the output of 'top -f >/tmp/top.out' . This may indicate something else. Generally speaking systems that run like this have a big shortage of RAM and or swap.

Re: system performance

Massimo Bianchi — Tue, 17 Jun 2003 10:03:07 GMT

Hi,
in addition to previous cheks,
post the output of

ipcs -ma

so we will see if there is an application hungry for RAM.

What application have you installed on the server ?

Massimo

Re: system performance

Bill Hassell — Tue, 17 Jun 2003 10:10:08 GMT

The high swap usage and the high system (versus user) usage indicates that your system is crippled with a lack of memory. What is likely happening is that too many processes need to run at the same time (like 7) yet there is only one processor and not enough RAM. So the load factor (which is really the run queue) is high because programs can only run one at a time, but worse than that, there is not enough RAM so swapping must take place. That places more than 100:1 penalty on performance. Your system is struggling to make progress but it is far too small (RAM and CPU count) for the workload you are trying to put on it. The only workaround would be to stop running most of the processes and let one or two jobs run at the same time.

Re: system performance

Rosli Ahmad — Tue, 17 Jun 2003 10:31:21 GMT

Hi,
I don't think I should post the system stats for now, it's not peak hours. I'll run the suggested commands and post the output. Thank you for all the explainations.

regards
-rosli-

Re: system performance

John Meissner — Tue, 17 Jun 2003 10:36:05 GMT

some things you can use.

Top (as you know)
glance plus (licensed product)
perf view & measureware (licensed product)

Perf view is not a "real time" monitor like top or glance plus is - but you can get info for a large specified range of time including a few minutes ago. You can use this and drill down and determine exactly where the bottle neck is and figure a way on how to correct it. You can create a graph of whatever selected item you are monitoring or look at a spreadsheet layout. A great tool that we use ALL the time.

Re: system performance

Gavin Clarke — Tue, 17 Jun 2003 11:47:58 GMT

As usual I'm way behind the rest. I'm curious to know how this has happened? It sounds like it only gets like this at peak time? Is this a relatively new machine/implementation. Do you know if anything has changed?

I think I would be looking at asking for more memory. Those stats would be interesting to see.

Re: system performance

Rosli Ahmad — Wed, 18 Jun 2003 04:58:45 GMT

Hi,
I'm posting the stats for references. As for the application, this server is an ERP server with QAD and Progress database. Again, it's with 1 CPU and 1GB memory. Yes, this is a new server, we've just migrated last week from a D390 with 750MB RAM. But no one was complaining then.

Pls. advise, my seat is getting hot by the hour.

thanks
-rosli-

Re: system performance

Massimo Bianchi — Wed, 18 Jun 2003 05:36:02 GMT

Hi, llokig at the stats i have some questions:

- i saw many java process, version 1.3. This version has a poor performance, a a lot of memory request. Why do you need it ? For the progress, QAD, or for other purpose ?

- i saw a sanmgr, lookslike this server is attachaed to a SAN. what devices ? why is sanmgr active ?

- maybe it's a chance, but top usage where shell from a user. If this is normal, check with the users. 7% for a sh process looks much to me.

- most of your system wait state in in the system area. How are the stats for the disks ? maybe your disks are not well tuned, and are causing the bottleneck. check with "sar -d 3 25" and post the output, please.

- how are your lan card set ? maybe there is negotiation between nivc and switch, try setting them to the correct speed, avoiding auto-negotiation.

- you said that you changed server last week. did you, or some of your fellow, changed somethig ? areas to check: version of db, version of executables, settings for db and executables.

HTH,
Massimo

Re: system performance

schneider_15 — Wed, 18 Jun 2003 05:51:20 GMT

Hi,

you say you've just migrate your server, did you tune the kernel parameters ? They can have a real influence on performance even if the lack of memory is the most obvious bottelneck.

To get the kernel parameters use kmtune

regards

Re: system performance

Rosli Ahmad — Wed, 18 Jun 2003 06:35:47 GMT

Hi,
To answer Masimo's question:
1. Java are for the QAD's processes
2. We have an enclosure but not a SAN. I believe sanmgr is part of the ems. We did have problem dm_ses_enclosure eating-up 99% CPU load but was advised to install a patch by HP.
3. will check the users sh accordingly, but this is isolated cases.
4. Attached is the disks stats, pls. go thru.
5. how do I go about checking my lan card setting?
6. The DBA installed the db as per his QAD/Progress guidelines. I can't say much on that.

AS for the kernel, it was agreed between the HP engineer and our DBA.

I wonder if I have to go thru the kernel tuning all over again.

thanks.
-rosli-