system performance

[Rosli Ahmad]

Hi,
How can we check system's performance other than top? My user is complaining that the system is slow but the load avg from top is less than 10. I noticed that the system is 60% while user roughly 40% from top.
What other resources should I look at and what command to use. Pls. advise.

thank you,
-rosli-

[BFA6]

Hi Rosli,

I tend to use glance to monitor a system where users are complaining about performance.

You could also use sar to get disks stats, and other info.

Regards,

Hilary

[Massimo Bianchi]

Hi,
usefull tools for checking performance:

free
sar, vmstat : io for disks and memory

sar -d 3 1000 : check for disks
if %busy it too high you are reaching disk maximum throughput.


sar 3 1000 : general check

uptime :gives you the run queue, should be 3 or less

swapinfo -mt : gives you swap information, usually the lesser the better.

see man for details.

payment tool (but it deserve):

glance

HTH,
Massimo

[Stefan Farrelly]

Your load average is < 10, so what is it ?

> 0 or 1 ? The normal guideline is a load average of 1 per cpu, if you have 10 cpu's then a load average of 10 is ok. If you have 1 cpu then a load average > 1 is not ok.

[Ian Lochray]

A system CPU figure of 60% seems way too high. Which processes does top report are using the most CPU? A load average of 10 is also very high (unless as already mentioned you have a lot of CPUs) but I am not surprised it is so hihg if your CPU is 100% used.

[Rosli Ahmad]

Hi,
My load average is 7 with only 1 CPU. So I guess the load is very heavy then. The swap utilization is at 70%. Any idea how can I drill down to the real culprit that's eating up my resources? I'll have to provide a report on this. Appreciate your reply.

thank you.
-rosli-

p/s: Glance is not yet 'affordable' (that's what they said)

[Gavin Clarke]

Yes I'd agree with that too, isn't it supposed to be around 20% sys? I've got a document here (only in paper form I'm afraid) that suggests turning off system accounting or auditing, is this a possibility?

On a general note I tend to worry most about memory. I use vmstat 2 10 and look for po's. There is also an adb command to get the minfree desfree lotsfree figures which you can compare against the free column from vmstat. Still this doesn't seem to help your immediate concern.

[Stefan Farrelly]

load of 7 with 1 cpu is terrible. The question is what is causing it, could be cpu, memory or disk.

If you have swap usage of 70% then it sounds like the cause is lack of memory, basically our physical ram size is 100% used, and an extra 70% used on top of that!

To find memory hoggers without glance use this;

UNIX95= ps -e -o vsz=Kbytes -o ruser -o pid,args=Command-Line | sort -rnk1 | more

It lists processes by memory usage (rough total only - not exact). Hopefully you can id the heavy memory users and get rid of some. But with swap 70% used your going to need to do something drastic to get it back down to 0%, you may have to purchase more ram.

[Ian Lochray]

Rosli,
can you tell us what processes are using most of the CPU and most of the memory? Post the output from the following commands:

UNIX95= ps -eo "pid ruser pcpu vsz=Kbytes" -o args | sort -rn -k 3 | head -n 10

UNIX95= ps -eo "pid ruser pcpu vsz=Kbytes" -o args | sort -rn -k 4 | head -10

[Michael Steele_2]

Samplings during peak periods with this few commands tell volumes:

sar -u 5 5
sar -v 5 5
sar -b 5 5
sar -d 5 5
vmstat 5 5
swapinfo -tam
ipcs -mob

For memory leaks and other per process details then use 'glance advisor'.

[Michael Tully]

You might also post the output of 'top -f >/tmp/top.out' . This may indicate something else. Generally speaking systems that run like this have a big shortage of RAM and or swap.

[Massimo Bianchi]

Hi,
in addition to previous cheks,
post the output of

ipcs -ma

so we will see if there is an application hungry for RAM.

What application have you installed on the server ?

Massimo

[Bill Hassell · Accepted solution]

The high swap usage and the high system (versus user) usage indicates that your system is crippled with a lack of memory. What is likely happening is that too many processes need to run at the same time (like 7) yet there is only one processor and not enough RAM. So the load factor (which is really the run queue) is high because programs can only run one at a time, but worse than that, there is not enough RAM so swapping must take place. That places more than 100:1 penalty on performance. Your system is struggling to make progress but it is far too small (RAM and CPU count) for the workload you are trying to put on it. The only workaround would be to stop running most of the processes and let one or two jobs run at the same time.

[Rosli Ahmad]

Hi,
I don't think I should post the system stats for now, it's not peak hours. I'll run the suggested commands and post the output. Thank you for all the explainations.

regards
-rosli-

[John Meissner]

some things you can use.

Top (as you know)
glance plus (licensed product)
perf view & measureware (licensed product)

Perf view is not a "real time" monitor like top or glance plus is - but you can get info for a large specified range of time including a few minutes ago. You can use this and drill down and determine exactly where the bottle neck is and figure a way on how to correct it. You can create a graph of whatever selected item you are monitoring or look at a spreadsheet layout. A great tool that we use ALL the time.

[Gavin Clarke]

As usual I'm way behind the rest. I'm curious to know how this has happened? It sounds like it only gets like this at peak time? Is this a relatively new machine/implementation. Do you know if anything has changed?

I think I would be looking at asking for more memory. Those stats would be interesting to see.

[Rosli Ahmad]

Hi,
I'm posting the stats for references. As for the application, this server is an ERP server with QAD and Progress database. Again, it's with 1 CPU and 1GB memory. Yes, this is a new server, we've just migrated last week from a D390 with 750MB RAM. But no one was complaining then.

Pls. advise, my seat is getting hot by the hour.

thanks
-rosli-

[Massimo Bianchi]

Hi, llokig at the stats i have some questions:

- i saw many java process, version 1.3. This version has a poor performance, a a lot of memory request. Why do you need it ? For the progress, QAD, or for other purpose ?

- i saw a sanmgr, lookslike this server is attachaed to a SAN. what devices ? why is sanmgr active ?

- maybe it's a chance, but top usage where shell from a user. If this is normal, check with the users. 7% for a sh process looks much to me.

- most of your system wait state in in the system area. How are the stats for the disks ? maybe your disks are not well tuned, and are causing the bottleneck. check with "sar -d 3 25" and post the output, please.

- how are your lan card set ? maybe there is negotiation between nivc and switch, try setting them to the correct speed, avoiding auto-negotiation.

- you said that you changed server last week. did you, or some of your fellow, changed somethig ? areas to check: version of db, version of executables, settings for db and executables.

HTH,
Massimo

[schneider_15]

Hi,

you say you've just migrate your server, did you tune the kernel parameters ? They can have a real influence on performance even if the lack of memory is the most obvious bottelneck.

To get the kernel parameters use kmtune

regards

[Rosli Ahmad]

Hi,
To answer Masimo's question:
1. Java are for the QAD's processes
2. We have an enclosure but not a SAN. I believe sanmgr is part of the ems. We did have problem dm_ses_enclosure eating-up 99% CPU load but was advised to install a patch by HP.
3. will check the users sh accordingly, but this is isolated cases.
4. Attached is the disks stats, pls. go thru.
5. how do I go about checking my lan card setting?
6. The DBA installed the db as per his QAD/Progress guidelines. I can't say much on that.

AS for the kernel, it was agreed between the HP engineer and our DBA.

I wonder if I have to go thru the kernel tuning all over again.

thanks.
-rosli-

[Michael Tully]

I would suggest you shutdown 'ems' for the moment and anything else you can get away with, even diagnostics for a little while.

# /sbin/init.d/emsa stop
# /sbin/init.d/diagnostics stop

lancard e.g.

# lanadmin -x 0

Speed = 100 Full-Duplex

The disks are busy. It is a little hard to tell what can be done.

One further note, I hope this hasn't been asked before. What is the output of these?

# kmtune -l -q dbc_max_pct
# kmtune -l -q dbc_min_pct

[Massimo Bianchi]

Hi,
as Tully said, disk are busy.
In particular c8t0d0, look if you can move data away from this.

ask your dba differences between previous server and this. if he/she changed some version of software, this may have high impact on performance.

HTH,
Massimo

[Rosli Ahmad]

Hi,
I wonder what are the consequences of shutting down ems & diagnostic tools, if needs to then I will shut them down.
My NIC is set to half-duplex as per attachment. I'll be changing the dbc_max_pct from 50 to 10 later today. I'm thinking of creating another swap area, currently there's one 2GB swap,/dev/vg00/lvol2. How do I go about in creating the next swap space especially I'm with mirrored disk.

Pls. advise.

Thank you
-rosli-

p/s: The software version is the same with the previous server.

[Stefan Farrelly]

Im afraid I do not recommend shutting down the diagnostics. LVM uses them to find out if you have a disk thats going bad, or has died, and wether to use the pvlink, or make it stale, or break the mirror etc. The diags dont use much memory anyway.

Far more important you reduce the dbc_max_pct first and reboot. This will give a big performance boost and free up lots of memory.

Its easy to create more swap;
Create a new lvol in vg00, extend it to the size you want, then mirror it (lvextend -m 1). If you have enough diskspace on primary and mirror disks the lvextend -m 1 will work. Then add it into /etc/fstab as swap, eg;

/dev/vg00/lvol9 / swap defaults 0 0 # secondary swap

And do a swapon -a
Thats it.

[Massimo Bianchi]

Hi,
just a little note to Stefan hint.

Try the swap before the reboot, because you may have to change some kernel parameter to allow more swap, like maxswapchunks.

I wuold raccomend:
dbc_max_pct 8
dbc_min_pct 5
maxswapchunks 16384
nswapdev 25

Let wait for other comments.

HTH,
Massimo