Performance issues on K360 system

[Miah Tan]

The K class hpux 10.2 system with 1G RAM has once in a while encountered slowness in performance.
I performed a 'sar -d' command.I noted that the wcache and rcache are always less than 60% and 20% (sometimes even <5%). Problem?
Do I need to increase the value to 90% for rcache and 60% for wcache as what stated in some books/articles. What could be the possible cause? Maybe the way the data are accessed is very randomized.
I was told to increase the size of kernel buffer cache. However, I found out that the value is set to 0 (dynamic).
Should I proceed with the change to increase the percentage for rcache and wcache? If so, what is the recommended value?
A check on swapinfo shows that the total memory utilized is usually < 50% but the %idle in 'sar -u' has often returned a value close to ZERO. Is adding an extra CPU the only alternative?
The pageout rate is 0 constantly with the 'vmstat' command which I think is good.
Sorry for not structuring my questions in an orderly manner.

[Andy Zybert]

The usually default setting for dbc_max_pct is 50%, this is usually too high and should be set to around 300 Mb or on your system 30%.

If the filesystem buffer cache is sized correctly I would expect caching rates of rcache 98-100% and wcache 60-70%.

Hope this is useful.

[Stefan Farrelly]

Your memory usage is fine - not a problem. If your %idle is almost zero constantly then yes - youre definitely out of CPU ! only another CPU or replacing with a faster one will help.

First do a check by running top to see who is using most of the CPU - just in case you have a runaway process which you could kill.

[Steve Steel]

Hi

2 quick scripts

Top 10 cpu with ps

UNIX95= ps -e -o pcpu -o ruser -o vsz -opid -o args|head -n1
UNIX95= ps -e -o pcpu -o ruser -o vsz -opid -o args|grep -v %CPU| sort -nr|tail-n +2|head -n 20

Top memory

#!/usr/bin/ksh
#
# Show processes sorted by size of core image
#
# Usage:
# psram [ quantity ]
#
# where quantity is the top RAM processes to show (default is 20)
#
set -u
if [ $# -gt 0 ]
then
TOPPROCS=$1
else
TOPPROCS=20
fi

MYNAME=$(basename $0)
TEMPFILE=/var/tmp/$MYNAME.$$
trap `rm -f $TEMPFILE > /dev/null 2>&1` 0 1 2 3 15

UNIX95= ps -e -o ruser,vsz,pid,args > $TEMPFILE
head -1 $TEMPFILE
DASH5="-----"
DASH25="$DASH5$DASH5$DASH5$DASH5$DASH5"
echo "$DASH5---- $DASH5- $DASH5 $DASH25$DASH25"
grep -v "VSZ COMMAND" $TEMPFILE | cut -c -78 | sort -rn -k2 | head -${TOPPROCS}
rm $TEMPFILE > /dev/null 2>&1
#### END OF SCRIPT


Steve Steel

[Ruediger Noack]

Your cache rates are to less, much to less. I think this causes also in high disk usage (check out with sar -d). I also believe you have a high wio value in sar output. This causes your idle value near 0. Therefore a second cpu will not solve your problem.
I guess your main problem is disk bottleneck or really a buffer cache problem.
In your case I don't agree to reduce dbc_max_pct. What is your current value? I believe you may need a large buffer cache if your box isn't running an application with own buffer cache like oracle.

Ruediger

[Sridhar Bhaskarla]

Hi Miah,

You are correct if your application does a lot of random I/O, you less likely get cache hits.

If your dbc_max_pct is default, then you may want to reduce it to between 20-25%. Adjust dbc_min_pct between 10-15%.

Look out for wio in the same sar command. If your %usr is high (around 70%) then you will need to buy some CPU power.

Make sure you install reasonably latest patches.

-Sri

[Miah Tan]

Steve,
Your scripts will come in handy.

Ruediger,
My %wio is hovering around values 0 to 30 most of the time. This is not a high value. Except for certain disks whose %busy can hit 50%, the rest of the diskio are ok (<10).

Sri,
My %usr is usually around 75 most of the time.
It has been quite a long while when patches are being applied to the system.

To All,
Thanks for the replies. they did help to cast some lights to what can be done to tune the wcache and rcache and the near zero CPU idling time for my system.

The values of dbc_max_pct and dbc_min_pct are currently 20 and 5 respectively.

I think CPU power need to be increased and I shall change the value of dbc_min_pct to 10. However, due to an imminent upgrade of the systems, I shall live with the near zero idling CPUtime for the time being...unless it really starts to crawl.
Thanks again.

[MANOJ SRIVASTAVA]

Hi Miah

Performance tuning for this specific system can be done based on the following :

1. Why is the CPU busy ?
2. Is the system doing more physical read / writes than logical read writes
3. Which voulmes in sar -d are having most waits.

Incase you find that wio is high then may be u need to tune the buffers or even realloacte the maximum used vg's so that he load is balanced

A good point to start is to change dbc_max and dbc_min actually I would recoomend that if you have them set to 5 and 50 which is default then decrease it to 25 and 5 this will give you a 20% of the total memory as the dynamic buffer . Please keep in mind that it will better to change one parameter at a time and then see its result.

Manoj Srivastava