system performance

[Michael Steele_2]

Regarding the 'sar -d' report: You have a disk bottleneck on c8t0d0 and an I/O bottleneck on c8t2d0. Is c8t0d0 the boot disk?

Everything else is fine.

Attach 'pvdisplay -v /dev/dsk/c0t8d0' and one for /dev/dsk/c2t8d0.

[Michael Steele_2]

Didn't see 'sar -v 5 5'. Need this also.

[schneider_15]

Do you have an oracle database on your, server.
If so, check if the rollback segment, data and index tablespaces are on separate disk.

If these 3 tablespaces are on the same disk, you can have such bottelneck. I've already seen some pl/sql procedure crash because of that.

regards

[Gavin Clarke]

Sorry got collared by work, so I've missed alot. Does seem like a raw deal that you've now potentially got more memory available yet performance is slower. I like the idea of reducing dbc_max_pct. What did the D390's lan card run at? I don't suppose they are different at either end? Mind you seems like the disk is bearing the brunt of the troubles.

I've not got much to add to what's already been said, just hoping you're getting nearer to the solution.

[Gavin Clarke]

Okay so the NIC is auto negotiating, missed that, sorry.

[Rosli Ahmad]

Hi,
I've tuned the kernel parameters by setting dbc_max_pct to 10 from 50 and increase maxswapchunks to 4096 from 1024. I've added another swap space of 2GB to the existing 2GB. We'll see wether users will still be complaining tomorrow. I hope not.Thank you.

rgds
-rosli-

[Gavin Clarke]

What's been happening, has the situation improved?

[Mike Patterson]

Here is an old script of mine that you cron every 15 min. to gather info into a log. The logic is based on basic hp-ux performance step-by-step troubleshooting methods (supplied by hp support and docs years ago). The key to finding transient performance problems is gathering info over time. (something glance will not typically do). Once you have this data in a log, you can see when performance problems occur (maybe when you are sleeping) and hopefully correlate it to some conditions. This script will also page (via emai) if cpu idle% is low. Please excuse the very old code, but it should still work:

hp - gather performance and page on low cpu
#!/usr/bin/sh
# /usr/local/scripts/sa.perf.sh
#
# Gathers performance info, pages on cpu low - run in crontab every 15 min.
#
# EXPLANATION:
#
# Performance Problems Troubleshooting Flowchart:
#
# Step 1.
#
# sar -u (or sar -Mu for multiple processors)
#
# Is the %idle low?
#
# No -> The system is not CPU bound. Goto Step 3;
# Yes -> The system is possibly either CPU, memory, or I/O bound.
# Goto Step 2;
#
# Step 2.
#
# Is %usr high? (What is normal on this system?)
#
# No -> The system may be experiencing either a CPU, memory, or
# I/O bottleneck. Goto Step 3;
# Yes -> CPU bottleneck due to user processes
# Goto Section 3, Part A, Tuning a system with CPU bottleneck
#
# Step 3.
#
# Does %wio have a value > 15?
#
# Yes -> Possible disk or tape involvement in a bottleneck. Goto Step 4;
# No -> Goto Step 4;
#
# Step 4.
#
# sar -d
#
# Is %busy for any disks >50? (or Is %busy higher than normal?)
# For same disa(s), is avwait > avserv?
#
# No -> Most likely no disk bottlenecks, go to Step 6;
# Yes -> There seems to be an I/O bottleneck on this device. Goto Step 5;
#
# Step 5.
# There is a disk bottleneck. What is on that bottlenecked disk?
#
# Raw Partitions,
# File Systems -> Use glance for more info and tune the disk problem
# Swap -> Possibly caused by a memory bottleneck. Goto Step 6;
#
# Step 6.
#
# vmstat (vmstat -n 3 -> readable 80 column output, 3 samples)
#
# Over sustained periods of time, is po > 0? Is (free * 4k) < 2 MB,
# for a s800 system (free * 4k < 1 MB for s700)? (The values 2 MB and
# 1 MB are rough guides, the actual value of LOTSFREE, the point at
# which a system begins paging, is calculated at system boot time and
# is based on the size of system memory.)
#
# No -> If, in step 1, %idle was low, the system is most likely CPU
# If %idle not low, something other than cpu, memory, disk is problem.
# Yes -> There is a memory bottleneck on the system.
#
#
umask 177
HOST=`hostname`
PERFLOG="/var/tmp/sa.hp.perf.log"
ALARM="/var/tmp/sa.hp.perm.alarm"
SALOG="/var/tmp/sa.hp.log"
PERCENT_IDLE_MIN=10
#
## This script should be run by root:

if [ x`id | grep root | awk '{print $1}'` = "x" ]; then
echo "Root privileges are required to run sa.hp.perm" >> $ALARM
exit
fi

echo "" >> $PERFLOG
echo "**************** `date` ******************" >> $PERFLOG
echo "**************** users on ****************" >> $PERFLOG
who -q >> $PERFLOG
echo "" >> $PERFLOG

# Check cpu with sar
echo "" >> $PERFLOG
echo "sar -Mu output:" >> $PERFLOG
echo "" >> $PERFLOG
sar -Mu 1 3 >> $PERFLOG

# Page someone if cpu idle is too low
PERCENT_IDLE=`sar 1 3 | tail -1 | awk '{print $5}'`
if [ $PERCENT_IDLE -lt $PERCENT_IDLE_MIN ]; then
echo "CPU on $HOST is only $PERCENT_IDLE idle" >> $ALARM
fi

# Check buffers with sar
echo "" >> $PERFLOG
echo "sar -b output:" >> $PERFLOG
echo "" >> $PERFLOG
sar -b 1 3 >> $PERFLOG
# %rcache should be >= 90
# %wcache should be >= 70

# Check io with iostat
echo "" >> $PERFLOG
echo "iostat output:" >> $PERFLOG
echo "" >> $PERFLOG
iostat >> $PERFLOG

# Check disk io with sar
echo "" >> $PERFLOG
echo "sar -d output:" >> $PERFLOG
echo "" >> $PERFLOG
sar -d 1 3 >> $PERFLOG

# Check virtual memory with vmstat
echo "" >> $PERFLOG
echo "vmstat -n 1 3 output:" >> $PERFLOG
echo "" >> $PERFLOG
vmstat -n 1 3 > /tmp/vmstat.txt
echo "" >> $PERFLOG
cat /tmp/vmstat.txt >> $PERFLOG
rm /tmp/vmstat.txt

# Check top process with top
echo "" >> $PERFLOG
echo "top -d1 output:" >> $PERFLOG
echo "" >> $PERFLOG
top -d1 > /tmp/top.txt
echo "" >> $PERFLOG
cat /tmp/top.txt >> $PERFLOG
sleep 2
rm /tmp/top.txt

# Notify someone!
if [ -f $ALARM ]; then
MAILLIST=[your mail list]
mailx -s "$HOST SYSALARM" "$MAILLIST" < $ALARM
#echo `cat $ALARM`
echo "`date +'%b %d %y %H:%M'` `cat $ALARM`" >> $SALOG
rm $ALARM