How can I gather performance history for redhat linux

[wvsa]

Greetings fellow admins;

Wondering how to gather and keep performance stats for our linux blades. On hpux we can use glance and ovpa (measureware) software to extract data from ovpa logfiles (logappl, etc) to spreadsheets. How can this be done on linux, see there is glance and ovpa available for linux, is anyone using glance/ovpa? We have HP-SIM but have not used it much wondering if that is a option or is there (no doubt) another option.

Thank you for your responses, and will do my best to assign points.


Norm

[Ivan Ferreira]

You can use the sar command and schedule via cron. This is all done if you install the sysstat package.

Another, my preferred option, is to use collectl (collect for linux).

http://collectl.sourceforge.net/

[Glenn S. Davidson]

I've installed OPVA on Linux servers and it seems to work like the HP version. I'm able to collect similar information.

I use it primarily with OVO so I have never done exports or anything but I would imagine it would work the same.

Easiest way is to download the Linux version and check it out.

[Steven McCoy]

I've thrown together a "quick n dirty" script for us here (until we purchase the license for Measureware):

----------------------------------------------------
#!/bin/sh

date=`date +%Y/%m/%d\ %H:%M:%S`
date=": $date "
loadavg=`uptime |awk {'print $11'}|cut -d ',' -f1`
cpuidle=`top -b -H -n 1|grep Cpu|awk {'print $5'}|cut -d '%' -f1`
cpuload=$(echo 100 - $cpuidle| bc -l)
io=`vmstat |grep -v -e io -e bi|awk {'print $9, $10'}`
#perc=`sar -r 1 1|grep Average|awk {'print $4'}`
mem_output=`free -m|grep Mem|awk {'print $2,$3,$7'}`
mem_total=`echo $mem_output|cut -d ' ' -f1`
mem_used=`echo $mem_output|cut -d ' ' -f2`
mem_cached=`echo $mem_output|cut -d ' ' -f3`
mem_actual=$(( $mem_used - $mem_cached ))
mem_perc=$(echo "scale=2; $mem_total / $mem_actual" | bc -l)
swap=`vmstat |grep -v -e io -e bi|awk {'print $8, $9'}`

echo "$date: $loadavg" >> /rdisk01/stat/load-stat.log
echo "$date: $cpuload" >> /rdisk01/stat/cpu-stat.log
echo "$date: $io" >> /rdisk01/stat/disk-stat.log
echo "$date: ${mem_perc}" >> /rdisk01/stat/mem-perc-stat.log
#echo "$date: ${actual}MB" >> /rdisk01/stat/mem-used-stat.log
echo "$date: $swap" >> /rdisk01/stat/swap-stat.log
----------------------------------------------------

[Alexander Chuzhoy]

I use sar and it's great. It actually installs 2 cron jobs that sample the system:

*/10 * * * * root /usr/lib/sa/sa1 1 1
53 23 * * * root /usr/lib/sa/sa2 -A

But if you want to show the stats to the boss - better use graphs: mrtg,cacti or even gnuplot.

[MarkSeger · Accepted solution]

Just to add a different perspective, and part of the reasons I wrote collectl, is I'm a firm believer in collecting a lot of data and collecting it often. The problem I find with sar is most people set it up using the default monitoring interval of one sample every 10 minutes. I suppose this is fine if you're looking for very coarse averages, but if you want to actually try and figure out what your system is doing that's not nearly frequent enough. collectl takes 10 second samples by default when run as a daemon and depending on the types of analysis I'm doing, I'll sometimes crank the sampling interval down to 1 second. Even at that level of granularity you're using less than 1% of the cpu and this will rarely effect the rest of what the system is doing. Of course your mileage may vary.

The other problem I have with sar is its output format. I want to be able to see everything that's happening each monitoring interval on the same line and that's why by default collectl displays its output in brief format. It you make your window wide enough you can see almost everything. Sar on the other hand chooses to include 2 decimal points of precision for virtually everything, wasting 3 columns per field. Do you really care how many MB/sec you're getting from your disk at that granularity?

On the other hand if you run sar at a monitoring frequency of 5-10 samples, you'll get useful data out of it. But then you still have to figure out how to visually parse it.

Have other played with collectl at all yet? One of my goals was to eliminate the need for all the *stat tools and others and I'd like to know if I succeeded or not.

Just a few comments to stimulate some discussion. ;-)

-mark

[MarkSeger]

Just to add a different perspective, and part of the reasons I wrote collectl, is I'm a firm believer in collecting a lot of data and collecting it often. The problem I find with sar is most people set it up using the default monitoring interval of one sample every 10 minutes. I suppose this is fine if you're looking for very coarse averages, but if you want to actually try and figure out what your system is doing that's not nearly frequent enough. collectl takes 10 second samples by default when run as a daemon and depending on the types of analysis I'm doing, I'll sometimes crank the sampling interval down to 1 second. Even at that level of granularity you're using less than 1% of the cpu and this will rarely effect the rest of what the system is doing. Of course your mileage may vary.

The other problem I have with sar is its output format. I want to be able to see everything that's happening each monitoring interval on the same line and that's why by default collectl displays its output in brief format. It you make your window wide enough you can see almost everything. Sar on the other hand chooses to include 2 decimal points of precision for virtually everything, wasting 3 columns per field. Do you really care how many MB/sec you're getting from your disk at that granularity?

On the other hand if you run sar at a monitoring frequency of 5-10 seconds, you'll get useful data out of it. But then you still have to figure out how to visually parse its output.

Have other played with collectl at all yet? One of my goals was to eliminate the need for all the *stat tools and others and I'd like to know if I succeeded or not.

Just a few comments to stimulate some discussion. ;-)

-mark

[dirk dierickx]

sar _is_ great, if you know how to use it. it's probably not as user friendly as glance.

ovpa works just as it does on hpux and solaris.

a wonderfull open source tool is collectd (http://collectd.org/index.shtml) and to a lesser extend cacti (http://www.cacti.net/screenshots.php).

[MarkSeger]

2 comments about sar:

If you do use it, set your monitoring interval to something that will actually show you what's really happening on your system. Using an interval of 10 minutes results in mush since you can't see any abnormalities.

One of the biggest problem I have with sar is its output format which does not make very efficient use of screen real estate. I want to see a bunch of columns of the same data, one line/sample, almost like a spreasdheet. This makes it much easier to see when something changes. It also makes it real easy to scrape that data and load it into other tools.

-mark

[westb]

Does collectl handle disks named like xvda, xvdb etc. (xen guest disks) or the dm-1 type of disks ?