Script to monitor performance and alert adminstrators.....

[MAD_2]

Need some ideas on how to develop a script to monitor system performance and send alerts to administrator's pagers if the performance is extremely low for a period of time. Commands I would like to utilize include....
sar
bdf
vi if necessary for temp files...

Anyone has anything pre-built??? Thanks for any input

[Sridhar Bhaskarla]

Hi Mynor,

You certainly can make use of basic utilitiles like sar,vmstat, iostat in scripts and generate alarms. But there is one thing to remember. All these scripts are going to consume resources. And shorter the interval of polling, better the accuracy but at the same time, more load to the system.

This is the reason why administrators do not prefer to write scripts to monitor performance on their own. There are some softwares available that can do the jobs professionally and more elegantly. Glance+ Pak is one of them that has been very popular. Following URL has the documentation.

http://docs.hp.com/hpux/netsys/index.html

If you have application CDs, Glance+ comes with Free Trial version. You can give a try. There is good documentation in /opt/perf/docs.

'sarcheck' is a third party tool, that makes use of sar and generate reports. More details are here

http://www.sarcheck.com/

You can give a good reading at the following documents

DocID: A4460661, KBRC00002702

Search with docID option providing the above docIDs at this URL.

http://us-support.external.hp.com/cki/bin/doc.pl

I hope the above will give you some direction.


-Sri

[MAD_2]

Sri, thanks for your answer, it provides me with some leads I might be able to use.

I do know that creating such script would consume resources and of course that was a concern, but at the same time I would like to find a tool that alerts me about upcoming trouble before my customers do; in other words, I would like to be a little proactive. That is why I came here, because I know from past experience that someone would provide me with some good advise.

Thanks.

ADAM

[Phil Daws_2]

Take a look @ http://www.netsaint.org
There are a number of plugins that will check diskspace, load etc and you can also write your own plugins aswell. It is a nice reporting tool.

[Jason VanDerMark]

You may also consider looking into HP's OpenView product. OpenView works in tandem with the Measureware agents on each server. The software can then retrieve statistics for disk i/o, cpu, memory, disk space utilization, etc. You can then setup thresholds which when crossed will notify the admins via email or page. We are using OpenView at my site and I am very happy with it. You can find more information about HP OpenView at the following URL:

http://www.openview.hp.com/

I hope that this helps.

Regards,
Jason V.

[Wodisch]

Hello Mynor,

one product being able to do exactly what you ask for is "OpenView Performance Agent" (formerly known as MWA=MeasureWare Agent).
It does continually measure a lot of data (and you can add even more metrics, if you need), and it does check those against thresholds (which you can set as you like them) and even execute local commands, as an reaction onto crossing such a threshold.

Read the documentation about it (and the metrics it does use) from there:
User's Guide http://ovweb.external.hp.com/ovnsmdps/pdf/b4967-90049.pdf
Metrics http://ovweb.external.hp.com/ovnsmdps/pdf/b4967-90046.pdf

HTH,
Wodisc

[MAD_2]

Thanks everyone for your valuable advise.

What I was really looking for is for something we do not have to necessarily purchase, a solution within the tools that we already have at hand, without having to go an spend more on new applications. I guess that is what I face with OpenView. However, Jason and Wodish, this seems exactly like what I would love to have. I just would not dare go to my boss with a request to get this application, specially because at this moment it is not essential in our case.

Where I work at it is a little difficult to consider getting more tools than those that have helped us survive until now. Extra spending is just not necessarily welcomed; ingenious ideas on how to circumbent challenging situations are rather what is expected from the IT team (isn't that the norm?).

The solution Phil is providing sounds a little riskier to implement, and unfortunatly we do not have a test environment where we can monitor Netsaint "AS IS" product, since it was primarily developed for LINUX. A good thing about it is that it is free, and written in "C". Once we get a test server in place I may be able to load it in that server and see how things work out!!!

[Steven Sim Kok Leong]

Hi,

Here's another free solution.

HP-UX's snmp MIB comes with a lot of information on system performance such as CPU load, memory utilisation, swap utilisation, no. of TCP connections, bdf listing etc.

Snmp queries ride on top of UDP traffic, thus it uses considerably less network resources.

Once you run HP-UX snmpd and mib2agt on your system, you will be able remotely query the MIB values via snmp protocol.

A free thresholding+alerting+web-graphing tool that does that is MRTG which can be found at http://www.mrtg.org.

MRTG allows you to specify the value you wish to trigger off an alert and also the command you use to execute the alert eg. pager alert, email, pop-up message etc. MRTG runs primarily on Unix and Linux.

You can also trigger SNMP traps to another SNMP server on based on thresholds and triggers.

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[jack Hu_1]

Hi
I just use another free tools.
"RRDTOOL"
You can search it on the internet.
Basic,it use the snmp mib.
So I can use it to get some server information.
Like: loading,memory free.....
I also write some script to update the RRD file.
Like: informix users
So it can display all the information I want.
Enjoy it.
Jack Hu

[MAD_2]

Steven & Jack:

Just got back to work today, had a nice four days off!!!

Thanks for your replies, this may look like what I need, I just need to go and evaluate the material and then give it a try. It also looks like there may be less risk involved since from what I understand you both are saying, it is already included with HP-UX.

We'll see, I will assign you both points once I figure out how this works and if this is really what I was looking for.

[Krishna Prasad]

I write my own scripts. I found it easier to check the things I know need checked and I set my own threhholds per machine and application. I then send messages to X-statations Windows to myself and other members of my team. I also send them to the help desk staff that is 24x7.

The problem is of course things will always change in any environment so I do have to change some scripts, write new ones, change thresholds from time to time. But I still found it easier then setting up a package like ITO. I found that I spent more time making sure ITO was working correctly then the machines I was monitoring.

[harry d brown jr]

Mynor,

Look at this:

http://bb4.com/


live free or die
harry

[Mark Greene_1]

Here's a script to monitor space:

MAX_PERCENT=95
LOW=0
CMD="bdf|tail +2";;
OLD_IFS=$IFS
### set the field separator to a newline ###
IFS="
"
eval $CMD >/tmp/space_tmp.$$ 2>&1
exec 4 while read -u4 LINE; do
if echo $LINE|grep "%">/dev/null; then
if [ `echo "$LINE"|tr -s " "|cut -d' ' -f5|tr -d "%"` -ge $MAX_PERCENT ]; then
LOW=1
fi
fi
done
IFS=$OLD_IFS
if [ $LOW -eq 1]; then
mailx -s "`hostname` Low Space Warning!" user@system <$/tmp/space_tmp.$$
fi

You can add to cron a line to run the script two-three times a day. Be sure to change "user@system" to a valid e-mail address for your environment.

HTH
mark

[MAD_2]

Mark, thanks for sharing the script. This partially helps, since space available, of course, is the one thing that everyone really needs to keep an eye on all the time. I can use the script as a backup tool, although I am really getting interested on Harry's "Big Brother" recommendation. I will assign points as I check and test the portability of these tools in my environment.

Thanks everyone, you are all the best support system available anywhere. There is no better way to get abundant ingeniuos ideas on how to resolve an issue than asking the experts in the field.

[jack Hu_1]

Hi:
There are one attachment below.
I just correct the loading data.
And then save it to the rrd (database file).
gererate the image file.
also show it on the web site.
I also correct data,then create another
RRD file,image file.show it on the web site,too.(like number of informix user,
transcation time of a program)

http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/

[MAD_2]

Steven and Jack,

I tried to run HP-UX snmpd and got the following errors:
---------------------------------------------
Start SNMP Master Network Management daemon
Start SNMP HP-UNIX Network Management subAgent
EXIT CODE: 255
ERROR: FAILED TO START SUBAGENT
Start SNMP MIB-2 Network Management subAgent
EXIT CODE: 255
ERROR: FAILED TO START SUBAGENT
---------------------------------------------
I am trying to get started and see what my best options are, can any of you guide me in right direction to see what may be wrong?

Thanks,

[MAD_2]

Can anyone provide me with links or guidance on where to find GOOD reference material "A dummies guide per say" about "HP-UX's snmp MIB".

I have always relied on then man pages as a quick reference for explanation on subjects I am kind of familiar with, but in the case of "snmpd and mb2agt" (with which I really have no familiarization) they leave me clueless.

Thanks.

[Steven Sim Kok Leong · Accepted solution]

Hi,

Check that these configuration files in /etc/rc.config.d have the respective startup variables set to 1:

1) SnmpHpunix
2) SnmpMaster
3) SnmpMib2
4) SnmpTrpDst

If you want to manually startup the Snmp daemons, execute the following in sequence:

/sbin/rc2.d/S560SnmpMaster start
/sbin/rc2.d/S565SnmpHpunix start
/sbin/rc2.d/S565SnmpMib2 start
/sbin/rc2.d/S565SnmpTrpDst start

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Steven Sim Kok Leong]

Hi,

You will see two processes running:
root 1336 1 0 Jan 5 ? 0:10 /usr/sbin/mib2agt
root 1283 1 0 Jan 5 ? 0:00 /usr/sbin/snmpdm

Both these binaries should be executable:
$ ls -la /usr/sbin/mib2agt /usr/sbin/snmpdm
-r-xr--r-- 1 bin bin 192512 Oct 27 1997 /usr/sbin/mib2agt
-r-xr--r-- 1 bin bin 471040 Oct 27 1997 /usr/sbin/snmpdm

Here are some OIDs to get you started which I extracted from my mrtg.cfg configuration file. Replace server with your server's DNS name or IP address and public with your server's public community string (default is public).

# CPU load:

Target[server.cpu]: 1.3.6.1.4.1.11.2.3.1.1.3.0&1.3.6.1.4.1.11.2.3.1.1.3.0:public@server

# Available memory:

Target[server.mem]: 1.3.6.1.4.1.11.2.3.1.1.7.0&1.3.6.1.4.1.11.2.3.1.1.7.0:public@server

# Available swap:

Target[server.swap]: 1.3.6.1.4.1.11.2.3.1.1.12.0&1.3.6.1.4.1.11.2.3.1.1.12.0:public@server

# No. of concurrent processes:

Target[server.proc]: 1.3.6.1.4.1.11.2.3.1.4.1.0&1.3.6.1.4.1.11.2.3.1.4.1.0:public@server
Target[server.tcp]: 1.3.6.1.2.1.6.9.0&1.3.6.1.2.1.6.9.0:public@server

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[MAD_2]

Steven,

The file "/sbin/rc2.d/S565SnmpTrpDst start" was not listed in that directory.

[jack Hu_1]

Hi
after U can start these process(list from Steven)
U can use these mib informix like this
http://smurfland.cit.buffalo.edu/ftp/pub/mibs/HP/hp-unix
I think it's the fist step,You can get the informix of HPUX system.
I always get these information to a Linux
system.
And write shell to get these information
to the RRDTOOLS.And also call pager if necessary.
And then,I found that I can also get some other
information than snmpd to RRDTOOLS.
So now that's what I done to the web.
see the attach file
Hope that's what you want.

[MAD_2]

I am coming back to this forum to check and see if there still anyone who can give me any other pointers about the original subject.

I have checked the replies posted to my original question and there is a lot of very useful links and ideas. As previously stated, I don't want to implement any monitoring tools for which we have to pay (maybe after I see the other alternatives).

Thanks to all of you who have participated...

[Paula J Frazer-Campbell]

Hi

This script will give you a good start point.

-----------------cut here--------------------

#!/bin/ksh
###################################################
# Server Checker programme
# PJFC 10/2001
###################################################
# Check date stamp on backup log
###################################################
cd /var/sam/log
a=`date | awk '{print $3}'`
b=`ll br_log | awk '{print $7}'`
if [ $a != $b ]
then
mailx -s "Check N0's Backup log" paula@avro.co.uk mailx -s "Check N0's Backup log" ltoliver@cosmos-uk.co.uk fi
###################################################
# Check Backup status
###################################################
c=`tail -n 1 br_log | awk '{print $4}'`
###################################################
# 2 = failed backup
if [ $c = 2 ]
then
mailx -s "Failed Backup on N0 -CHECK IT" paula@avro.co.uk mailx -s "Failed Backup on N0 -CHECK IT" ltoliver@cosmos-uk.co.uk fi
###################################################
# 4 = backup with errors
if [ $c = 4 ]
then
mailx -s "Backup with errors on N0-Check file locking" paula@avro.co.uk mailx -s "Backup with errors on N0-Check file locking" ltoliver@cosmos-uk.co.uk fi
###################################################
# 2 = backup with errors
if [ $c = 0 ]
then
mailx -s "Backup OK on N0" paula@avro.co.uk mailx -s "Backup OK on N0" ltoliver@cosmos-uk.co.uk fi
###################################################
# Get the bdf info
###################################################
echo "BDF INFO " >/tmp/morncheck
echo " " >>/tmp/morncheck
bdf >> /tmp/morncheck
echo " " >>/tmp/morncheck
echo " " >>/tmp/morncheck
###################################################
# Get the Lanscan info
###################################################
echo "LANSCAN INFO " >>/tmp/morncheck
echo " " >>/tmp/morncheck
/sysadmin/lanstat >> /tmp/morncheck
echo " " >>/tmp/morncheck
echo " " >>/tmp/morncheck
###################################################
# Get Phanton status
###################################################
echo "PHANTOM STATUS " >>/tmp/morncheck
echo " " >>/tmp/morncheck
ps -ef | grep phant >> /tmp/morncheck
echo " " >>/tmp/morncheck
echo " " >>/tmp/morncheck
###################################################
# Send Security check status
###################################################
echo "SECURITY CHECK INFO " >>/tmp/morncheck
echo " " >>/tmp/morncheck
cat /tmp/ftp_temp >> /tmp/morncheck
echo " " >>/tmp/morncheck
echo " " >>/tmp/morncheck
###################################################
echo "ROOT USERS " >>/tmp/morncheck
echo " " >>/tmp/morncheck
cat /sysadmin/rootalert >> /tmp/morncheck
echo " " >>/tmp/morncheck
echo " " >>/tmp/morncheck
###################################################
echo "WHO IS ON THE SYSTEM " >>/tmp/morncheck
echo " " >>/tmp/morncheck
who -u | sort >> /tmp/morncheck
echo " " >>/tmp/morncheck
echo " " >>/tmp/morncheck
###################################################
echo "SERVER UPTIME AND LOAD LEVEL " >>/tmp/morncheck
echo " " >>/tmp/morncheck
uptime >> /tmp/morncheck
echo " " >>/tmp/morncheck
echo " " >>/tmp/morncheck
mailx -s "N0 MORNING STATUS" paula@avro.co.uk
-------------------cut here-------------------

The attached script checks to see if the tape is in prior to backup. Cron both to suit.



HTH

Paula

[Paula J Frazer-Campbell]

Hi
forgot other script.

----------------cut------------------------
#!/bin/sh
#############################################
# Check for tape in drive
# PJFC 2002
#############################################
# Check if tape is in Drive (set for correct dev file)
if /usr/bin/mt -t /dev/rmt/0m rew 2>/dev/null ; then
#############################################
echo "----------------------------" >>/tmp/tape-log
date >> /tmp/tape-log
echo "------ Tape in ------" >>/tmp/tape-log
else
#############################################
# Inform if no tape in drive
echo "----------------------------" >>/tmp/tape-log
date >> /tmp/tape-log
echo "------ TAPE NOT LOADED -----" >>/tmp/tape-log
mailx -s "Tape not mounted on N0" paula@avro.co.uk mailx -s "Tape not in on N0" helpdesk@avro.co.uk exit
fi
----------------cut------------------------

Paula

[MAD_2]

Thanks Paula.

However, Steven, Jack, and most of the folks who replied gave a series of tools that actually get the results I am looking for. Unfortunately, only one, I think, is free (OpenView from HP, sarcheck, bb4-Big Brother, MRTG, RDDTOOLS, etc.). The only tool I found that is free and I may be able to adapt is MRTG, along with the associated default Unix tools.

We already have something similar to the script you are giving me, but I really appreciate it, mostly we use it to monitor successful backup completions. If our backups fail, then we get paged (this includes if tape writing fails). Our script also provides us with the basic information about backup start, end, blocks used, list of files backed up, bdf, etc. I think your script is mostly to give you a first start (morning or beginning of day) status of your system (a way for you to ask your server: how are things going my friend?)

What I am looking for is a way to page us if system performance drops below set params, like let's say average idle processes (sar check) drop to zero for a while, any filesystem space gets above 98% usage, or something like that when we are not around, so that someone can react before we get a call letting us know that things are not working. We want to get an alert prior to that point.

Thanks for sharing your script, I will keep it in my archives.