Tools To help Manage HP-UX Environment

[Michael Gretton]

Fellow Admins:

I need help. I have about 12 hp-ux servers (10.20 and 11). I am finding it more and more time consuming checking things every morning such as disk space (sometimes it fills up fast), daemons running (such as bind), applications working (such as oracle) and etc. I am being asked by "management" to make sure that everything stays operational and to be alerted when things go arwy. What I am looking for is some ideas or a product that I can use (freeware, shareware or commercial) tha is easy to set up that can contact me via pager or an email when certain things go out of wack. I do have openview but it looks like it takes a lot of time to get into and to set up. Any ideas? Thoughts?

Thanks,

Mike

[Patrick Wallek]

I have a script that runs every morning on my HPs and it e-mails me bdf output, dmesg, a tail of syslog.log and a couple of other things. I also have a script that runs from 6am-11pm or so that checks disk space every 15 minutes and if it reaches 99% then it pages me. (there was a thread on this in the last couple of days that had some good scripts).

Just about anything can be scripted if you have the time to play with it.

You are right about NNM. It is a pain to set up and get configured just the way you want it.

[Michael Gretton]

Patrick:

Could I get a cy of your script that does the disk checking and then paging? i am not a perl or shell script miester but i sure could use a starting point. Plus I could tweak the script to suit my needs easier than writing it all together. The other script(s) you mentioned would be usefull too. If you want you can email them to me: grettm@osd.pentagon.mil.

Thanks,

Mike

[Rick Garland]

There are several tools available from the porting center that are useful for keeping tabs on the systems as well. These tools have been ported for HPUX and they come in source form as well as depot format.

[Patricia C. Witten]

Patrick, could you share your script?

Patricia

[Brian Markus]

Here is a script I wrote a while back I have running in cron. I set a threshold on various mount points. In this script if oracle gets above 95% full it sends a msg to the console. I have my staff monitor the system console. It would be easy to send this output to email. useing the mail or mailx commands.
script code | mailx -s 'SUBJECT check your system' user@mail.domain.com


#####################################################################
# Script name: oraclenotify
# Script purpose: place a notification message on system console
# Written by: Brian Markus -
# bmarkus@rcoe.k12.ca.us
# Date: 4/16/1999
#####################################################################

percent=`bdf | grep -i oracle\$ | awk ' { print $5 } ' | sed 's/\%//'`
if [ ${percent} -gt 95 ]
then
banner "/oracle is" "at ${percent}%" "Notify DBA" > /dev/console
else
echo "/oracle is at ${percent}%" > /dev/console
fi

[Steven Sim Kok Leong]

Hi,

You may also like to use the logger command to send alerts to a central station's syslog daemon so that you have all the alerts at a centralised location.

To email or send an alert, simply interface your log file on the centralised server with your email script or your comms software such as kermit to perform the necessary email or paging alert respectively.

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[William C. Hermon]

Big Brother is shareware (free) and not as tough to figure out as NNM. I monitor about 20 servers, mixed archetecture, including hp-ux, sun solaris, and WinNT servers with it, with minimal problems. Check it out at http://bb4.com, it includes a pretty web interface and keeps history files that can be used to monitor uptime.

[Patrick Wallek]

OK Y'all here is the script that I run to check disk space every 15 minutes.

Here is some documentation for the script:

The
2nd line from the bottom (/usr/telalert....) is what pages me. We have
a paging system called telalert that can send out text pages to our
alphanumeric pagers. If you wanted to send an e-mail to your pager or
to where ever, then you can just change that command line to call
whatever program you want.

The awk portion of the program will look at all lines of a 'bdf -l' (I
use the -l so that it will only see local LVs and not NFS mount points)
except the first line head and the cdrom if it is mounted. I had to do
the if statement in the because I have some LVs that have rather long
names and the bdf output for the one LV is on two lines like the
following.

And here is the sript:

#!/bin/ksh
H=`hostname`
fs=`bdf -l | awk '
/%/ && ($6 !~ /cdrom/ ) && ($1 !~ /Filesystem/ ){
if ( $1 > 0 )
{ split($4,perc,"%"); if (perc[1]>=99) {print h": " $5 " at " $4} }
else
{ split($5,perc,"%"); if (perc[1]>=99) {print h": " $6 " at "$5} }
}
' h=$H`
if [[ ! -z $fs ]]
then
# echo $fs
/usr/telalert/telalertc -host hp_ebs -i WallekTextPager -m $fs
fi

[Patrick Wallek]

Attached to this message is the script that I at 6AM every morning (Monday-Friday) via cron. I have it setup in cron
to run the script and pipe the output to mailx to e-mail the info to me.

The first thing this scipt does is check all my volume groups to make
sure all mirrors are synced. I use MirrorDisk/UX quite extensively and
if I lose a disk it can be difficult to catch sometimes. I do the
vgdisplay and grep for stale so that if it comes back with anything,
then I know I've got problems. I think everything else is pretty
straightforward.

[Tim Malnati]

Big Brother is an excellent product for the price (free). Most of the bread and butter items are already part of the standard monitors. There is also a lot of contributed BB monitors on their ftp site for things not so bread and butter. If anything is needed beyond that, you can have the originator write something (for a price) or you can write a module to do it yourself. Interfacing to this monitoring system is a piece of cake. You will need a C compiler to install it; either the HPUX add-on compiler or gcc are fine. Take the time to get fully up to speed with how it works.

Disk space issues are a concern. This suggests that your DBA's are not communicating their needs with you. If you find that you are cleaning the same areas continually, you probably need to automate the purging of these areas.

[Tom Gore]

Patrick Wallek:
I am a new player to UNIX (two weeks of HP classes and some on-the-job-training). I too have been looking for something like this. I was playing with using "cut" and a bunch of other game playing. Your solution looks much easier. When I tried your script I keep getting "bdf4.sh[4]: Syntax error at line 5 : `{' is not expected. ". I have checked and double-checked my keying of your script. Where did I go wrong?

[Patrick Wallek · Accepted solution]

Tom,

I can't tell you exactly where you went wrong without seeing the script, but I suspect that there is something in the if statement. There are some requirements for spaces in some of the statements. Did you try just cutting and pasting out of the browser? Post what you have done and I'll take a look at it.

[Tom Gore]

Patrick,

Here is the script (in an attachment). Thanks for your help.

[Tom Gore]

Patrick,

It looks like the attachment didn't take. I guess I had to "cut & paste".

# bdf4.sh
H=`hostname`
fs='bdf -l | awk '
/%/ && ($6 !~ /cdrom/ ) && ($1 !~ /Filesystem/ ){
if ( $1 > 0 )
{ split($4,perc,"%"); if (perc[1]>=90) {print h": " $5 " at " $4} }
else
{ split($4,perc,"%"); if (perc[1]>=90) {print h": " $6 " at "$5} }
}
'h=$h'
if [[ ! -z $fs ]]
then
echo $fs
fi
exit

[Patrick Wallek]

Tom,

OK.... I see several things that will make a difference.

1) Put #!/bin/ksh as the first line of your script so that you are sure you are using the ksh.

2) The second split statement should be split($5 instead of $4

3) Towards the end of the script you have 'h=$h' It should be:
' h=$H ` With the back tick ` at the end of the line the finish off the bdf command statement in the fs variable.

Hope this helps.

[Tom Gore]

Patrick,

Thanks, that did the trick. Actually, I just did a "cut & paste" from your original.

ps-I thought a "#" in the first position flagged the line as a comment.

[Patrick Wallek]

Normally a # in the first position does mean a comment, BUT when it has the syntax #!/usr/bin/ksh or something similar, it is telling the script what shell to execute the script in. In this case it is telling the script to use the ksh.