How to measure the server's availability ?

[Augusto Vinhaes]

Hi all !

I need to measure the server's availability from the operating system view and I'd like to know if someone has done it. Is there an hp-ux tool to collect it ?
I was thinking about to write a script launched through the inittab file to collect some data.
Any idea will be appreciated.

Regards,
Augusto - Intelig Telecomunicacoes

[Pete Randall]

What do you mean by "the server's availability from the operating system view"? To me, the operating system is the server. Do you mean is it available on the network, is it up and running, is the database up?

Could you clarify?


Pete

[Augusto Vinhaes]

Hi Pete !

I'd like to measure if the server is available to the end users (if it's up and running).
I've already written some scripts to collect the database avalability and now I'm thinking on how to do it for the unix servers.

Regards,
Augusto

[Arockia Jegan]

You can have a script with ping command and run that script from another server that has the access to the server that needed to be montiored..

I use TNG monitoring to monitor the server.

[Augusto Vinhaes]

What's TNG ? Could you explain, please ?

The problem I see is that some times the ping works, but it's not possibly to access the server via telnet. I'm searching for a more accurate way...

Regards,
Augusto

[Pete Randall]

Augusto,

I would suggest a cron job that runs every minute or every 10 minutes, but you want notification after the server is no longer available so it's going to be hard for the server to send an email at that point. How about using the ping idea periodically from an other server and notifying you if the ping fails?


Pete

[Pete Randall]

Augusto,

By the way, I do have such a monitor on my systems: it's called "users". The minute anything is even slightly delayed, they start screaming.

I just listen for the screams.


Pete

[twang]

Run this script every hour using crontab:
-----------
#!/bin/sh

HOSTLST=/tmp/svr.lst
TMP_FILE=/tmp/svr.tmp
MAILLST="root, sysadm"

for host in $(cat $HOSTLST)
do
ping $host -n 1 | grep -q '1 packets received'
if [ $? = 0 ]
then
echo "$host: OK" >> $TMP_FILE
else
echo "$host: FAIL" >> $TMP_FILE
fi
done
if [ `grep FAIL $TMP_FILE |wc -l` -gt 0 ]; then
mailx -s "Server Availability Report" $MAILLST < $TMP_FILE
fi
exit 0
-----------

svr.lst contains hostname list, eg:
svr_1
svr_2

[Stuart Abramson_2]

We run this HP software:

1. GlancePlusPak:
B3701AA GlancePlus Pak 2000 (includes MeasureWare)
2, PerfView:
B7407AA OV VP Performance Manager
B7408AA OV VP Performance Monitor

Glance shows me how my system is running.

MeasureWare collects performance data over time.

PerfView produces graphs of system performance, in color, suitable for presentation to management:

cpu utilization over 24 hours over the week/month/year.

memory utilization over 24 hours over the week/month/year.

disk utilization, network utilization, etc.

My advice is, if you are not running these products, you should. They are powerful performance tools, and they come in handy to debug problems.

[Augusto Vinhaes]

Hi !

Perfview, glance, measureware are tools to evaluate performance. Are they able to collect availability ?

Regards,
Augusto

[Pete Randall]

Augusto,

No, those tools don't really offer what you seek. If you were monitoring them and they stopped, then you would know. What you're looking for is an OpenView product which used to be called Operations Center (it's now ITO or some such thing) - it monitors machines across the network and is capable of reporting on a wide variety of conditions. It's also quite expensive.


Pete

[Stuart Abramson_2]

If you want to be paged in the middle of the night when the server goes down, then ITO is for you.

If you want to say at the end of the month, that the server was up 97% of the time, then you can get that kind of data from PerfView.

But what you really want to know is, that my memory utilization for the last six months, has averaged 81%, 83%, 88%, etc, and climbing, and I should start looking at finding out why, and maybe buying more memory. That's what PerfView can tell you.

You can write little scripts all day, if it makes you happy, but Industrial Strength performance monitoring is done through the use of PerfView/MeasureWare, etc. Spend some money and do the job right.

[Augusto Vinhaes]

Good replay ..

I've got some Openview products (NNM and VPO). I will check how can I use then to collect and analyze the availability data


Thanks

[Elena Leontieva]

Augusto,

Unicenter TNG is the enterprise management product from Computer Associates (CA) and it consists of Real World Interface, Common Object Repository ( the central storage mechansm of all components), Managers (facilities that provide resource management ), and Agents. In our datacenter we tried to implement some parts of it and were not successfull enough I guess. So, I am in a process of removing Agents and Workload from the Unix servers.

Elena.

[Martin Johnson]

Augusto,

I'll have to differ with Pete. Glance and MeasureWare (MWA) can give you availability from an OS point of view. It cannot give you availability from an application point of view (we use the Precise tools suite for that - very expensive).

Pete's concern it that MWA is not 100% accurate because MWA can go down and you lose your OS monitoring. I have MWA running on 200+ servers with an uptime of MWA 99%+.

The real question is how accurate do you wnat to be and how much are you willing to pay for it. MWA should meet your needs. And, as mentioned before, using Perfview you can make pretty graphs for management.

HTH
Marty

[Pete Randall]

Marty,

Glance, Measureware, et all, run locally. What Augusto is looking for is the ability to know if the machine is alive and well. A tool the runs locally will die when the machine does. You've got to monitor remotely.


Pete

[Augusto Vinhaes]

Stuart,

it seems that it's easy to do it via Perfview. Could you explain how can I get the availability information from Perfview ?
Is it necessary any special configuration ?

Regards,
augusto

[Augusto Vinhaes]

I've got ITO that will send an alarm when the server goes down. What I really want is a tool, that runs locally or not, able to collect and provide availability graphs. For example, I'd like to know the server's availability during the first week of the month
.

Regards,
Augusto

[Pete Randall]

I stand corrected. It seems the original goal has changed somewhat.


Pete

[Chris Vail · Accepted solution]

I concur with the others who recommend Measureware and Perfview to collect availability and performance statistics, working with OpenView. We do this here on more than 300 platforms. We've developed a web page on our intranet where management can look at current and previous availability numbers (all 99.9's and better for Unix, less than perfect for Windows, but this expected). This is part of our monitoring for SLA (Service Level Agreement) compliance.
This is a REALLY expensive solution, its time consuming and a genuine, certified hassle. We have a developer dedicated full time to building and maintaining this. We've spent many 100's of thousands of US Dollars doing it too. But its really the best solution to the issue.

Barring that, you could do what we do in parallel to this. We built a series of scripts that collect uptime reports from each system on a regular basis. The following example uses an external list of hosts, and uses secure shell to run an uptime report on them. This number is accumulated, and the result is emailed.

#!/bin/ksh
#
# Upandrunning
# Jan 03
# C.Vail
# Check to see that each of several hosts are up and running.
# They won't be if they don't respond to an ssh command.

FFILE=/root/work/disks/hphosts.txt # where the list of hosts is
PPATH=/root/work/up
OFILE=$PPATH/upandrunning.txt # Output file
MFILE=$PPATH/upandrunning.mail # List of email addys
NUM=1

if test -f $OFILE
then
rm $OFILE
fi

for HOST in `cat $FFILE`
do
REM_UP=`ssh $HOST "uptime"|awk '{ print $3 }'`
if test "$?" -eq "0"
then
echo "$HOST is up and running $REM_UP days.">>$OFILE
else
echo "$HOST is NOT up and running, update your resume'." >> $OFI
LE
fi
(( CUM_UP = "$CUM_UP" + "$REM_UP" ))
(( NUM = "$NUM" + "1" ))
done


NUM=`echo "$NUM - 1"|bc`
AVERAGE=`echo "$CUM_UP / $NUM"|bc`
echo "$NUM hosts, Total $CUM_UP days, Average $AVERAGE days">>$OFILE


if test -f $MFILE
then
for ADDY in `cat $MFILE`
do
cat $OFILE|mailx -s "Running Hosts" $ADDY
done
fi

[Martin Johnson]

Pete,

You didn't see Augusto's post in the ITO forum. I knew he could monitor MWA via ITO.


:-)
Marty

[Joseph Bague]

You can try nagios

http://www.nagios.org

Hope it help

Joseph

[Augusto Vinhaes]

Hi !

I agree with Chris Vail when she says that it's necessary some effort to have availability graphs in a web page. This is exactly what I want, but I don't if I have money to develop this solution. I'd like to thank everybody involved in this question.

Best regards,

Augusto - Intelig Telecomunicacoes (Brasil)