Script Needed

[Allanm]

Hi!

I have a file which has the following entry -

NSRService
http://someurl.com:port1/....
TSRService
http://someurl2.com:port2/...

which has different service names with their corresponding URLs.

I want to echo the servicenames and curl each URL and find out if the URL is accessible or not.

Thanks,
Allan.

[Matti_Kurkela · Accepted solution]

Something like this?

---------------------------
#!/bin/sh

RESULT=0

# read all input from the specified file
exec <"$1"

while true
do
read SERVICE
if [ "$SERVICE" = "" ]
then
# no more services
exit $RESULT
fi

read URL
if [ "$URL" = "" ]
then
# error in service list file
echo "ERROR: no URL for service $SERVICE" >&2
exit 2
fi

echo "Testing $SERVICE: "

# adjust the options to suit your needs
curl -I $URL

if [ $? -eq 0 ]
then
echo "Success"
else
echo "Fail"
RESULT=1
fi
done
------------------------------

Save the script to a file, then run with e.g.

sh scriptname.sh servicelist.txt

NOTE: the script assumes there are no blank lines in the service list file.

The script returns exit code 0 if all services were accessible, 1 if at least one curl command failed, and 2 if there is an error in the service list file.

MK

[Allanm]

Hi All,

I am reopening this thread the curl that I run against the URL returns a response (has a text called alive in the return ), so I need to validate that I do see the text alive in the page before we say the URL is working else send an email saying that this service failed.

Thanks,
Allan.

[Steven Schweda]

man grep

> Script Needed

Are you asking for help in writing a script,
or do you simply want someone else to do your
whole job for you? What have _you_ done?

[Allanm]

This is what I had done before -

#!/bin/bash

CONFFILE=/tmp/url1ist

read URL < "$CONFFILE"

echo "Fetching $URL..."
curl "$URL"

case "$?" in
0)
echo "Service is ok"
exit 0
;;
1)
echo "WARN"
exit 1
;;
2)
echo "ERROR OR FATAL"
exit 2
;;
esac

[Steven Schweda]

> #!/bin/bash

Why not something simple, like, say,
"/bin/sh" (which is available everywhere)?
Are you using exotic bash-only features?

> curl "$URL"

If you send the output to a file, ...

curl "$URL" > out_file
or:
curl -o out_file "$URL"

then you could look at the exit status from
a command like, say:

grep alive out_file
or:
grep alive out_file > /dev/null


man grep
curl -h

You could also use one big pipeline:

curl "$URL" | grep alive [> /dev/null]

but that makes it harder to look at the exit
status values of the individual programs (if
you wish to do that). (I'd start with
separate commands, and, perhaps, try to
combine things later.)

[Allanm]

Thanks Steve, I really want to avoid the use of file to save data temporarily.

I tried this as well (which works)-

curl -s http://URL|perl -0ne '$x=/"alive"/?"OK":"FAIL";print "$x\n"'

But want a script in perl or shell and not a combination of the two. If I get an error then it should send me an email.

Thanks,
Allan.

[Steven Schweda]

> You could also use one big pipeline:
>
> curl "$URL" | grep alive [/dev/null]

Still true. Did you try that? Doesn't "$?"
work with that?

> [...] If I get an error then it should send
> me an email.

Is that a problem? A Forum search for
keywords like, say:

script send e-mail mailx

should find many examples of scripts which do
that.

> [...] I really want to avoid the use of
> file to save data temporarily.

Ok, but why? Sometimes that's the easy way.
Especially when one is composing an e-mail
message. There are, of course, good ways and
bad ways to use a temporary file. Including
the process ID in the file name can help to
avoid problems when two people try to do
something at the same time, for example. As
usual, many things are possible:

alp$ cat td.sh
#!/bin/sh

TMPDIR=${TMPDIR:-/tmp}
base_name=` basename $0 `

temp_file_1_name="${TMPDIR}/${base_name}_1_$$.tmp"

echo "t_f_n_1: ${temp_file_1_name}"


alp$ ./td.sh
t_f_n_1: /tmp/td.sh_1_538971991.tmp

[Steven Schweda]

One other nice thing about a temporary file
is that it can provide some clues when
something goes wrong in your script.
Normally, when a script creates any temporary
files, it would delete them before it exits.
If the script behaves badly, and explodes
before it can exit normally, then it would
leave its temporary files lying around, where
you can examine them. Just finding an
unexpected temporary file tells you that
something went wrong. Then, the file's
contents may tell you more.

[Mel Burslan]

I am not very familiar with usage of curl but did you consider using wget instead ? I believe curl has the capability of form submissions or similar over wget, but if all you are interested in is to capture a string saying "alive" in the output, wget may do the job for you.

wget $URL | grep -q -i alive; r=${?}
if [ ${r} -eq 0 ]
then
echo "System is alive : `date`" >> $MYLOGFILE #if needed
else
echo "$URL is not ALIVE" | sendmail me@mydomain.com
fi

Is there something else you want, that I am missing ?

You can go fancy and use mailx instead of sendmail but it is a choice. I chose the simple one.

[Steven Schweda]

> I am not very familiar with usage of curl

curl -h

> but did you consider using wget instead ?

I also normally use "wget" for this sort of
thing, but I doubt that it would make much of
a difference.

> wget $URL | grep -q -i alive; r=${?}

On the other hand, I'd expect "wget" to save
its result in a file, unless I used its "-O"
option to persuade it to do otherwise, so I
wouldn't expect ever to see "alive" flowing
through this particular pipeline.

How carefully did you test this suggestion?

> You can go fancy and use mailx instead of
> sendmail but it is a choice. I chose the
> simple one.

Define "simple". I prefer an e-mail message
with a Subject. "mailx" makes _that_ simple.

[James R. Ferguson]

Hi Allan:

> But want a script in perl or shell and not a combination of the two.

So, Matti, Steven and Mel have already offered several variations using a shell.

The Perl snippet :

# curl -s http://${URL} | perl -0ne '$x=/"alive"/?"OK":"FAIL";print "$x\n"'

...is simply reading the pipe output as one, big record. It then simply reports the presence or absence of a string '"alive"'.

What's wrong with using this piece of Perl in a shell script? How is that different from saying a shell script with an 'awk' or 'grep' command to perform matching isn't a shell script? Where you draw the line is somewhat a matter of taste. If I had a shell script with many Perl commandline scripts embedded within, I would certainly consider re-writing it purely in Perl, but for performing the task as you have stated it, I'd do what is ever comfortable to you.

Regards!

...JRF...

[Allanm]

Thanks for your replies, one more thing, due to the sake of false alerting (flapping) can we build something like "4 times service failure checks with 20 seconds intervals before raise an alarm"?

Thanks,
Allan

[James R. Ferguson]

Hi (again) Allan:

> one more thing, due to the sake of false alerting (flapping) can we build something like "4 times service failure checks with 20 seconds intervals before raise an alarm"?

Are you using Nagios? If so, then you should be able to add this protection with relative ease.

Regards!

...JRF...

[Allanm]

Hi JRF,

This is a contingency for Nagios, as its down currently.

Thanks,
Allan.

[James R. Ferguson]

HI (again) Allan:

> This is a contingency for Nagios, as its down currently.

I would think that Nagios would be of more importance given that is probably supports a great deal more things then just the subject of this query (?)

A crude suggestion would be to test the viability of each URL several times with a wait (sleep) in between. Then if there are more than some number of connection failures, per URL, consider it unavailable. At least in this fashion, one failure would not spell doom.

Regards!

...JRF...

[Allanm]

Hi JRF!

This is as far as I have been able to go -

#!/bin/sh

OP=0

exec <"/fs1/ainuser/ain/user-1/list"

while true
do
read APP
if [ "$APP" = "" ]
then
exit $OP
fi

read URL
if [ "$URL" = "" ]
then
echo "ERROR no URL defined for $APP" >&2
exit 2
fi
v=1
while ( $v -le 6 )
do
sleep 3

echo "Checking $APP: "
curl $URL |grep -q -i alive
x=$?
if [ ${x} -eq 0 ]
then
echo "App is alive : `date`"
else
echo "$APP is down in env-1" | mail -s "$APP is down in env-1" testalert@email.com
fi
v=$(( $v + 1 ))
done
done

Now I want some help on how to go about storing the curl results and checking to see if it falied atleast 3 times before I send an alert email.

Thanks,
Allan

[James R. Ferguson]

Hi (again) Allan:

> Now I want some help on how to go about storing the curl results and checking to see if it falied atleast 3 times before I send an alert email.

You shouldn't need to store the 'curl' results. All this is necessary is to count the number of successes of failures. For instance, if you sample 10-times, quit after the allowed number of failures (e.g. 5) is exceeded. At the end of either 10-samples or a failure threshold exceeded, exit the enclosing loop; test for the failures counted and send an email appropriately.

Regards!

...JRF...

[Allanm]

I tried here , please review and modify-

#!/bin/sh

OP=0

exec <"/fs1/ainuser/ain/user-1/list"

while true
do
read APP
if [ "$APP" = "" ]
then
exit $OP
fi

read URL
if [ "$URL" = "" ]
then
echo "ERROR no URL defined for $APP" >&2
exit 2
fi
v=1
while ( $v -le 4 )
do
sleep 3

echo "Checking $APP: "
curl $URL |grep -q -i alive
y=0
x=$?
if [ ${x} -eq 0 -a ${y} -eq 4 ]
then
echo "App is alive : `date`"
else
echo "$APP is down in env-1" | mail -s "$APP is down in env-1" testalert@email.com
y=$(( $y + 1 ))
fi
v=$(( $v + 1 ))
done
done

[Allanm]

I have found few issues, fixing those, specially to do with the place of $? in the script.

[Allanm]

Can you please review it, I have gotten this far but I would need help -

#!/bin/sh

OP=0

exec <"/fs1/ainuser/ain/user-1/list"

while true
do
read APP
if [ "$APP" = "" ]
then
exit $OP
fi

read URL
if [ "$URL" = "" ]
then
echo "ERROR no URL defined for $APP" >&2
exit 2
fi
v=1
while ( $v -le 4 )
do
sleep 3

echo "Checking $APP: "
curl $URL |grep -q -i alive
x=$?
y=0
if [ ${x} -eq 0 -a ${y} -eq 5 ]
then
echo "App is alive"
else
echo "$APP is down in env-1" | mail -s "$APP is down in env-1" testalert@email.com
y=$(( $y + 1 ))
fi
v=$(( $v + 1 ))
done
done

[Steven Schweda]

> I tried here , please review and modify-

For me, it fails here:

exec <"/fs1/ainuser/ain/user-1/list"

> Can you please review it, [...]

Can't you review it? Does it work? What
fails? Where is the mystery?

[James R. Ferguson]

Hi (again) Allan:

I was thinking more alone these lines [ untested ]:

# cat ./mymonitor
#!/bin/sh

typeset -i CYCLENBR=0
typeset -i FAILURES=0

typeset MYLIST="/fs1/ainuser/ain/user-1/list"

while true
do
read APP
if [ "$APP" = "" ]; then
exit 0
fi

read URL
if [ "$URL" = "" ]; then
echo "ERROR no URL defined for ${APP}"
exit 2
fi

CYCLENBR=0
FAILURES=0

while (( ${CYCLENBR} < 4 ))
do
sleep 3
echo "Checking ${APP}: "
CYCLENBR=$(( CYCLENBR + 1 ))

curl ${URL} | grep -q -i alive
[ $? -eq 0 ] && continue

FAILURES=$(( FAILURES + 1 ))
done

if [ ${FAILURES} -ge 0 ]; then #...adjust to your need...
echo "${APP} _may_ be down in env-1" | mail -s "${APP} is down in env-1" testalert@email.com
fi

done < ${MYLIST}

exit 0

Regards!

...JRF...