Shell Script for checking webserver pages are working or not

[Shivkumar]

Hi,

I am looking out for a shell script to check whether a web server is serving the page correctly or not.
One of the option is to check whether "get" is returning successful authentication status code 200 or not.

Any other idea is welcome.

Appreciate your help.

Thanks,
Shiv

[James R. Ferguson]

Hi SHiv:

How about 'wget':

http://hpux.cs.utah.edu/hppd/hpux/Gnu/wget-1.10.2/

Regards!

...JRF...

[James R. Ferguson]

Hi (again) Shiv:

By the way, to learn more about 'wget', see:

http://www.gnu.org/software/wget/manual/wget.html

Regards!

...JRF...

[Steven Schweda]

Also:

http://wget.sunsite.dk/

(I do this on my VMS system with wget.)

[Shivkumar]

How to put it in a shell script ?

Thanks,
Shiv

[Steven Schweda · Accepted solution]

How would you put any command into a shell
script? I use a text editor.

Wget's normal output to stdout includes a line like this:

HTTP request sent, awaiting response... 200 OK

You should be able to save it in a temporary
file for later examination, or else pipe it
through "grep" to see if the "200" appeared
as expected.

If you want more details, you might need to
specify the shell you wish to use. If
you've never written a shell script before,
that would also be useful information.

[Steven E. Protter]

Shalom,

I suggest you install a utility called wget

Its available from the porting center and should be compiled not installed from depot because the depot install doesn't work well.

Then have your shell script do this:

wget http://website/index.html

Important pages.

Though this does not test how it works in a browser, it does attempt to download the page like a browser does and is a good quality control check.

SEP

[Shivkumar]

my question is:-

i want to use "
wget http://website/index.html" in a shell script.
i want to run this script via cron job so that whenever status code 200 is not returned it should send some email alert to a user.

I am not good in shell scripting and even if i start doing it would take longer than expected.

appreciate your help.

Regards,
Shiv

[Yang Qin_1]

#!/usr/bin/ksh

/path_to/wget http://website/index.html > /tmp/webchk.log

grep -i "HTTP request sent" /tmp/webchk.log |grep -i "200 OK"

if [ $? -ne 0 ] ; then
mailx -s "http://website/index.html is not OK" your_name@your.org.com < /tmp/webchk.log
fi

rm /tmp/webchk.log

exit


Yang

[James R. Ferguson]

Hi Shiv:

The best way to become proficient at something is to begin by *trying*. Here's a simple 'wget' session that verifies a list of web pages specified in 'wget.list'.

# cat wget.list
http://www.google.com
http://www.dummy.gov
http://www.hp.com

# cat wget.sh
typeset LOG=wget.log
typeset LST=wget.list
rm -f ${LOG}
wget --spider -a ${LOG} -i ${LST}
awk '/^Resolving/ {if (/failed/) {print $2,"bad"} else {print $2,"ok"}}' ${LOG}
exit 0

...If you run this (and you may need to change the shell interpreter to '/usr/bin/sh' on HP-UX) you will see the following output:

# ./wget.sh
www.google.com... ok
www.dummy.gov... bad
www.hp.com... ok

The link to the 'wget' documentation I provided will enable you to decipher the 'wget' logic. The shell script encapsulates the logic we need and should be easy to understand. I used 'awk' to extract the lines I wanted from the logfile created by 'wget'. 'awk' numbers fields in a line starting from one (1). By default, 'awk' splits a line into fields based on whitespace. The character strings bounded by "/" denote a regular expression match. By default, 'awk' attempts to match the bounded characters anywhere in each line it reads. Hence, in our example we matched lines in the logfile that began with "Resolving". Te caret ("^") in front of the "R" means anchor the string to a line's beginning. Thus, we are looking for lines that *begin* with "Resolving". The remainder of the 'awk' program should be easily deducible.

Regards!

...JRF...

[Shivkumar]

James, I appreciate your help.

As always best regards to you.
Shiv

[James R. Ferguson]

Hi Shiv:

I inadvertently dropped the shell shebang line --- the first line of a script that specifies the program (here, the shell) to use:

# cat wget.sh
#!/usr/bin/sh
typeset LOG=wget.log
typeset LST=wget.list
rm -f ${LOG}
wget --spider -a ${LOG} -i ${LST}
awk '/^Resolving/ {if (/failed/) {print $2,"bad"} else {print $2,"ok"}}' ${LOG}
exit 0

Regards!

...JRF...

[Shivkumar]

Thanks Yang and others!!

Regards,
Shiv

[rmueller58]

Your best bet would be to use LYNX rather then WGET as if you have robots.txt to disallow robots, WGET will be rejected but Lynx will not..

Use #lynx -mime_header

to get your 200 status code.

[Steven Schweda]

> [...] if you have robots.txt to disallow
> robots, WGET will be rejected [...]

Of course, that depends on what's in the
server's "robots.txt" file. Also, as
"wget -h" will tell you, the user may decide
how wget identifies itself (here, from "GNU
Wget 1.10.2c" on VMS):

-U, --user-agent=AGENT identify as AGENT instead of Wget/VERSION.

On the other hand, I'm sure that wget causes
cancer, so it's probably best to avoid it for
that reason, too.

[Shivkumar]

Someone told me to use openssl. Is it possible to use openssl ?

Thanks,
Shiv

[Steven Schweda]

> Someone told me to use openssl. Is it
> possible to use openssl ?

It probably is possible, but shouldn't you
be asking "How?" of the one who told you to
use it?

Wget can incorporate OpenSSL, which it uses
for a URL which begins "https://". All you
said in your original posting was "the page",
so everyone has been assuming that you meant
something simple, like "http://xxx...". If
you say more than "the page", you might get
better responses.

It's possible to do the job with Telnet, too,
but it's not simple, and if you are "not good
in shell scripting", then it's probably not
the easiest or best way to attack the
problem.

And besides wget, there's also "curl":

http://curl.haxx.se/

Either one should be able do the job
(depending on exactly what "the job" is, of
course).

[James R. Ferguson]

Hi Shiv:

I might add that if you drop the requirement for a *shell* script, the number of different solutions expand further. Depending upon your requirements, *Perl* scripts with the 'LWP::Simple' module would work, too.

Regards!

...JRF...

[Ralph Grothe]

If you have got the openssl suite installed on your box (HP offers a free bundle) then you could use the incredibly versatile openssl command for most things crypto.
e.g.
You can pull certificate info of an SSL/TLS enabled webserver by issuing something like
(I assume you have the webserver running on the same box, but any other reachable IP also will work)

$ openssl s_client -connect 127.0.0.1:443

However, this may block if the DocumentRoot is AuthType "protected" because the webserver is sending an Authentication request which your client has to answer.

If you have a recent Perl you could pull the header of any https website.
If it is AuthType protected you could use the
-C switch like below.
The HTTP response header received also contains entries about the SSL-Cert-Issuer (which I omitted below).
Most important the HTTP response status is sent in the first line (usually you would expect a 200 which is OK)

$ lwp-request -m head -C grothe:riddle https://127.0.0.1/|head -1
200 OK


You could also get the nagios-plugins, extract the tarball and only compile the check_tcp and check_http plugins.
They are specifically designed for monitoring purposes and offer a whole lot of options to virtually test any HTTP services.
When invoked with -h or --help you get a full screen where the options are explained.

Unfortunately the nagios server where I've got those check commands available is a legacy AIX system that neither has a /dev/urandom nor an entropy gathering daemon why I only get this nasty error

$ check_http -H somehost -N -S -a grothe:riddle
CRITICAL - Cannot make SSL connection
35196:error:24064064:random number generator:SSLEAY_RAND_BYTES:PRNG not seeded:md_rand.c:474:You
need to read the OpenSSL FAQ, http://www.openssl.org/support/faq.html
Error on receive

But HP-UX provides patches that would create those random devices to make openssl or other SSL libs happy.