topic Re: In need of a script - in Operating System - HP-UX

In need of a script -

Rob Johnson_3 — Fri, 14 Jul 2006 12:29:05 GMT

I have a number of CiscoWorks Campus Mgr servers that can send Syslog messages to 1 of 3 Syslog servers. I’m trying to find a way to grep through the /var/log/rtrlog file every hour and email any messages from this file that have :%CISCOWORKS-CampusManager and LINK_TRUNK in the message. Getting something to run every hour is the easy part (just cron it). Having it email me the output is easy as well.

I’m thinking the hard part is going to be not emailing messages that were emailed the previous hour.

Can someone help me put a script together that can do this? Oh yea, my scripting skills are very, very weak.

$ grep -i campus rtrlog|grep -i link_trunk
Jul 13 02:43:13 nmsccm05.kp.org 3124:2006 Jul 13 5:43:13 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcamosc1.oak.ca.kp.org,GigabitEthernet5/48],[10.236.45.4,6/24]
Jul 14 03:43:50 nmsccm06.kp.org 3683:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc133.str.ca.kp.org,GigabitEthernet0/1],[lcastrc1.str.ca.kp.org,3/4]
Jul 14 03:43:50 nmsccm06.kp.org 3685:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc121.str.ca.kp.org,GigabitEthernet0/1],[lcastrc1.str.ca.kp.org,3/3]
Jul 14 03:43:50 nmsccm06.kp.org 3687:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc118.str.ca.kp.org,GigabitEthernet0/2],[lcastrc2.str.ca.kp.org,3/2]
Jul 14 03:43:50 nmsccm06.kp.org 3688:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc151.str.ca.kp.org,GigabitEthernet0/1],[lcastrc1.str.ca.kp.org,3/5]
Jul 14 03:43:50 nmsccm06.kp.org 3690:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc154.str.ca.kp.org,GigabitEthernet0/2],[lcastrc2.str.ca.kp.org,3/5]
Jul 14 03:43:50 nmsccm06.kp.org 3692:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc139.str.ca.kp.org,GigabitEthernet0/2],[lcastrc2.str.ca.kp.org,3/7]
Jul 14 03:43:50 nmsccm06.kp.org 3693:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc124.str.ca.kp.org,GigabitEthernet0/2],[lcastrc2.str.ca.kp.org,3/3]
Jul 14 03:43:50 nmsccm06.kp.org 3694:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc1.str.ca.kp.org,3/8],[lcastrc143.str.ca.kp.org,GigabitEthernet0/1]
Jul 14 03:43:50 nmsccm06.kp.org 3695:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc116.str.ca.kp.org,GigabitEthernet0/1],[lcastrc1.str.ca.kp.org,3/2]
Jul 14 03:43:50 nmsccm06.kp.org 3696:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc135.str.ca.kp.org,GigabitEthernet0/2],[lcastrc2.str.ca.kp.org,3/4]
Jul 14 03:43:50 nmsccm06.kp.org 3700:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc138.str.ca.kp.org,GigabitEthernet0/1],[lcastrc1.str.ca.kp.org,3/7]
Jul 14 03:43:50 nmsccm06.kp.org 3701:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc2.str.ca.kp.org,3/8],[lcastrc142.str.ca.kp.org,GigabitEthernet0/2]
$

Re: In need of a script -

Victor Fridyev — Fri, 14 Jul 2006 13:09:13 GMT

Hi,

The simplest way, IMHO, is to copy log file onto another file after each cron running, i.e.

LOGF=var/log/rtrlog
BACKLOG=var/log/rtrlog.bck
MAILF=/tmp/$$
grep -e ":%CISCOWORKS-CampusManager -e LINK_TRUNK $LOGF > $MAILF
if [ -s $MAILF ]; then
cat $MAILF | mailx -s subject your@address
fi
cat $LOGF >> $BACKLOG
>$LOGF

By this way you begin with empty file after each the script running

HTH

Re: In need of a script -

Mark Fenton — Fri, 14 Jul 2006 14:26:38 GMT

Another approach that doesn't involve copying the logs:

#!/bin/sh

my_mon=`date +%b`
my_day=`date +%d`
my_hr=`date +%H`

if [ $my_day -lt 10 ]
then
my_date="$my_mon $my_day $my_hr"
else
my_date="$my_mon $my_day $my_hr"
fi
grep ^"${my_date}" rtrlog|grep -i cisco|grep -i link_trunk

inelegant, perhaps, but quick and easy to understand.

Re: In need of a script -

Mark Fenton — Fri, 14 Jul 2006 14:46:26 GMT

the pair my_date lines don't look quite right in proportional font -- there should be an extra space where the day of the month is less than 10.

Re: In need of a script -

Rob Johnson_3 — Fri, 14 Jul 2006 14:50:13 GMT

You guys are both awesome!!

I'm not able to get to either of the 3- servers at the moment but will let you know which route I choose to use and if I have any problems.

Again, Thanks so much!!!!!!!!!!!!!!!!!!!!!!!!!!

Re: In need of a script -

Rob Johnson_3 — Fri, 14 Jul 2006 14:59:05 GMT

I'm not sure I understand what your saying when you say

"the pair my_date lines don't look quite right"

Can you elaborate?

Re: In need of a script -

Mark Fenton — Fri, 14 Jul 2006 15:22:21 GMT

in the first line with my_date=...
there should be two spaces between ${my_mon} and ${my_day}, whereas in the second there should only be one.

I guess the posting algorhythm on this site removes "extra" spaces in some lines of text.

Re: In need of a script -

Rob Johnson_3 — Fri, 14 Jul 2006 15:30:19 GMT

Gotcha...

Thank you!!

'Not sure I quiet understand it but, like I said, I'm not programmed to be a programmer!

I'll be able to try this out later this afternoon or 1st thing tomorrow.

Re: In need of a script -

A. Clay Stephenson — Fri, 14 Jul 2006 15:50:20 GMT

I have a better idea and it doesn't involve copying files. Let's create a "watchfile" that consists of one 1 having 2 data values: SYSLOG_PID LAST_LINE

If that file is not found or if the PID of the current executing syslogd daemon doesn't match the stored value then we start the syslog.log scan at line 1 otherwise we start at LAST_LINE + 1.

At the end of our scan, we rewrite the "watchfile" and we are ready for the next invocation.
---------------------------------------
#!/usr/bin/sh

typeset TDIR=${TMPDIR:-/var/tmp}
typeset INFILE="/var/adm/syslog/syslog.log"
typeset WATCHFILE=${TDIR}/Syswatch.dat
typeset A1=${TDIR}/X${$}_1.awk

trap 'eval rm -f ${A1}' 0 1 2 3 15

get_syslogd_pid()
{
typeset -i PSTAT=0
typeset -i SPID=$(UNIX95= ps -Csyslogd -o pid=)
PSTAT=${?}
echo "${SPID}"
return ${PSTAT}
} # get_syslogd_pid

# if syslogd's PID does not match the stored value
# or the watchfile is not found or syslogd is not running
# then read syslog from beginning else start at stored line + 1

cat << !EOF! > ${A1}
{
while (NR < begin_line) next;
if (\$0 ~ "%CISCOWORKS-CampusManager.+LINK_TRUNK")
{
printf("%s\n",\$0)
}
}
END {
printf("%d %d\n",pid,NR) > wfile
}
!EOF!

typeset -i BEGIN_LINE=1

typeset -i OLDPID=-1
typeset -i OLD_LINE=0

typeset -i STAT=0
typeset -i CURRPID=$(get_syslogd_pid)
STAT=${?}
if [[ ${STAT} -eq 0 && ${CURRPID} -gt 0 ]]
then
if [[ -r "${WATCHFILE}" && -s "${WATCHFILE}" ]]
then
read OLDPID OLD_LINE < "${WATCHFILE}"
if [[ ${OLDPID} -eq ${CURRPID} ]]
then
BEGIN_LINE=$((${OLD_LINE} + 1))
fi
fi
fi

awk -v begin_line=${BEGIN_LINE} \
-v pid=${CURRPID} -v "wfile=${WATCHFILE}" -f "${A1}" "${INFILE}"
STAT=${?}
exit ${STAT}
------------------------------------------

All you have to do is capture the stdout of this script and send it to mail and you are done.

Re: In need of a script -

Rob Johnson_3 — Fri, 14 Jul 2006 16:22:39 GMT

I wasn't able to follow what A. Clay Stephenson is talking about. If I have to, I will try his method.

I tried the other method and looks like something is wrong.

*******************************************
$ cat syslogwatch.sh
#!/bin/sh
set -x
my_mon=`date +%b`
my_day=`date +%d`
my_hr=`date +%H`

if [ $my_day -lt 10 ]
then
my_date="$my_mon $my_day $my_hr"
else
my_date="$my_mon $my_day $my_hr"
fi
grep "${my_date}" /var/log/rtrlog|grep -i campus
$

Here's the output with set -x
$./syslogwatch.sh
+ date +%b
my_mon=Jul
+ date +%d
my_day=14
+ date +%H
my_hr=14
+ [ 14 -lt 10 ]
my_date=Jul 14 14
+ + grepgrep Jul 14 14 /var/log/rtrlog
-i campus
$

I know there is something in the file because...

$ grep -i campus /var/log/rtrlog
Jul 12 03:37:48 nmsccm06.kp.org 26:2006 Jul 12 6:37:48 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_MULTI:[10.246.222.50,GigabitEthernet0/2], [lcamanc2.ca.kp.org,8/13]
Jul 12 03:37:48 nmsccm06.kp.org 37:2006 Jul 12 6:37:48 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_MULTI:[lcamanc1.ca.kp.org,9/6], [lcamanc38-ter.ca.kp.org,GigabitEthernet0/1]

Re: In need of a script -

A. Clay Stephenson — Fri, 14 Jul 2006 16:32:05 GMT

It's rather simple, we create a little file that stores two pieces of data. The pid of syslogd and the line number where we finished our scan last time. We then run ps and determine the PID of the currently running syslogd. It's possible that someone has stopped syslogd, copied syslog to syslog.old, and restarted syslogd. This is typically done nitely. We want to detect that condition and if a new syslogd is running then we need to start reading from the beginning. If the same syslogd is still running as when we last executed the script then we should start at the next new line.
Most of the work is done by calling awk and the last thing awk does is rewrite the little file.

The nice feature of this approach is if no new matching lines are found, nothing is output to stdout.

Re: In need of a script -

Rob Johnson_3 — Fri, 14 Jul 2006 16:46:21 GMT

OK. I created a file called syslogwatch2.sh and put the contents of your post in it.

I made it executable. What else do I need to do?

Do I also need to create another file? If so, do I need to give it a certain name?

Re: In need of a script -

Rob Johnson_3 — Fri, 14 Jul 2006 16:48:35 GMT

Here's what happens when I try to run this one.

$ chmod +x syslogwatch2.sh
$ ./syslogwatch2.sh
./syslogwatch2.sh: typeset: not found
./syslogwatch2.sh: typeset: not found
./syslogwatch2.sh: typeset: not found
./syslogwatch2.sh: typeset: not found
./syslogwatch2.sh: syntax error at line 14: `(' unexpected
$

Re: In need of a script -

A. Clay Stephenson — Fri, 14 Jul 2006 17:05:56 GMT

It's rather simple, all you need now is a wrapper script that you call from cron. We need to do a little extra work because cron's PATH is very sparse.

--------------------------------
#!/usr/bin/sh

typeset TDIR=${TMPDIR:-/var/tmp}
typeset T1=${TDIR}/X${$}_1.txt
typeset -i STAT=0
typeset WHO="mickey@mouse.com donald@duck.com"

export PATH=${PATH}:/usr/bin:/usr/local/bin
# should include your directory containing
# the watchsyslog2.sh

# automatically remove temp file upon exit
# or receipt of a signal e.g. Ctrl-C
#
trap 'eval rm -f ${T1} 0 1 2 3 15'

watchdog2.sh > ${T1}
STAT=${?}
if [[ -s "${T1}" ]] # any output?
then
mailx -s "Network stuff" "${WHO}" < ${T1}
STAT=${?}
fi
exit ${STAT}
------------------------------------------

That should do it. The first time you run it you should get tons of mail; the next time only the new lines.

You do not need to create the /var/tmp/Syswatch.dat file because if not found, it assumes this is the first run. It will always rewite the file whether or not it existed before.

Re: In need of a script -

A. Clay Stephenson — Fri, 14 Jul 2006 17:17:21 GMT

Well, I thought surely there must be a typo in all of that stuff since I simply copy-and-pasted from a similar script but when I copied-and-pasted from my posting back to a UNIX box, it ran perfectly.

I did see one logical error in the wrapper script:
watchdog2.sh > ${T1}
should be
watchsyslog2.sh > ${T1} # or whatever you name the inner script.

If you copied and pasted and are getting a typeset not found error then something in "The Force" tells me you are not running the POSIX shell.

Re: In need of a script -

Rob Johnson_3 — Fri, 14 Jul 2006 17:23:28 GMT

Wow, I'm having a difficult time following you!!

It looks like I'm running the ksh:

$ echo $SHELL
/usr/bin/ksh
$

I have 2 scripts, the 1st is sylogwatch2.sh. the wrapper is called syslogwrapper.sh. Take a look. I'm sure there are a number of things wrong. Please advise.

$ pwd
/export/home/kv13588/scripts
$ ls
syslogwatch2.sh syslogwrapper.sh
$ cat syslogwatch2.sh
#!/usr/bin/sh

typeset TDIR=${TMPDIR:-/var/tmp}
typeset INFILE="/var/adm/syslog/syslog.log"
typeset WATCHFILE=${TDIR}/Syswatch.dat
typeset A1=${TDIR}/X${$}_1.awk

trap 'eval rm -f ${A1}' 0 1 2 3 15

get_syslogd_pid()
{
typeset -i PSTAT=0
typeset -i SPID=$(UNIX95= ps -Csyslogd -o pid=)
PSTAT=${?}
echo "${SPID}"
return ${PSTAT}
} # get_syslogd_pid

# if syslogd's PID does not match the stored value
# or the watchfile is not found or syslogd is not running
# then read syslog from beginning else start at stored line + 1

cat << !EOF! > ${A1}
{
while (NR < begin_line) next;
if (\$0 ~ "%CISCOWORKS-CampusManager.+LINK_TRUNK")
{
printf("%s\n",\$0)
}
}
END {
printf("%d %d\n",pid,NR) > wfile
}
!EOF!

typeset -i BEGIN_LINE=1

typeset -i OLDPID=-1
typeset -i OLD_LINE=0

typeset -i STAT=0
typeset -i CURRPID=$(get_syslogd_pid)
STAT=${?}
if [[ ${STAT} -eq 0 && ${CURRPID} -gt 0 ]]
then
if [[ -r "${WATCHFILE}" && -s "${WATCHFILE}" ]]
then
read OLDPID OLD_LINE < "${WATCHFILE}"
if [[ ${OLDPID} -eq ${CURRPID} ]]
then
BEGIN_LINE=$((${OLD_LINE} + 1))
fi
fi
fi

awk -v begin_line=${BEGIN_LINE} \
-v pid=${CURRPID} -v "wfile=${WATCHFILE}" -f "${A1}" "${INFILE}"
STAT=${?}
exit ${STAT}

$ cat syslogwrapper.sh
#!/usr/bin/sh

typeset TDIR=${TMPDIR:-/var/tmp}
typeset T1=${TDIR}/X${$}_1.txt
typeset -i STAT=0
typeset WHO="rob.a.johnson@verizon.com"

export PATH=${PATH}:/usr/bin:/usr/local/bin
# should include your directory containing
# the watchsyslog2.sh

# automatically remove temp file upon exit
# or receipt of a signal e.g. Ctrl-C
#
trap 'eval rm -f ${T1} 0 1 2 3 15'

watchdog2.sh > ${T1}
STAT=${?}
if [[ -s "${T1}" ]] # any output?
then
mailx -s "Network stuff" "${WHO}" < ${T1}
STAT=${?}
fi
exit ${STAT}
$

Re: In need of a script -

Rob Johnson_3 — Fri, 14 Jul 2006 17:40:37 GMT

Note:

syslogd is running
$ ps -ef|grep syslogd
root 156 1 0 Feb 06 ? 62:43 /usr/sbin/syslogd
kv13588 5122 4897 0 15:39:06 pts/1 0:00 grep syslogd

This is the location and file name of the file where the syslos messages are going to
$ ls /var/log/rtrlog
/var/log/rtrlog
$

Re: In need of a script -

A. Clay Stephenson — Fri, 14 Jul 2006 18:43:27 GMT

OK, I attached the inner script and modified it to use /var/log/rtrlog. As an attachment, it should be most difficult to inadvertently "improve".

If you modify this script and you see any lines that end with "\" make absolutely certain that nothing (including spaces) follows the backslash. This is a continuation and may be part of your syntax problems.

First rm /var/tmp/Syswatch.dat if it exists and then execute the attached script from the shell and see if you get any output.
It should create a new /var/tmp/Syswatch.dat file. Execute the script again and you should get no output.
At any time, you can rm the Syswatch.dat file and the script will forget all it knows about where it finished. Until you get the output of this script working then there is no need to go further. You should also check the Regular Expression matching in the awk script to see if you have any input that actually matches; that's this stuff (CISCOWORKS ...):

cat << !EOF! > ${A1}
{
while (NR < begin_line) next;
if (\$0 ~ "%CISCOWORKS-CampusManager.+LINK_TRUNK")
{
printf("%s\n",\$0)
}
}
END {
printf("%d %d\n",pid,NR) > wfile
}
!EOF!

When you finally get the inner script working then you can worry about the wrapper.

Re: In need of a script -

A. Clay Stephenson — Fri, 14 Jul 2006 19:14:54 GMT

By the way, I am making the silly assumption since you posted this in an HP-UX forum that this is actually running on an HP-UX box. If not, and (shebang) #!/usr/bin/sh doesn't actually invoke the POSIX shell then that explains much of your difficulty. Moreover, if this happens to be a Solaris box then you should use nawk rather than awk. You should probably change the shebang line to invoke the Korn shell if this isn't HP-UX.

Re: In need of a script -

Rob Johnson_3 — Fri, 14 Jul 2006 19:22:55 GMT

It looks like the box I'm trying to run this on is Solaris. I hope that's not causing the problem...

$ pwd
/export/home/kv13588/scripts
$ ls
pswatch.sh syslogwatch.sh syslogwatch2.sh syslogwrapper.sh
$ ls /var/tmp
112233-12.SUNWhea 113146-06.SUNWapchr
112963-18.SUNWtoo 116561-04.SUNWdmfex
112963-18.SUNWtoox install_it_Sun_StorEdge_SAN.log
113073-14.SUNWcarx restart_rsac.log
113073-14.SUNWcsr rsac-audit.log
113146-06.SUNWapchd
$ ./syslogwatch2.sh
./syslogwatch2.sh: typeset: not found
./syslogwatch2.sh: typeset: not found
./syslogwatch2.sh: typeset: not found
./syslogwatch2.sh: typeset: not found
./syslogwatch2.sh: syntax error at line 14: `(' unexpected
$ ls /var/tmp
112233-12.SUNWhea 113146-06.SUNWapchr
112963-18.SUNWtoo 116561-04.SUNWdmfex
112963-18.SUNWtoox install_it_Sun_StorEdge_SAN.log
113073-14.SUNWcarx restart_rsac.log
113073-14.SUNWcsr rsac-audit.log
113146-06.SUNWapchd
$ grep %CISCOWORKS-CampusManager /var/log/rtrlog|grep LINK_TRUNK
Jul 13 02:43:13 nmsccm05.kp.org 3124:2006 Jul 13 5:43:13 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcamosc1.oak.ca.kp.org,GigabitEthernet5/48],[10.236.45.4,6/24]
Jul 14 03:43:50 nmsccm06.kp.org 3683:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc133.str.ca.kp.org,GigabitEthernet0/1],[lcastrc1.str.ca.kp.org,3/4]
Jul 14 03:43:50 nmsccm06.kp.org 3685:2006 Jul 14 6:43:50 EDT:%CISCOWORKS-CampusManager-5-DCRP_LINK_TRUNK:[lcastrc121.str.ca.kp.org,GigabitEthernet0/1],[lcastrc1.str.ca.kp.org,3/3]

$ uname -a
SunOS wdcrtrlog1 5.9 Generic_118558-02 sun4u sparc SUNW,Sun-Fire-V240
$