grep lines under the event header

[Jeeshan]

In my event.log i have entries like the following

.................................................................................................................

Summary:
Disk at hardware path 0/4/1/0.2.8.0.1.7.0 : Software configuration error

................................................................................

and the summaries are coming consecutively when an event happens.

I need to write a shell script to grep the lines bellow in the Summary heading. and also can able to filter of that output.

i.e. I need only the output

Summary:
Disk at hardware path 0/4/1/0.2.8.0.1.7.0 : Software configuration error

[Dennis Handly]

I assume the summaries are multiple lines but followed by a blank? or a line of "..."?

If a fixed number of lines, you could use GNU grep with -A4 to print 4 lines.

Otherwise you could use an awk state machine:
awk '
/^Summary:/ { flag=1 } # start
/^\.\.\.\./ { flag=0 } # end
flag { print $0 }
' event.log

[Jeeshan]

Thanks Dennis for your reply.

>>I assume the summaries are multiple lines but followed by a blank? or a line of "..."?

Yes, there are multiple lines before and after.

But how can I grep the lines, after the Summary in a POSIX shell?

[Jeeshan]

using the bellow command

tail -150 /var/opt/resmon/log/event.log|awk ' /^Summary:/ { flag=1 } /^\.\.\.\./ { flag=0 } flag { print $0 } '

I got the following results

Summary:
Adapter at hardware path 0/6/1/0 : Received an interrupt indicating that a
primitive was transmitted


Description of Error:


lbolt value: 3806

The Fibre Channel Driver received an interrupt indicating
that a primitive was transmitted
Frame Manager Status Register = 0xa0085480


Probable Cause / Recommended Action:

The Tachyon TL adapter transmitted a primitive sequence.
This is caused by the driver issuing a primitive sequence
to the chip to be transmitted.
No action is required. Informative message

Additional Event Data:
System IP Address...: 172.16.10.4
Event Id............: 0x49be522a00000000
Monitor Version.....: B.01.00
Event Class.........: I/O
Client Configuration File...........:
/var/stm/config/tools/monitor/default_dm_TL_adapter.clcfg
Client Configuration File Version...: A.01.00
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):
0x49be515c00000000
Additional System Data:
System Model Number.............: 9000/800/rp4440
OS Version......................: B.11.11
EMS Version.....................: A.04.00
STM Version.....................: A.47.00
Latest information on this event:
http://docs.hp.com/hpux/content/hardware/ems/dm_TL_adapter.htm#18

v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v



Component Data:
Physical Device Path....: 0/6/1/0
Vendor Id...............: 0x0000103C
Serial Number(WWN)......: 50060B0000305A1C

I/O Log Event Data:

Driver Status Code..................: 0x00000012
Length of Logged Hardware Status....: 0 bytes.
Offset to Logged Manager Information: 0 bytes.
Length of Logged Manager Information: 61 bytes.

Manager-Specific Information:

Raw data from FCMS Adapter driver:
00000006 00000EDE 00000001 00000001 A0085480 2F75782F 6B65726E 2F6B6973
752F544C 2F737263 2F636F6D 6D6F6E2F 7773696F 2F74645F 6973722E 63

but I only need the summary points descriptions.

[Hein van den Heuvel · Accepted solution]

I don't have a real file example for this handy, but it looks like you just want to use the 'range' selection that AWK offers.
Start with 'Sum' at the beginning of a line.
End with the first empty line.

$ awk '/^Sum/,/^$/' tmp.txt
Summary:
Adapter at hardware path 0/6/1/0 : Received an interrupt indicating that a
primitive was transmitted


Alternative:
$ awk '/^Sum/,!NF' tmp.txt


If I misinterpreted the data format/request, then please consider replying with a .TXT attachement for a section of the file with 2 target zones.

hth,
Hein.

[Jeeshan]

It perfectly worked, Hein. Thanks a lot man.

another question.

in my eventlog there is another field named
Event Time. and the format is

Event Time..........: Tue Mar 17 20:57:58 2009

my question is how can I reformat the "Tue Mar 17 2009" like this "17032009"

[James R. Ferguson]

Hi:

With regards to your last question of reformatting the date from the "Event" line you could use:

# cat ./filter
#!/usr/bin/perl -nl
BEGIN{
%m=(Jan=>1,Feb=>2,Mar=>3,Apr=>4,May=>5,Jun=>6,
Jul=>7,Aug=>8,Sep=>9,Oct=>10,Nov=>11,Dec=>12)};
m{^Event.+(\w{3})\s(\d\d).+\s(\d{4})} and
printf "%02d%02d%4d\n",$2,$m{$1},$3;
1;

For example:

# X="Event Time..........: Tue Mar 17 20:57:58 2009"
# echo ${X} | ./filter
17032009

Regards!

...JRF...

[Dennis Handly]

>how can I reformat the "Tue Mar 17 2009" like this "17032009"

Why would you want to format the date into something that doesn't sort?
20090317 would be better.

[James R. Ferguson]

Hi (again):

I would agree with Dennis, that the representation of the converted date as YYYYMMDD is a superior one.

Moreover, although I don't have a raw file, we can quickly the Perl script I offered to read you file, find the "SUmmary" lines _and_ convert the event date line like this:

# cat ./filter
#!/usr/bin/perl -nl
BEGIN{
%m=(Jan=>1,Feb=>2,Mar=>3,Apr=>4,May=>5,Jun=>6,
Jul=>7,Aug=>8,Sep=>9,Oct=>10,Nov=>11,Dec=>12)};

print if ( /^Summary/../^\s*$/ );

m{(^Event.+)(\w{3})\s(\d\d).+\s(\d{4})} and
printf "%s%4d%02d%02d\n",$1,$4,$m{$2},$3;
1;

...then:

# ./filter eventfile
Summary:
Adapter at hardware path 0/6/1/0 : Received an interrupt indicating that a
primitive was transmitted

Event Time..........: Tue 20090317

...

Regards!

...JRF...

[Jeeshan]

Thanks Dennis, yeah may be you are right. but my main concern to reformat the date is to verify from system date that the log is today's or yesterday's date and notify the Summary to Admins.

Thanks James for your reply.

[Jeeshan]

More problems:

I have a script which is monitoring disk usage in every time interval like 15 or 20 minutes. If any file system usage reaches 90% it gives an SMS and email.
How can I set logic that if any file system reaches 90% it will give me an alert then if its going higher than the previous alarm then it send the new alert without giving me the same alert on every check?

[James R. Ferguson]

Hi:

> How can I set logic that if any file system reaches 90% it will give me an alert then if its going higher than the previous alarm then it send the new alert without giving me the same alert on every check?

You need to write the utilization level into a file. If the threshold has been exceeded, send an alert only if the current sample's value exceeds the threshold _AND_ is larger than the last value recorded.

Regards!

...JRF...

[Jeeshan]

thanks James for your reply.

I got your point but my problem is, say a file system reaches 90% and then the script send me an alert notification. in next checking it also gives me the same file system notification.
then how can i set logic that, if it does only send one notification and in next checking it only send if any changes occur.

[Dennis Handly]

>then how can i set logic that, if it does only send one notification and in next checking it only send if any changes occur.

As JRF said, you need to record the value when you send the message. If it goes down, you can record that and not send the message. Then when it goes up, send another message and record that value.

[Dennis Handly]

Here is a small script fragment to demonstrate the logic:
THRESHOLD=90
for i in 70 80 90 95 91 95 80; do
    if [ $i -ge $THRESHOLD ]; then
       # report if larger
       if [ $i -gt $(< reported) ]; then
          echo "send mail for: $i"
       else
          echo "No need to send mail for $i, previous was $(< reported)"
       fi
       echo $i > reported
    else
       echo $THRESHOLD > reported
    fi
done

[Dennis Handly]

Oops, improper edge arithmetic:
THRESHOLD=90
for i in 70 80 90 95 91 95 80 90; do # testing
    if [ $i -ge $THRESHOLD ]; then
        # report if larger
        if [ $i -gt $(< reported) ]; then
            echo "send mail for: $i"
        else

            echo "No need to send mail for $i, previous was $(< reported)"
        fi
        echo $i > reported
    else
        echo $(( THRESHOLD -1 )) > reported
    fi
done

[Jeeshan]

Hi Dennis

find the attached script of my file system notification script. please tell me

1.where can i put my logic.
2. whenever i want to put % usage it comes by mail but not in SMS. why?

[Dennis Handly]

>1. where can I put my logic?

In check_slash:
a=$(awk '{print $2; next}' $HS.bdf_info)
b=$(head -n 1 $HS.bdf_info)
if [ $a -gt 55 ]; then
    # report if larger
    if [ $a -gt $(< $HS.reported) ]; then
        print $b% >> $HS.mes.txt
    else
        rm -f $HS.mes.txt
    fi
    echo $i > $HS.reported
else
    rm -f $HS.mes.txt
    echo 55 > $HS.reported
fi


Note: You are cat happy.
MES=`cat $1.mes.txt`
EV=`ssh @$1 bdf > $1.bdf.info`
HS=`echo $1`
SM=`echo "File system warning of $1 server:" > $1.mes.txt`

Not sure why you have EV and SM, they will will be empty:
HS=$1 # use $HS instead
MES=$(< $HS.mes.txt)
ssh @$HS bdf > $HS.bdf.info
echo "File system warning of $HS server:" > $HS.mes.txt
if [ ! -f $HS.reported ]; then
    echo 55 > $HS.reported
fi

You also have "rm $HS.bdf.info" vs "$HS.bdf_info".

[James R. Ferguson]

Hi:

I'd like to add to Dennis's comments about your "cat happiness". You can make your scripts far faster and far less resource intensive by _not_ spawning extra processes when you don't need them.

Whenever you write:

# cat file | some_command

...you spawn at least two processes --- one of which is the 'cat' to read the file and then write each line read out to 's STDIN. Thus, not only have you used two processes where one is needed, but you have done an extra read and write of your file!

This is probably not noticable if your script reads small amounts of data and does little work. It will degrade the performance of large scripts that iteratively process large amounts of data.

Instead of the above, simply write:

# some_command file

Thus whenever the command is (for example) 'grep' or 'awk' or 'sed' and you start to write a pipeline that begins with 'cat', STOP!

Another variation was when you wrote:

# MES=`cat $1.mes.txt`

...which Dennis changed to :

# MES=$(< $HS.mes.txt)

He improved _two_ things here. First, the 'cat' process was eliminated and the shell's internal facilities were used to read the file's contents into the 'MES' variable. Second, instead of using the archaic back-ticks to run and capture the output of the process, the more clear "$( ... )" syntax was used.

As you become more familar with 'awk', look for its use with pipelines involving commands like 'grep', 'sed', 'head' and 'tail'. The 'awk' facility is designed to match (for 'grep'); to substitute (with 'sub' and 'gsub' for 'sed'); and to track line numbers with its 'NR' variable instead of using 'head' and 'tail' in a pipeline.

Lastly, the overall structure of your script could be improved. Since you call your functions (subroutines) only once, you could eliminate them and inline all of the code. Otherwise, it would be clearer to write all your functions at the beginning of the script, leaving their calls to the "main" block at the end.

Regards!

...JRF...

[Jeeshan]

Thanks Dennis and JRF for your valuable reply.
I'll keep all your suggestions in my mind.

Actually what I wan to do is, to check file system usage. And file systems are not same in all servers. I want to set a threshold level of critical file systems like / threshold will be 70%, /tmp and /var will be 80% and rest of all will be 90%.

if any file system usage will be over its threshold level, it will send an alert (SMS or Mail) but not send same message repeatetively.
only if any changes(low or high of threshold) will occur.

[Jeeshan]

I have changed my script Dennis. Helpful if you please correct it to set logic

#!/usr/bin/sh

SMS_RECEIVER=''
HS=$1
MAIL_RECEIVER=""
ssh @$1 bdf|awk 'NF==1{f=$0;getline;$0=f$0}{if($5+0>50)print}'|awk '{print $6 " " $5}'|sed 's/'%'/''/' > $1.bdf.info
print "File system warning of $HS server:" > $HS.mes.txt

function check_slash {
a=`cat $HS.bdf.info|grep -w /`
b=`cat $HS.bdf.info|grep -w /|awk '{print $2}'`
if [[ $b > `expr 55` ]]
then
print "$a%" >> $HS.mes.txt
fi
}
check_slash

function sms {
if [[ -f $HS.mes.txt ]]
then
# /usr/local/bin/lynx -dump `$SMS_RECEIVER `cat $HS.mes.txt``
#/usr/local/bin/lynx -dump "$SMS_RECEIVER `cat $HS.mes.txt`"
mailx -s "File System warning of $HS server:" $MAIL_RECEIVER < $HS.mes.txt

fi

}
sms

I will run the script like this

file.sh

[Dennis Handly]

>I have changed my script

#!/usr/bin/sh
SMS_RECEIVER=''
HS=$1
MAIL_RECEIVER=""

function check_slash {
a=$(grep -w / $HS.bdf.info)
b=$(grep -w / $HS.bdf.info | awk '{print $2}')
if [ $b -gt 55 ]; then
    if [ ! -f $HS.reported ]; then
        echo 55 > $HS.reported
    fi
    # report if larger
    if [ $b -gt $(< $HS.reported) ]; then
        print $b% >> $HS.mes.txt
    else
        rm -f $HS.mes.txt
    fi
    echo $b > $HS.reported
else
    rm -f $HS.mes.txt $HS.reported
fi
}

function sms {
if [[ -f $HS.mes.txt ]]; then
    # /usr/local/bin/lynx -dump $($SMS_RECEIVER $(< $HS.mes.txt) )
    #/usr/local/bin/lynx -dump "$SMS_RECEIVER $(< $HS.mes.txt)"
    mailx -s "File System warning of $HS server:" \
    $MAIL_RECEIVER < $HS.mes.txt
fi
}

ssh @$1 bdf | awk '
NF==1 {f=$0; getline; $0=f$0}
{if ($5+0 > 50) print $6, $5 }' | sed 's/%//' > $HS.bdf.info
print "File system warning of $HS server:" > $HS.mes.txt

check_slash
sms

As JRF mentioned move the functions to the top and just invoke their names.

(I wasn't sure why you had stuttered your single quotes in sed?)

[Jeeshan]

Hey Dennis
thanks buddy.

Really appreciate your thought. But i have slightly changed the logic. Its working now.

Hope to get further help from you.

[Jeeshan]

another issue.

Create a shell script which will edit a file content which have some syntax. An user must call this script to modify the file and while saving it will first notify to the group owner of that file. If the owner permit him the script will than save the changes in that file and push the file on all servers.

[Dennis Handly]

>Create a shell script which will edit a file content which have some syntax.

This is vague. Make changes to a copy so you can do the rest.

>it will first notify to the group owner of that file. If the owner permit him the script will than save the changes in that file

You are actually going to send mail to the group owner? Or are you going to check the group permissions and overwrite it anyway if you can?

>push the file on all servers.

Look into rdist?