topic Re: Shell script or command(awk, sed, head or tail) in Operating System - HP-UX

Shell script or command(awk, sed, head or tail)

Qcheck — Wed, 14 Mar 2007 14:20:19 GMT

Hi All,

There is a file which contains 9 lines within and
range and keep appending with too many records. Please see the
example at the end:

How can I print from that file to tail with in and
range, I don't want to break the record when I print. When I tail -n
100, as that file will be growing, I don't want to get like the
following:

12-27-2006
1167234693351
0
SVCS
INFO

Please help me.

Example of the file:(It will be huge, I just gave an instance)

12-27-2006
1167234693351
0
SVCS
INFO
gl.log.Test
main
10
This test is a test INFO message

12-27-2006
1167234693507
1
SVCS
WARNING
gl.log.Test
main
10
This is a test WARNING message

12-27-2006
1167234693507
2
SVCS
SEVERE
MyCustomLogging
main
10
This is a test SEVERE message

Please assist me....How can I setup the begin and end rules for the above.

awk '//,// filename, doesn't work.... Any input is greatly appreciated....I wasted whole day working on it..

Re: Shell script or command(awk, sed, head or tail)

Court Campbell — Wed, 14 Mar 2007 14:42:21 GMT

Not sure I follow what you are looking for but try

tail -11 file_name

Re: Shell script or command(awk, sed, head or tail)

Qcheck — Wed, 14 Mar 2007 14:51:18 GMT

I want to tail so that the line should end with .... like:

12-27-2006
1167234693507
1
SVCS
WARNING
gl.log.Test
main
10
This is a test WARNING message

12-27-2006
1167234693507
2
SVCS
SEVERE
MyCustomLogging
main
10
This is a test SEVERE message

But if I do tail -number filename, I will get:
WARNING
gl.log.Test
main
10
This is a test WARNING message

12-27-2006
1167234693507
2
SVCS
SEVERE
MyCustomLogging
main

Since the file will be growing when I tail it might end without at the last line. So I should always print within the pattern and

Re: Shell script or command(awk, sed, head or tail)

James R. Ferguson — Wed, 14 Mar 2007 14:54:23 GMT

Hi:

# tail -n20 file | perl -ne 'print if ??'

Regards!

...JRF...

Re: Shell script or command(awk, sed, head or tail)

Hein van den Heuvel — Wed, 14 Mar 2007 15:05:27 GMT

So do you want a function which prints only the last whole record block?

This awk program will do that:

(r==1) {rec[tmp,i++]=$0}
/^/ {i=0;r=1}
/^<\/record>/ {good=tmp; recs=i-1; tmp=1-tmp}
END {for (i=0;i
Line 1: If in a record block, stash away lines in tmp array portion of multidimensional record array

Line 2: If start of record block, then remember in a block, and start in begin of temp array

Line 3: If end of record block, promote complete tmp array to 'good' array, flag end, and start new tmp.

Line 4: At END, print last rec array which has last good whole block.

If this is nto exactly what you desire, maybe it help define your exact needs.

For example instead of just two 'big' values you have a circular list of N+1 and print the last N whole blocks.

For very large data volumes the most efficient solution would be to just remember (C, Perl) each location in a circular buffer and at end of file SEEK back to the appropriate one, and restart reading and printing.
That way you could also JUMP to where a prior run left of.

hth,
Hein van den Heuvel
HvdH Performance Consulting

Re: Shell script or command(awk, sed, head or tail)

Qcheck — Wed, 14 Mar 2007 15:05:28 GMT

James,

But it is giving only last record....even if I do tail -n60...

Re: Shell script or command(awk, sed, head or tail)

James R. Ferguson — Wed, 14 Mar 2007 15:10:31 GMT

Hi (again):

> James, But it is giving only last record....even if I do tail -n60...

I understood you to say that you *only* wanted the *last block* that was bounded by and .

That is what:

# tail -n13 file | perl -ne 'print if ??'

...was designed to do, using your sample input.

Regards!

...JRF...

Re: Shell script or command(awk, sed, head or tail)

Bill Hassell — Wed, 14 Mar 2007 15:21:00 GMT

Your awk example will work if you escape the / in the search string:

awk '//,/<\/record>/'

awk needs a way to find the search string delimiters so you escape the special characters between the / /. Now finding the last complete section between and will need some additional awk code.

Re: Shell script or command(awk, sed, head or tail)

Hein van den Heuvel — Wed, 14 Mar 2007 15:23:57 GMT

Ok, if you are feeding it with tail,
and if <\record> follow each other tightly then you can:

- start remembering all records as soon as is seen
- remember the last end seen:

tail x | awk '/^/ {r=1}
(r) {rec[i++]=$0}
/^<\/record>/ {end=i}
END {for (i=0;i
Hein.

Re: Shell script or command(awk, sed, head or tail)

Qcheck — Thu, 15 Mar 2007 07:01:00 GMT

Morning All,

No no no...I DID NOT say I want only last block..... When I tail I WANT the whole block to be displayed. If the tail doesn't get the whole block at the end then I should be able to trim the last one....The result I should get is from block to block(I mean record to record)....

James! With your command I get the last record, but I dynamically I should be able to display the multiple records, like the following:

<....constantly 9 lines will be there>

<....constantly 9 lines will be there>

<....constantly 9 lines will be there>

<....constantly 9 lines will be there>

BUT I DO NOT WANT TO DISPLAY:

<....constantly 9 lines will be there>

<....constantly 9 lines will be there>

<....constantly 9 lines will be there>

<....constantly 9 lines will be there>

<....only 6 lines displayed(any number other than 9 lines>

I hope you all understood what I am trying to explain....

Thank you for all your time in advance!

Re: Shell script or command(awk, sed, head or tail)

Peter Godron — Thu, 15 Mar 2007 07:21:26 GMT

Hi,
how about:
sed -n "1,`grep -n -e'' a.lis| tail -1 | cut -d':' -f1`p" a.lis

print line 1 to (find all /records, then get the line number of the last one)
a.lis is your datafile

http://forums1.itrc.hp.com/service/forums/helptips.do?#33 shows how to reward any useful answers given to your questions.

Re: Shell script or command(awk, sed, head or tail)

Qcheck — Thu, 15 Mar 2007 08:21:25 GMT

Hi Peter,

Thank you for your time...It is working, but how can I restrict the tail number like, tail -n 100 or 250 or etc.....

I don't want to get the whole file.....i should be able to get the number of the records, as I want....

like #tail -n 80 filename
or
# tail -n 120 filename

Your command is working fine to get the records from block to block, but how can I get the latest 50 records or 100 records etc....Do you know what I am saying....

Re: Shell script or command(awk, sed, head or tail)

Hein van den Heuvel — Thu, 15 Mar 2007 09:05:02 GMT

So did you try the awk solution I posted earlier?

tail -n 120 filename | awk '/^/ {r=1}
(r) {rec[i++]=$0}
/^<\/record>/ {end=i}
END {for (i=0;i

Hein.

Re: Shell script or command(awk, sed, head or tail)

Qcheck — Thu, 15 Mar 2007 09:15:54 GMT

Thank you so much Hein...It worked...when I want 20 lines, with tail 30, I get 20 lines which is two records....but hey no complain, atleast I am getting the records.....

I like scripting, but man! How can I learn this complicated scripting....;)

Thanks again so much for you and everyone who tried to assist me...

Re: Shell script or command(awk, sed, head or tail)

OldSchool — Mon, 19 Mar 2007 10:39:40 GMT

I *believe* what he wants is to display the last X blocks where:

a) "block" is bounded by
b) only complete blocks (ignore any incomplete trailing block)
c) X is variable

but I could be wrong....

Re: Shell script or command(awk, sed, head or tail)

Peter Nikitka — Mon, 19 Mar 2007 13:19:41 GMT

Hi,

I think OldSchool guessed right - at least would be a flexible solution, which seems interesting. So try:
awk -v sta='' -v fin='' -v num=2 -f last_n_blocks.awk infile

cat last_n_blocks.awk
$1 ~ sta {found=-1;i=0;bl++}
found<0 {rectmp[++i]=$0}
$1 ~ fin {found=1; for(j=1;j<=i;j++) {rec[bl%num,j]=rectmp[j]}; rn[bl%num]=i}
END {if(found<0) bl--;for(k=(bl-num+1);k<=bl;k++) {for(j=1;j<=rn[bl%num];j++) print rec[k%num,j]}}

You see, you can set the start / end pattern as well as the number of output blocks.

mfG Peter

Re: Shell script or command(awk, sed, head or tail)

Qcheck — Thu, 22 Mar 2007 07:14:38 GMT

Hi Peter,

Thanks for the reply...sorry I was in training so couldn't check this....

Anyways, how do I execute this? My file name is xmlfile. When I tried to execute the following it errored out as last_n_blocks.awk couldn't find. Can you bear with me and explain please....

awk -v sta='' -v fin='' -v num=2 -f last_n_blocks.awk infile

cat last_n_blocks.awk
$1 ~ sta {found=-1;i=0;bl++}
found<0 {rectmp[++i]=$0}
$1 ~ fin {found=1; for(j=1;j<=i;j++) {rec[bl%num,j]=rectmp[j]}; rn[bl%num]=i}
END {if(found<0) bl--;for(k=(bl-num+1);k<=bl;k++) {for(j=1;j<=rn[bl%num];j++) print rec[k%num,j]}}

Thanks

Re: Shell script or command(awk, sed, head or tail)

Peter Godron — Thu, 22 Mar 2007 07:22:29 GMT

Hi,
have you got your last_n_blocks.awk in the correct directory ? Any typing mistakes in the filenames ?

It works for me.

Re: Shell script or command(awk, sed, head or tail)

Qcheck — Thu, 22 Mar 2007 07:27:55 GMT

Hi Peter,

Do you want to me to create last_n_blocks file and insert the following lines as follows:

$1 ~ sta {found=-1;i=0;bl++}
found<0 {rectmp[++i]=$0}
$1 ~ fin {found=1; for(j=1;j<=i;j++) {rec[bl%num,j]=rectmp[j]}; rn[bl%num]=i}
END {if(found<0) bl--;for(k=(bl-num+1);k<=bl;k++) {for(j=1;j<=rn[bl%num];j++) print rec[k%num,j]}}

And then do I need to execute the following at the command prompt:

# awk -v sta='' -v fin='' -v num=2 -f last_n_blocks.awk xmfile

$$$$ xmlfile is the output of my records file which I would like to trim.....

Thank you...

Re: Shell script or command(awk, sed, head or tail)

Peter Godron — Thu, 22 Mar 2007 07:38:59 GMT

Yes, that is correct.
Create the last_n_blocks.awk file and insert the script as is.
Then run the awk line (Please note your latest post had a typingmistake in the input filename (xmfile rather than xmlfile))