Shell script or command(awk, sed, head or tail)

[Qcheck]

Hi All,

There is a file which contains 9 lines within and
range and keep appending with too many records. Please see the
example at the end:

How can I print from that file to tail with in and
range, I don't want to break the record when I print. When I tail -n
100, as that file will be growing, I don't want to get like the
following:


12-27-2006
1167234693351
0
SVCS
INFO

Please help me.

Example of the file:(It will be huge, I just gave an instance)


12-27-2006
1167234693351
0
SVCS
INFO
gl.log.Test
main
10
This test is a test INFO message


12-27-2006
1167234693507
1
SVCS
WARNING
gl.log.Test
main
10
This is a test WARNING message


12-27-2006
1167234693507
2
SVCS
SEVERE
MyCustomLogging
main
10
This is a test SEVERE message


Please assist me....How can I setup the begin and end rules for the above.

awk '//,// filename, doesn't work.... Any input is greatly appreciated....I wasted whole day working on it..

[Court Campbell]

Not sure I follow what you are looking for but try

tail -11 file_name

[Qcheck]

I want to tail so that the line should end with .... like:


12-27-2006
1167234693507
1
SVCS
WARNING
gl.log.Test
main
10
This is a test WARNING message


12-27-2006
1167234693507
2
SVCS
SEVERE
MyCustomLogging
main
10
This is a test SEVERE message



But if I do tail -number filename, I will get:
WARNING
gl.log.Test
main
10
This is a test WARNING message


12-27-2006
1167234693507
2
SVCS
SEVERE
MyCustomLogging
main

Since the file will be growing when I tail it might end without at the last line. So I should always print within the pattern and

[James R. Ferguson]

Hi:

# tail -n20 file | perl -ne 'print if ??'

Regards!

...JRF...

[Hein van den Heuvel · Accepted solution]

So do you want a function which prints only the last whole record block?

This awk program will do that:

(r==1) {rec[tmp,i++]=$0}
/^/ {i=0;r=1}
/^<\/record>/ {good=tmp; recs=i-1; tmp=1-tmp}
END {for (i=0;i
Line 1: If in a record block, stash away lines in tmp array portion of multidimensional record array

Line 2: If start of record block, then remember in a block, and start in begin of temp array

Line 3: If end of record block, promote complete tmp array to 'good' array, flag end, and start new tmp.

Line 4: At END, print last rec array which has last good whole block.


If this is nto exactly what you desire, maybe it help define your exact needs.

For example instead of just two 'big' values you have a circular list of N+1 and print the last N whole blocks.

For very large data volumes the most efficient solution would be to just remember (C, Perl) each location in a circular buffer and at end of file SEEK back to the appropriate one, and restart reading and printing.
That way you could also JUMP to where a prior run left of.



hth,
Hein van den Heuvel
HvdH Performance Consulting

[Qcheck]

James,

But it is giving only last record....even if I do tail -n60...

[James R. Ferguson]

Hi (again):

> James, But it is giving only last record....even if I do tail -n60...

I understood you to say that you *only* wanted the *last block* that was bounded by and .

That is what:

# tail -n13 file | perl -ne 'print if ??'

...was designed to do, using your sample input.

Regards!

...JRF...

[Bill Hassell]

Your awk example will work if you escape the / in the search string:

awk '//,/<\/record>/'

awk needs a way to find the search string delimiters so you escape the special characters between the / /. Now finding the last complete section between and will need some additional awk code.

[Hein van den Heuvel]

Ok, if you are feeding it with tail,
and if <\record> follow each other tightly then you can:

- start remembering all records as soon as is seen
- remember the last end seen:


tail x | awk '/^/ {r=1}
(r) {rec[i++]=$0}
/^<\/record>/ {end=i}
END {for (i=0;i
Hein.

[Qcheck]

Morning All,

No no no...I DID NOT say I want only last block..... When I tail I WANT the whole block to be displayed. If the tail doesn't get the whole block at the end then I should be able to trim the last one....The result I should get is from block to block(I mean record to record)....

James! With your command I get the last record, but I dynamically I should be able to display the multiple records, like the following:

<....constantly 9 lines will be there>


<....constantly 9 lines will be there>


<....constantly 9 lines will be there>


<....constantly 9 lines will be there>


BUT I DO NOT WANT TO DISPLAY:

<....constantly 9 lines will be there>


<....constantly 9 lines will be there>


<....constantly 9 lines will be there>


<....constantly 9 lines will be there>


<....only 6 lines displayed(any number other than 9 lines>


I hope you all understood what I am trying to explain....

Thank you for all your time in advance!

[Peter Godron]

Hi,
how about:
sed -n "1,`grep -n -e'' a.lis| tail -1 | cut -d':' -f1`p" a.lis

print line 1 to (find all /records, then get the line number of the last one)
a.lis is your datafile

http://forums1.itrc.hp.com/service/forums/helptips.do?#33 shows how to reward any useful answers given to your questions.

[Qcheck]

Hi Peter,

Thank you for your time...It is working, but how can I restrict the tail number like, tail -n 100 or 250 or etc.....

I don't want to get the whole file.....i should be able to get the number of the records, as I want....

like #tail -n 80 filename
or
# tail -n 120 filename

Your command is working fine to get the records from block to block, but how can I get the latest 50 records or 100 records etc....Do you know what I am saying....

[Hein van den Heuvel]

So did you try the awk solution I posted earlier?


tail -n 120 filename | awk '/^/ {r=1}
(r) {rec[i++]=$0}
/^<\/record>/ {end=i}
END {for (i=0;i

Hein.

[Qcheck]

Thank you so much Hein...It worked...when I want 20 lines, with tail 30, I get 20 lines which is two records....but hey no complain, atleast I am getting the records.....

I like scripting, but man! How can I learn this complicated scripting....;)

Thanks again so much for you and everyone who tried to assist me...

[OldSchool]

I *believe* what he wants is to display the last X blocks where:

a) "block" is bounded by
b) only complete blocks (ignore any incomplete trailing block)
c) X is variable

but I could be wrong....

[Peter Nikitka]

Hi,

I think OldSchool guessed right - at least would be a flexible solution, which seems interesting. So try:
awk -v sta='' -v fin='' -v num=2 -f last_n_blocks.awk infile

cat last_n_blocks.awk
$1 ~ sta {found=-1;i=0;bl++}
found<0 {rectmp[++i]=$0}
$1 ~ fin {found=1; for(j=1;j<=i;j++) {rec[bl%num,j]=rectmp[j]}; rn[bl%num]=i}
END {if(found<0) bl--;for(k=(bl-num+1);k<=bl;k++) {for(j=1;j<=rn[bl%num];j++) print rec[k%num,j]}}

You see, you can set the start / end pattern as well as the number of output blocks.

mfG Peter

[Qcheck]

Hi Peter,

Thanks for the reply...sorry I was in training so couldn't check this....

Anyways, how do I execute this? My file name is xmlfile. When I tried to execute the following it errored out as last_n_blocks.awk couldn't find. Can you bear with me and explain please....

awk -v sta='' -v fin='' -v num=2 -f last_n_blocks.awk infile

cat last_n_blocks.awk
$1 ~ sta {found=-1;i=0;bl++}
found<0 {rectmp[++i]=$0}
$1 ~ fin {found=1; for(j=1;j<=i;j++) {rec[bl%num,j]=rectmp[j]}; rn[bl%num]=i}
END {if(found<0) bl--;for(k=(bl-num+1);k<=bl;k++) {for(j=1;j<=rn[bl%num];j++) print rec[k%num,j]}}

Thanks

[Peter Godron]

Hi,
have you got your last_n_blocks.awk in the correct directory ? Any typing mistakes in the filenames ?

It works for me.

[Qcheck]

Hi Peter,

Do you want to me to create last_n_blocks file and insert the following lines as follows:

$1 ~ sta {found=-1;i=0;bl++}
found<0 {rectmp[++i]=$0}
$1 ~ fin {found=1; for(j=1;j<=i;j++) {rec[bl%num,j]=rectmp[j]}; rn[bl%num]=i}
END {if(found<0) bl--;for(k=(bl-num+1);k<=bl;k++) {for(j=1;j<=rn[bl%num];j++) print rec[k%num,j]}}

And then do I need to execute the following at the command prompt:

# awk -v sta='' -v fin='' -v num=2 -f last_n_blocks.awk xmfile

$$$$ xmlfile is the output of my records file which I would like to trim.....

Thank you...

[Peter Godron]

Yes, that is correct.
Create the last_n_blocks.awk file and insert the script as is.
Then run the awk line (Please note your latest post had a typingmistake in the input filename (xmfile rather than xmlfile))

[Peter Nikitka]

Hi,

you nearly have got it:
I want you to create a file
last_n_blocks.awk

with the content you saw from the cat-command and then execute the statement
awk -v sta='' -v fin='' -v num=2 -f last_n_blocks.awk xmlfile

where xmlfile contains the data to be trimmed.

mfG Peter

[Qcheck]

Hi Peter,

Thank you so much...it is working perfectly.... This is what I am looking for...

I am happy now......

But the thing is I want to learn and will be more happy if I do on my own...

I really appreciate for your assistance and time.

Thanks again...

[Hein van den Heuvel]

>> But the thing is I want to learn and will be more happy if I do on my own...


Good attitude!

be sure to read the man pages, or buy a book.

Keep your eyes open in this
forum for other (simple) awk question.

Realize that AWK is just a simple, limited, start.
Move over to PERL as soon as possible for just a little more work, but a lot more possibilities.

Finally, be sure to check my earlier, partial, solution where I gave a detailed stpe by step explanation applicable to your problem/context.

Good luck in the learning!
Hein.

[Qcheck]

Thank you Hein....I always want to spend time to learn but will easily distracted and will be busy with some other projects....

Anyways, I should build linux at home so that I can play around....

Thanks anyways for your suggestion and support...

Have a good one!
Thank you