topic Re: extracting data from string in Operating System - Linux

extracting data from string

Chartier Jerome — Thu, 11 Aug 2005 08:09:14 GMT

Hi All,

I need to extract some tagged line from a file formated like this:

flag_start
line1
line2
...
...
Serial_number1
linea
lineb
...
...
flag_stop
flag_start
line1
line2
...
...
Serial_number2
linea
lineb
...
...
flag_stop

In the file, there are several flagged patern with Serial_number1, and I would like to extract the whole pattern and write it in a new file ...

Can someone help me with this

Thanks in advance for your help

Best Regards

Jérôme

Re: extracting data from string

Doug O'Leary — Thu, 11 Aug 2005 08:15:14 GMT

Hey;

My understanding of your request is that you want everything from "Serial_number1" to flag_stop in a different file. Assuming that's the case, it's easy:

sed -n -e '/^Serial_number1/,/^flag_stop/p' ${in_file} > ${new_out_file}

HTH;

Doug

Re: extracting data from string

Leif Halvarsson_2 — Thu, 11 Aug 2005 08:15:52 GMT

Hi,
Not sure about exact what you want to do but, awk is a very useful tool for processing text files. Search the forum for "awk" and you will find a lot of examples how to use awk.

Re: extracting data from string

Chartier Jerome — Thu, 11 Aug 2005 08:22:23 GMT

Hi all,

Thanks for your answers, in fact, In each file, the pattern is flagged beetween flag_start and flag_stop.
In each file, I recognize the pattern by Serial_number, and there are several per file.
What I would like is to extract all the patterns (beetween flag_start and flag_stop) that have the same Serial_number.

Thanks in advance for your help

JÃ©rÃ´me C

Re: extracting data from string

Simon Hargrave — Thu, 11 Aug 2005 08:48:54 GMT

Hmm, so you want your logic to say: -

If the "serial number" pattern within this particular flagged block matched X, then extract the whole block?

If so then try something like:

awk -F"\n" 'BEGIN { matched=0 ; block="" } \
$1 !~ "^flag_s*" { block = block$1"\n" } \
$1 == "Serial_number1" { matched = 1 } \
$1 == "flag_stop" { if ( matched == 1 ) { print block } ; block="" ; matched=0 }' infile

Basically it sets a flagged called "matched" to zero. Then reads each line in turn, and appends it to a variable. If it gets to a matching serial number, it sets the "matched" flag to 1. Then when it comes to "flag_stop" it knows it's finished a block, and if at this time "matched" is 1, it prints the block, otherwise it doesn't.

This suit?

Re: extracting data from string

Kent Ostby — Thu, 11 Aug 2005 08:49:35 GMT

Jerome --

Can you give us like 10 to 20 lines of a real file.

The problem I'm having is figuring out if Serial_numberN appears multiple times or what.

The following script will print ALL of the data between flag_start and flag_stop. I suspect that's not what you want, but I thought I would post that much while I was asking my questions.

Create a file called useme.awk with the following in it:

BEGIN{daflag=0;}
/flag_start/ {daflag=1;next}
/flag_stop/ {daflag=0;next}
daflag==1 {print $0}

Run:

awk -f useme.awk < inputfile > outputfile

Re: extracting data from string

Simon Hargrave — Thu, 11 Aug 2005 08:50:22 GMT

Oh and my example assumed you wanted to strip the flag_start and flag_stop entries. If you want to keep them, simply change the second line to be:

{ block = block$1"\n" } \

and thus removing the ^flag_s* ignorance.

Re: extracting data from string

Rajesh SB — Thu, 11 Aug 2005 08:53:41 GMT

Hi,

I got some idea to extract the flagged patern to share with you.
Using combination of commands.

grep -n filename

where -n option give you line no. of startflag.
Extract the pattern using the head or tail command option to -/+.

Hope this hint may give some more idea.

Regards,
Rajesh

Re: extracting data from string

Hein van den Heuvel — Thu, 11 Aug 2005 09:02:40 GMT

How do you recognize a serial number?
Does it always start with "Serial" in the begin of a line?
You wanted the lines BETWEEN start and stop right, not inclusive right?
If so ten I think the perl below is what you need.
You'll need to adapt the 'Serial' match to the actual pattern, and you probably want a different algoritme for the file name generation ($file = ...)

hth,
Hein.

while (<>) {
if (/^flag_start/) {
$save = 1;
next;
}
if (/^flag_stop/) {
$save = 0;
if ($file) {
open (FILE,">> $file") or die "could not append to $file";
print FILE $line while ($line = shift @lines);
close FILE;
}
}
next unless $save;
push @lines, $_;
if (/^Serial/) {
chop;
$file = $_ . ".tmp" ;
}
}

Re: extracting data from string

Chartier Jerome — Thu, 11 Aug 2005 09:29:23 GMT

Thanks everyone ...

I would like to include the lines.
I am sorry, I am very bad in scripting so ..
Where do I put the input filename .. in the script?

Thanks in advance

Best Regards

JÃ©rÃ´me

Re: extracting data from string

Simon Hargrave — Thu, 11 Aug 2005 09:41:23 GMT

Who's script do you refer to? If you mean mine (the use of awk), then the input filename is after the code, eg you will see "infile" at the end of my post, that is the name of the input file. If you want the output in an output file then append >outfile at the end of the last line.

Re: extracting data from string

Hein van den Heuvel — Thu, 11 Aug 2005 09:50:01 GMT

Still not too clear.
But here is an other PERL variation.
This one takes the serial number desired as input variable.

If you cut & paste the perl below into a file called for example 'extract.p' then you execute as:

#perl extract.p Serial_number1 < all.data > extracted.data

Between my first example and this tweak you should have all possible needs covered (the second outputs start and stop... I'm sure you can figure out how)

My first example just execute as:

#perl extract.p < all.data

it will create: Serial_number1.tmp and Serial_number2.tmp

Good luck!
Hein

$serial = shift @ARGV or die "please provide serial number to select on";
while (<>) {
if (/^flag_start/) {
$save = 1;
$file = 0;
undef @lines;
}
if (/^flag_stop/) {
$save = 0;
push @lines, $_;
if ($file) {
print while ($_ = shift @lines );
}
print "\n";
}
push @lines, $_ if $save;
$file++ if /$serial/;
}

Re: extracting data from string

Chartier Jerome — Thu, 11 Aug 2005 10:16:39 GMT

Hello all,

Thanks for your answers.
Hein, your second script seems to run .. congratulations and thanks for all.
A last question Hein, If I want to do this on all files in the same directory?
For example files starting with a patern ABC.

Thanks again

JÃ©rÃ´me C

Re: extracting data from string

Chartier Jerome — Fri, 12 Aug 2005 02:49:39 GMT

Hi All,

Thanks a lot for your help.
Is it possible to extend the second perl script to pick every file in a directory starting with a common pattern and store all the results in one file?

Thanks all again

Best Regards

JÃ©rÃ´me

Re: extracting data from string

Muthukumar_5 — Fri, 12 Aug 2005 03:26:03 GMT

You can execute perl script as,

perl extract.pl ABC*.log

It will check all the files with ABC*.log pattern.

You can also use this script as,

for file in `ls `
do
perl -ne '{if(/^Serial_number1/){undef @arr;next;}if(/^flag_stop/){exit;}push @arr,$_;}END{print @arr;}' $file
done

For example:

test.log test1.log test2.log

for file in `ls test*.log`
do
perl -ne '{if(/^Serial_number1/){undef @arr;next;}if(/^flag_stop/){exit;}push @arr,$_;}END{print @arr;}' $file
done > Serial_number1.tmp

hth.

Re: extracting data from string

Hein van den Heuvel — Fri, 12 Aug 2005 08:55:05 GMT

JÃ©rÃ´me, Good to see you are all set now.
I kanda had read the requirement for multiple input files, but wanted to leave somethign for you to do :-).

Muthukumar, I discarded a one liner along the lines you showed because it was my understanding that the 'serial number' was after an arbitrary number of data lines that would also be needed. So you need to start remembering at flag_start, not just when you see you are reading data for the right serial number.

Cheers,
Hein.