nedd to extrac some line from a file.

[vivek_5]

HI,

I have asked for a small script but i this this was not right way.

i need help for following and i have also given the sample code which i wrote.( which is not good).

based on a field ( 13th field) on first line of the file i have to delete some rows from file and create a one big file.

if the value of 13th column is 'Y' then i have to remove first three and last line from the files or if it is 'N' then first two and last lines need to be removed.

this is to be done for all the files in a directory.

example:
#create empty file
touch bigfile

for filename in *
do

ind=`head -1 filename|awk '{print $13}'
total_lines=`wc -l filename|awk '{print $1}'

if [ $ind = Y ]
then
rows_to_delete=3
fi

if [ $ind = Y ]
then
rows_to_delete=2
fi

#remove last line
rows=`expr $total_lines - 1 `
head -$rows filename > filename_1

#remove header rows
rows=`expr $rows - $rows_to_delete `
tail -$rows file_name >filename_2

#merge into a one big file
cat fileanme_2 >>bigfile

done

# now i have to remve any EOF mark except last one from the big file.

????? ( nedd help).


now i think i should be helped because i am also trying but not in a good way. please help me to make this script work.

waiting for reply.

vivek S.

[Darrell Allen]

Hi Vivek,

You are setting rows_to_delete twice based on the same condition. Thus, if your 13th field = Y you are setting rows_to_delete to 3 then immediately setting it to 2.

When you append a file to another with cat "filename_2 >>bigfile" you don't have to worry with removing end of file marks. There will only be one at the end of the output file (bigfile).

In your code "remove header rows" you should use tail on filename_1 instead of file_name if you want to remove the last couple of lines from the file you just removed the first line from.

I presume fileanme_2 in the last cat statement should be filename_2.

A warning: tail and head are limited in the size of a file they will work on. See the man pages.

Darrell

[Sridhar Bhaskarla]

Hi Vivek,

I would use sed and awk to get what you are trying to achieve.

//start


for filename in *
do

ind=`sed -n 1p filename|awk '{print $13}'`

case $ind in
"Y") ROWS=3;;
"N") ROWS=2;;
esac

sed -e '$d' -e '1,'$ROWS'd' filename >> bigfile

done

//End


-Sri

[vivek_5]

hi,

Yes i do agree with you allen. those were typo. i do understand there are some size restriction with 'tail' and 'head'. that is why i am looking for some good ways to do that. sice my script is suppoded to be very efficient and fasy because there would be around 200 files and each files would be having around 65 thousand lines.

sridhar i would like to know that whatever u suggest me is the best way for me, keeping in mind the volume of data and files need to be precessed.

i have to look manual to understand the sed commad.

please do reply.

THANKS.

[Sridhar Bhaskarla]

Vivek,

This script is what I would write to accomplish your task. I cannot say it is the best way. But it will not do any harm as it is not going to mess the existing files. There are more experienced members here than me that may suggest a better way. If you want, you can wait for their responses too.

sed and awk are intended for text parsing. Head and tail will work for relatively small files.

-Sri

[Joseph A Benaiah_1]

Vivek,

You could try this awk statement in your for loop:

awk '
NR == 1 {
if ( $13 == "Y" )
{
ind = "Y"
}
row[NR] = $0
}

NR > 1 {
row[NR] = $0
}

END {
if ( ind == "Y" )
{
for ( i = 4 ; i <= (NR - 1) ; i++ )
{
print row[i]
}
}
}' $filename >> bigfile

Regards,

Joseph.

[Justo Exposito]

Hi,

What about using ed for doing this?

Regards,

Justo.

[Justo Exposito]

Hi Vivek,

With ed you can do this by an easy way, you need a file with the commands to do like this:
1,2 d
$,$ d
1,$ w
First command deletes the 2 first lines of the file, second command deletes the last line, third command write the file with the changes.

And then run ed file < FileWithCommands.

If you want you can generate the command file at runtime.

Best Regards,

Justo.