Header and Trailer Records

[intp]

Unix Box..

I have a datafile with header and trailer records...i want to remove header and trailer records and move rest of the contents to new file ... Since size is huge (arround 3-4GB) pls let me know which is the best way to do this ?

[A. Clay Stephenson]

You don't have that many options using standard UNIX utilities. Dd is probably your least evil standard utility. Man dd for details and note that you have an option to seek to the desired input starting point. You can also specify a number of output records to write.

A better option would be a small C program or a Perl script. Perl is capable of reading binary data so this would be a straightforward exercise in either C or Perl provided that you know the data.

[James R. Ferguson · Accepted solution]

Hi:

If the header and the trailer line are the first and the last line of the file, then this is quite quick:

# sed -e '1'd -e '$'d filein > fileout

Regards!

...JRF...

[A. Clay Stephenson]

Of course, text-based tools like sed and awk can only work if these are line-oriented data -- and even then if the record lengths (lines) are of reasonable size. Since you haven't bothered to describe your data, it's not possible to be more helpful.

[James R. Ferguson]

Hi (again):

Another way is to use regular expressions to match records (assuming a line-oriented, text file) and discard what you define as a "header" or a "trailer". For example to discard the interpreter line of a shell script and a last line that says "exit", do:

# perl -ne 'print unless m/bin\/sh|^exit/' filein

This prints everything that isn't "bin/sh" or "exit" anchored to the beginning of a line.

Regards!

...JRF...

[intp]

Thanks all.

This worked for me.
# sed -e '1'd -e '$'d filein > fileout