topic grep string from compressed file in Operating System - HP-UX

grep string from compressed file

lawrenzo_1 — Tue, 06 Mar 2007 08:52:19 GMT

Hi,

I have created a 500mb file by concatenating lots of small files together:

file:abc.txt
data
data
data
:EOF

I have then compressed the file. What I would like to know is the command that can extract all the data between file:abc.txt and :EOF without uncompressing the file.

I have installed gnu.grep (zgrep) which can find the string in that file.

Thanks in advance.

Chris.

Re: grep string from compressed file

TwoProc — Tue, 06 Mar 2007 09:04:07 GMT

How did you concatenate the files together?
"cat"?

If so, I doubt that there are any file header indicators in the file - so all you could use to key off of for the end of abc.txt would be whatever the last piece of text is in abc.txt file.

Re; the second requirement of not uncompressing. Well you're going to have to uncompress either via your own code or some code from a program that understands the compression for that file - but you're going to have to uncompress.

UNLESS, you mean you don't want to uncompress to a file - you could uncompress to a pipe. Is that the question here, how to retrieve the 3 data files from a pipe which has uncompressed the file ?

Re: grep string from compressed file

Patrick Wallek — Tue, 06 Mar 2007 09:07:39 GMT

What did you use to compress the file? gzip or compress?

If you used gzip, try the following:

# gzcat file.gz | grep whatever

If you used compress, try the following:

# zcat file.Z | grep whatever

Re: grep string from compressed file

James R. Ferguson — Tue, 06 Mar 2007 09:10:43 GMT

Hi Chris:

# gzip -d -c file|perl -nle 'print if /abc.txt/../:EOF/'

...JRF...

Re: grep string from compressed file

Peter Godron — Tue, 06 Mar 2007 09:12:06 GMT

Chris,
can't you use something like:
zgrep -v -e'^file:' -e'^:EOF' input > output

Re: grep string from compressed file

lawrenzo_1 — Tue, 06 Mar 2007 09:12:57 GMT

ok thanks guys.

to create the files I ran a small script

for aa in `ls dir`
echo "file:$aa">>$outfile
cat $aa >> $outfile
echo ":EOR" >>$outfile
done

I will try gzcat.

Re: grep string from compressed file

lawrenzo_1 — Tue, 06 Mar 2007 09:31:45 GMT

ok lots more ideas to try here ....

I am playing with the gzip utility which works well to extract the string I require however the string is a unique number and will only appear once therefore once the string has been found then the search should stop.

here is what I require:

string to search is BGM001025176958

I need to recreate the file that was orginally on the server:

file:abc.txt
data
data
BGM001025176958
data
:EOR

so once the string has been identified then I want to create the file abc.txt containing

data
data
BGM001025176958
data

I have an idea howevr my awk syntax ain't quite correct.

Thanks

Re: grep string from compressed file

Hein van den Heuvel — Tue, 06 Mar 2007 12:08:18 GMT

Here is some awk that would do what you desire:

awk -F: '/BGM001025176958/{p=1} /^:EOR/ && (p) {while (i name} exit} {a[n++]=$0} /^file /{name=$2; n=0}'

C:\Temp>type abc.txt
file aaa.txt
data
BGM001025176958
data

With comments....

/BGM001025176958/{p=1} ## Desired pattern? Set print flag!

/^:EOR/ && (p) ## Line starting with end mark? Also have print flag set? then...
{while (i{print a[i++] > name} ## Print onto fresh file with remembered name
exit} ## and done

{a[n++]=$0}' ## Remember every line in its own place.

/^file:/ {name=$2; n=0} ## Line starting with new file? remember name part and reset number of lines.

Enjoy!
Hein.

Re: grep string from compressed file

lawrenzo_1 — Wed, 07 Mar 2007 07:41:35 GMT

The command isn't returning anything ....

any other idea's?

Thanks

Re: grep string from compressed file

Dennis Handly — Wed, 07 Mar 2007 07:55:52 GMT

>The command isn't returning anything

I got an error message showing the problem:
awk: A print or getline function must have a file name.

I then initialized name to "bad_news" and got the output.

The problem is an extra space after the "e" in file. I fixed it here:
/^file/{name=$2; n=0}

It might be easier to read as:
awk -F: '
BEGIN { name="bad_news" }
/BGM001025176958/{p=1}
/^:EOR/ && (p) {while (i name } exit}
{a[n++]=$0}
/^file/{name=$2; n=0}'

Re: grep string from compressed file

Hein van den Heuvel — Wed, 07 Mar 2007 08:02:28 GMT

> >The command isn't returning anything

Ooops, like Dennis indicates i made a cut & paste error. I tried it on a Windoze box and the awk had trouble with specifying a ":" as field seperator. So I changed the data to use spaces, for the test replacing them in the posted string.. except for the /^file:/ match. Sorry.

And, You need to feed (pipe) it the input stream of course,

Cheers,
Hein.

Re: grep string from compressed file

lawrenzo_1 — Wed, 07 Mar 2007 08:04:51 GMT

that now works a treat!

Thanks guys.

Chris

Re: grep string from compressed file

lawrenzo_1 — Wed, 07 Mar 2007 08:40:27 GMT

ty ty ty