Compression of Big files

[Vibhor Kumar Agarwal]

Hi,

I have a 3 GB and bigger tar files.

What are the ways of compressing it.
While using gzip it says, file too large.

[Muthukumar_5]

You can use bzip2 for this purpose.

Refer this: http://www.digistar.com/bzip2/

hth.

[Arunvijai_4]

1) You can try with bzip2
2) Compress the file in a location where you have free space more than 3 GB.
3) Use gzip -9 (Better compression,bit slow)

-Arun

[Vibhor Kumar Agarwal]

I'll try bzip2 for this.

gzip -9 gives the same output.

This time i have copied it.
"Value too large for defined data type"

[Vibhor Kumar Agarwal]

No I can't bzip2.

Because i have to deliver this to different locations, and they won't install that.

[VEL_1]

Hi,

Which file system are u using? And OS?

[Arunvijai_4]

I think, you need to compile gzip to enable large file compression.

http://www.brandonhutchinson.com/gzip__Value_too_large_for_defined_data_type.html

[Arunvijai_4]

One more link, http://www.gzip.org/#faq10

[VEL_1 · Accepted solution]

Alternatively, you can try to use the dd command:

Please see the following link:

http://wdb1.sco.com/kb/showta?taid=112962


Note:

If you used old version of gzip, its better to upgrade gzip. Because old version of gzip may not support the large files.

[Vibhor Kumar Agarwal]

Just a small problem:

I am not an administrator, so don't have enough rights to do the installation.

[HugoAlmeida]

Hi

Try
compress -fv

HTH

[Arunvijai_4]

Can you compile gzip yourself on your server ? Do you have enough permission to do that ?
In such cases, you can build your own gzip and use it for compression.

[Alessandro Pilati]

Download gzip here and recompile it http://hpux.connect.org.uk/hppd/hpux/Gnu/gzip-1.3.5/
without restrictions

or
download this package:
http://h20293.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPUXIEXP1123

regards,
Alex

[Vibhor Kumar Agarwal]

Compressing is working, but its too slow and also it doesn't give a good compression ratio.

Compiling gzip by myself will end up in the same problem, when i deliver the file, the reciepient might not be able to decompress it.

[Vibhor Kumar Agarwal]

Hey,

That dd option seems to work. :-)
I just replaced compress with gzip and is working.
Will take some time to complete, will update you then.

[Vibhor Kumar Agarwal]

Wow,

That dd worked wonderfully.

At the end of dd it gave me the output:
300+1 records in
300+1 records out
0+52264 records in
0+52264 records out

What does it mean.

[Fabio Ettore]

Hi,

what dd command did you launch exactly which got that output?

Best regards,
Fabio

[Vibhor Kumar Agarwal]

Check out the raj's link.

[Michael Roberts_3]

cat bigFile | gzip -c - | cat - > bigFile.gz

'cat' is OK with large files and since 'gzip' is reading/writing a pipe stream it doesn't have to deal with a large file descriptor.

[Vibhor Kumar Agarwal]

Its not a file, its a whole directory structure tarred up.

Will it work.

[A. Clay Stephenson]

If it's tar'ed up; it's still a file (commonly called a "tarball") so that it will still work. Alternately you can pipe the output of tar to another utility like compress.