tar: size of thd0data.dat > 8GB. Not dumped.

[ziad_1]

Hi All,

Please can somebody help if we have any way to backup or copy directory structures containing files larger than 8Gb? I have tried to create .tar file with the command
#tar –cvf thd0data.dat.tar thd0data.dat but I get the message tar: size of thd0data.dat > 8GB. Not dumped. Are we limited to files up 8GB size with tar or we have any solution or patches or anything else to Backup files larger than 8 GB with the command tar. Thanks a lot for your urgent reply.

Kindly Regards

[Murat SULUHAN]

Hi Ziad

fbackup

http://docs.hp.com/en/B2355-90691/fbackup.1M.html

Best Regards

[ziad_1]

Hi,

Thanks for your prompt reply But I need a solution to make the backup with the command tar for files larger than 8GB. Is this possible with tar command or tar is limited to 8GB and there is no solution for files larger than 8GB.

Kindly Regards

[Murat SULUHAN]

Hi

following url may help you

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=181759

Best Regards
Murat

[Steven Schweda]

As usual, it might help if you disclosed your
HP-UX version.

"pax" and GNU "tar" are obvious potential
solutions. GNU "tar" is up to version 1.19
now:

man pax
http://docs.hp.com/en/B2355-60127/pax.1.html

http://www.gnu.org/software/tar/

> [...] I need a solution to make the backup
> with the command tar [...]

Why? If the data are staying on HP-UX, then
fbackup might make more sense. If the data
are going elsewhere, you need to have a
compatible "pax" or "tar" at the other end.

The original "tar" archive format was quite
limited in several ways: file size, name
length, ... It's been extended repeatedly,
but there are many "tar" programs which can't
deal with all the extensions.

[Bill Hassell]

tar is a very old legacy program that was never designed for large files. Originally, it would not backup anything larger than 2GB. It was then patched to handle 8GB (for the POSIX 1-1988 standard) but this capability may make the resultant backup incompatible with other versions of tar. Additionally, tar has no recovery capability in case the tape has a defect and restoring just one file near the end of the tape can take hours.

As mentioned, fbackup is the only tool that can backup and restore any file of any size. fbackup is proprietary to HP-UX, but any commercial backup program such as Data Protector or Legato will also be proprietary -- meaning that you must have the same program on the destination machine.

When you backup large amounts of data, the risks of a failure and the effort required can be enormous, so you need a modern backup program that can truly protect all that data. So if you are making a backup for use on the same machine or another HP-UX box, you can use the HP supported fbackup program. This tool is designed for tapes and can use high speed search to recover files very quickly. It is also much, much faster than any legacy backup program like tar or cpio due to multiple reader processes. And unlike most freeware backup programs, fbackup has many reliability features including the ability to recover from bad spots on the tape and preventing overwrite of tapes in a multi-tape backups. You'll need to create the following config file for DDS, DLT or LTO drives:

Contents of the config-file:
============================

blocksperrecord 4096
records 64
checkpointfreq 4096
readerprocesses 6
maxretries 5
retrylimit 5000000
maxvoluses 200
filesperfsm 2000

---------------------------------------------

Example for a complete backup starting at / (root):

fbackup -i / -v -c config-file -f /dev/rmt/0m

Display the tape header with dates:

frecover -V - -f /dev/rmt/0m

Display the table of contents:

frecover -I - -f /dev/rmt/0m

If you are exchanging the data with a non-HP-UX system, you'll have to purchase a commercial backup program. You might also look at GNU TAR but be careful as there are many versions and most do not support file sizes larger than 2GB or 8GB. You need a version that supports the POSIX 1-2001 standard.

[Steven Schweda]

> tar is a very old legacy program [...]

The "tar" supplied with HP-UX is.

> fbackup is the only tool that can backup
> and restore any file of any size.

And what sort of files can "pax" or GNU "tar"
_not_ handle?

> [...] on the tape [...]

Did the questioner say anything about tape?
The command shown did not look to me as if it
were using tape.

> It is also much, much faster [...]

For the command shown here?

Some people sure are generous with words like
"only", "have to", "must", and so on.

[Patrick Wallek]

According to the pax man page (on my HP-UX 11.11 system with the tar patch that increases tar's limit to 8GB), pax will not support files larger than 2GB.


WARNINGS
Because of industry standards and interoperability goals, pax does not support the archival of files larger than 2GB or files that have user/group IDs greater than 60K.


So pax is not a good solution here either.

GNU tar may work, but it would be wise to install it on the destination system as well to insure compatibility.

I agree that if you are moving data between HP-UX system, fbackup/frecover is a better tool to use.

[Patrick Wallek]

ON HP-UX 11.11, if you have PHCO_28414, or any superceding patch (most recent for 11.11 is PHCO_35998), then pax can handle up to 8GB files as well.

On HP-UX 11.23 you need patch PHCO_31634 for pax to support up to 8GB files.

[Murat SULUHAN]

Hi

I am not sure gnu tar and 8 GB limits
Have you ever tried ?

Best Regards
Murat

[Steven Schweda]

> I am not sure gnu tar and 8 GB limits

I believe ...

1. That the 8GB limit in "tar" comes from a
size field which is limited to 11 ASCII octal
digits, and 0o100000000000 = 8G. (Of course
it looks like a bad design decision now.)

2. That for sizes >= 8GB, GNU "tar" switches
over to a binary value in the same field
(which is a _whole_ lot of bits). (Tricks
like this tend to confuse older "tar"
programs which expect to find only ASCII
octal digits there.)

> Have you ever tried ?

Not that I can recall, but my curiosity may
be aroused enough to see if I have enough
free disk space on a UNIX system to give it a
try. (Examining the source code would
probably be faster, but might be less
convincing.)

[Steven Schweda]

sol# uname -a
SunOS sol 5.10 Generic_120011-14 sun4u sparc SUNW,Ultra-60

sol# gtar --version
tar (GNU tar) 1.19
[...]

That's the current one.

sol# ls -l /usr/gt8GB
-rw-r--r-- 1 root root 8589938688 Nov 10 13:49 /usr/gt8GB

Using Sun's "tar":

sol# ( cd /usr ; tar cf - gt8GB ) | gzip -c > /usr/gt8GB.tgz
tar: gt8GB too large to archive. Use E function modifier.

sol# ( cd /usr ; tar cfE - gt8GB ) | gzip -c > /usr/gt8GB.tgz

So, with a bit of coaxing, Sun's "tar" can
handle a file bigger than 8GB.

sol# ls -l /usr/gt*
-rw-r--r-- 1 root root 8589938688 Nov 10 13:49 /usr/gt8GB
-rw-r--r-- 1 root root 16661102 Nov 10 14:15 /usr/gt8GB.tgz

Here's how GNU "tar" handles the Sun "tar"
archive:

sol# pwd
/usr1

sol# gzip -dc /usr/gt8GB.tgz | gtar xf -

sol# ls -l /usr/gt8GB /usr1/gt8GB
-rw-r--r-- 1 root root 8589938688 Nov 10 13:49 /usr/gt8GB
-rw-r--r-- 1 root root 8589938688 Nov 10 13:49 /usr1/gt8GB

sol# diff /usr/gt8GB /usr1/gt8GB
diff: /usr/gt8GB: Value too large for defined data type

sol# /usr/local/bin/diff /usr/gt8GB /usr1/gt8GB
/usr/local/bin/diff: memory exhausted

sol# cmp /usr/gt8GB /usr1/gt8GB
sol#

Well, at least _someone_ could compare the
things.

One more time, using GNU "tar" to make the
archive:

sol# ( cd /usr ; gtar cf - gt8GB ) | gzip -c > /usr/gt8GB_gtar.tgz

sol# ls -l /usr/gt*
-rw-r--r-- 1 root root 8589938688 Nov 10 13:49 /usr/gt8GB
-rw-r--r-- 1 root root 16661102 Nov 10 14:15 /usr/gt8GB.tgz
-rw-r--r-- 1 root root 16661011 Nov 10 16:31 /usr/gt8GB_gtar.tgz

So, very similar, but not identical,
archives.

sol# rm /usr1/gt8GB

sol# gzip -dc /usr/gt8GB_gtar.tgz | gtar xf -

sol# ls -l /usr/gt8GB /usr1/gt8GB
-rw-r--r-- 1 root root 8589938688 Nov 10 13:49 /usr/gt8GB
-rw-r--r-- 1 root root 8589938688 Nov 10 13:49 /usr1/gt8GB

sol# cmp /usr/gt8GB /usr1/gt8GB
sol#

That's not all the permutations, but I'm
satisfied. I assume that GNU "tar" on HP-UX
would do as well on such data.

If you're looking for a version of VMSTAR
(for Alpha or IA64, but not VAX) which can
handle files this big, just let me know.

[Bill Hassell]

> And what sort of files can "pax" or GNU "tar"
_not_ handle?

Files larger than 8GB per the pax man page. The GNU tar documentation is unclear about POSIX 1-2001 as it does not associate this standard with a capability in a particular version. The release history seems to be silent about supporting files larger than 8 GB: http://www.gnu.org/software/tar/

>> [...] on the tape [...]

> Did the questioner say anything about tape?
The command shown did not look to me as if it
were using tape.

For tar, cpio and pax, they have almost no understanding of tape drives so whether the output is a tape or a disk file, the limitations apply equally. I should have used the term output file.

> It is also much, much faster [...]

For the command shown here?

It all depends. For real tape drives, yes. For a local disk, it will probably be faster assuming that the system is busy. On a quiescent system and separate source/destination disks, it's probably the same. fbackup spawns multiple reader processes to mitigate delays in opening files.

[Steven Schweda]

> The GNU tar documentation is unclear [...]

I thought that my demonstration was pretty
clear, but I could be wrong.

[whiteknight]

Zaid,

Have your installed tar latest patch ?
Please install
PHCO_36587 11.11 tar(1) cumulative patch


SR:8606279411 CR:JAGae43464 )
tar(1) from PHCO_26423 fails to backup files of size greater
than or equal to 4GB. It fails with the following message:
#tar cvf 4gb.tar 4gb-file
tar: Size of 4gb-file > 8GB. Not dumped.

WK