topic support files > 2 GB in Operating System - Linux

support files > 2 GB

Gerd Koenig — Thu, 07 Aug 2003 08:17:48 GMT

Hi,
we're running a x86 based server with SuSE 8.1.
Now we have to copy a file > 2GB to this machine. This original file is splitted into 500MB pieces, and now joining fails with the message "file too large".
Seems like limit 2GB. The partition is formatted with reiserfs3.6.2, Kernel 2.4.19

What do we have to do to support files larger than 2 Gig ??

thanks in advance ..GERD..

Re: support files > 2 GB

Sergejs Svitnevs — Thu, 07 Aug 2003 08:25:55 GMT

SuSE 8.1 Linux supports LFS (Large File Support). Check out the following link: http://www.suse.de/~aj/linux_lfs.html

Regards,
Sergejs

Re: support files > 2 GB

Jerome Henry — Thu, 07 Aug 2003 08:34:17 GMT

Hi,

Bad news is... you can't. The way the kernel is written limits the file size on reiserfs to 2GB (a little more on ext3). The next kernel, 2.6, is said to be supporting terabytes files, but while on 2.4.x, you get the max size.

There is a much better way of doing this copy job. And that is to use the multi-volume option of dump. A further advantage of dump over dd is that:
(1) The data can be restored to a disk of different size or type.
(2) Any file system or disk problems won't be replicated into the backup.
(3) Partition sizes may be changed (like to make an FS bigger or smaller) during the restore.
(3) Files/dirs can be selectively extracted from the backup.

The only disadvantages of dump over an image backup, like dd, is that you have to create file systems and swap space and re-install a bootloader after restore if you wind up replacing a failed disk drive. Since that's trivially easy with RedHat by booting off the CD in rescue mode I don't see it as a significant disadvantage.

The basic command to dump a large (greater than 2Gb) file system to files on another disk would look like:

dump 0jMf /path-to-backups/fs-name fs

This will create a series of files, each 2Gb in size, on /path-to-backups named like fs-name001, fs-name002, etc. , (M option). The data written to the to the files will be compressed (j option). It will be a full backup (0) option. Dump only operates on a file system so if your system had file systems of /, /var, /home you'd need to do something like:

dump 0jMf /backups/root /
dump 0jMf /backups/var /var
dump 0jMf /backups/home /home

As an enchancement, you can also use dump to do incremental backups after doing a full. This makes the incremental backups quite fast as only data changed since the last lower numbered backup need to be saved.

Take a look at the man pages for dump and restore for more information.

hth

J

Re: support files > 2 GB

Jerome Henry — Thu, 07 Aug 2003 08:39:43 GMT

Mmmm... be careful on LFS, it's indeed a good idea, but test it on a non production machine, as you have to re-compile your kernel, and many programs that will use these files must be tested and sometimes re-compiled...

:]

J

Re: support files > 2 GB

Sergejs Svitnevs — Thu, 07 Aug 2003 09:50:40 GMT

Release 7.1 and newer of SuSE Linux supports LFS and you do not need to recompile kernel.

Regards,
Sergejs