topic Re: File size limit exceeded in Operating System - Linux

File size limit exceeded

Tonatiuh — Sat, 03 May 2008 04:06:28 GMT

I compressed a file into a "tar.gz" format. this was done in a Red Hat Linux 3. Original size of the uncompressed file: 17884012544 bytes.

I transfered that file to a Red Hat Linux 4 and tried to uncompress it, but I recive the following error message:

"File size limit exceeded"

The error was raised when the filesize become: 17247252480.

Any idea?

Re: File size limit exceeded

Alexander Chuzhoy — Sat, 03 May 2008 06:31:11 GMT

This link should help you:
http://www.cyberciti.biz/faq/file-size-limit-exceeded-error-under-linux-and-solution/

Re: File size limit exceeded

skt_skt — Sun, 04 May 2008 14:36:54 GMT

that link did not work. Could you put a note if you remember what is the fix?

Re: File size limit exceeded

Alexander Chuzhoy — Mon, 05 May 2008 05:59:08 GMT

The link worked just fine:

$ ulimit -a

Output:

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) 5000
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
max rt priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 2047
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimitedThe above output clearly stat that you can create file size upto 5MB limit. To change this limit or if you do not wish to have a limit you can edit your /etc/security/limits.conf file (login as the root):
# vi /etc/security/limits.conf
Look for your username and fsize parameter. Delete this line or set new parameter. For example consider following entry where I am setting new file size limit to 1 GB:

vivek hard fsize 1024000Save the changes. Log out and log back in for the changes to take effect.

Now your limit is 1GB file size. If you do not want any limit remove fsize from /etc/security/limits.conf.

Discussion on This FAQDeepan.R Says:
April 28th, 2007 at 11:02 am
Hi,

I too facing the File size limit exceeded error under Linux. But as per the above suggested solution, the ulimit is not setted for the user root, in my case. But still Im getting the error while I connect my USB Hard Drive and try to copy files. No file can be copied more than 4.1GBâ ¦I need to copy some 23 GB files and I still have the space in USB.

Kindly inform me what can I doâ ¦.Mean while, I am aslo searching the web for a solution.

Regards,
Deepan.

Chris Says:
September 5th, 2007 at 5:21 pm
Deepan.R:

This is probably due to the filesystem that your hard drive is formatted to. Each filesystem has a file size limit of its own. Chances are that the hard drive is formatted as FAT32, which has a 4GB max filesize. If you need it to be able to deal with larger files, consider reformatting to ext3(with a limit of between 16GB to 2 TB)) or something else.

Mani Says:
September 20th, 2007 at 2:41 pm
Hi,
I tried as per your suggestion.The explanation is very good. It worked well for me. Thanks a lot.

Regards,
Manigandan

Purna Says:
April 24th, 2008 (2 weeks ago) at 3:37 am
try this command.
#dd if=/dev/zero of=/filesize bs=1024 count=xxxx // to create large file.
If this helps you to create expected filesize then filesystem/os does not limits you.

To create this sort of filesize you have to use LFS. becoz bydefault 32bit compilation is able to address at most 2^31 bytes(2GB).
1. open file with O_LARGEFILE flag orâ ed with other falgs.
2. compile your code with -D_LARGEFILE_SOURCE -D_LARGEFILE_SOURCE64 -DLARGEOFFSET_BITS=64

Re: File size limit exceeded

Tonatiuh — Mon, 05 May 2008 13:11:01 GMT

This is my ulimit output. The file size limit is unlimited.

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16384
virtual memory (kbytes, -v) unlimited
file locks (-x)
unlimited

Any other idea?

Re: File size limit exceeded

Tonatiuh — Wed, 07 May 2008 23:02:53 GMT

The link did not work. Fsize limit is unlimited with no success. Any other idea?

Re: File size limit exceeded

Phil Jakob — Fri, 16 May 2008 17:51:11 GMT

Are you using ext2/3 as a filesystem?
If these filesystems are created with a 1k blocksize they will have a filesize-limit of about 16G.

(...hope I am not mistaken here...)
-an ext2/3 with 1k blocksize can only allocate 16843020 blocks per file
maxnumberofblocks * blocksize = maxfilesize
16843020 * 1024 = 17247252480

to check your ext3 blocksize:
tune2fs -l /dev/XXXX | grep "Block size"

http://en.wikipedia.org/wiki/Ext3#Size_limits

Re: File size limit exceeded

PAVIC Thierry — Thu, 31 Mar 2011 12:20:21 GMT

The block size of your fs
use tune2fs -l in order know the block size use

tune2fs -l /dev/cciss/c0d0p3 |grep "Block size"
Block size: 4096

if you have 1024 the max file is 16G
So you have to use 4k block size --> use the option -b 4096 in mkfs order

Re: File size limit exceeded

Alzhy — Thu, 31 Mar 2011 17:36:05 GMT

Believe the ext3 Filesystem/Filesyze Table.

Only awy out of your predicament is to rebuild (mkfs.ext3) your filesystem with a bigger block size -- that is IF the restoted file is the lone occupant of the filesystem.