topic Re: wget file size limitation in Operating System - Linux

wget file size limitation

K.C. Chan — Thu, 30 Jan 2003 20:18:24 GMT

Does any one know if there is a filesize limitation with wget and if so, how do get around it? Is there a patch for it? I've got the following messages when trying to download a 3gbytes iso file. "2097100K -> .......... .......... .......... .......... ....File size limit exceeded (core dumped)". Thanks.

Re: wget file size limitation

K.C. Chan — Thu, 30 Jan 2003 21:30:34 GMT

the cmd is "wget -c --passive-ftp ftp://host/dir/file" where file is about 3gbytes. I've tried this with ftp via reget command, I got the same result. So I think this may be an ftpd file limitation. I am using wu-ftpd, does anyone have any idea? Thanks.

Re: wget file size limitation

Stuart Browne — Thu, 30 Jan 2003 22:35:32 GMT

Given the limited information you've given us here, I'm picking it's the obvious reason.

'wget' uses a pointer to a LONG to store where it's up to in a file.

Simply put, 2GB baby.

Unfortunately, their mailing lists are useless, and have no decent information in them.

Official home page (http://www.gnu.org/software/wget/wget.html) mentions nothing about a file size limit either.

And sorry, I don't feel like a source-code poking to confirm it, but it's what it seems.

Any particular reason you are using 'wget' instead of a real FTP client (NCFTP's 'ncftpget' command comes to mind) ?

Re: wget file size limitation

K.C. Chan — Thu, 30 Jan 2003 23:20:31 GMT

ftp bombs out as well when it reaches 2G. Any idea?

Re: wget file size limitation

Stuart Browne — Fri, 31 Jan 2003 00:20:28 GMT

which 'ftp' ?

There's the standard 'ftp' FTP client, then theres the 3rd party (but usually distributed) 'ncftp' packages.

There are a number of things on a Linux box that can do 'ftp'.

Also, what is the FTP server that's serving it out? It's possible that it has a limitation also, which might be worth looking into.

Re: wget file size limitation

K.C. Chan — Fri, 31 Jan 2003 01:09:46 GMT

I am using wu-ftpd-2.6.2-5 and running RH7.3. Filesystem quota is off and ulimit for user is unlimited. Any idea what ftp pukes on filesize ~3g? Thanks.

Re: wget file size limitation

K.C. Chan — Fri, 31 Jan 2003 01:18:30 GMT

I believe the ftp client is "ftp-0.17-13". Any idea what is causing ftp to bail out on me like this? Does any one of any ftp utils that would get ~3G across the internet without any hassle? Thanks.

Re: wget file size limitation

Stuart Browne — Fri, 31 Jan 2003 01:37:29 GMT

I don't know. Try 'ncftpget' (should be distributed with RH73).

See if it has the same issue.

Re: wget file size limitation

K.C. Chan — Fri, 31 Jan 2003 03:46:15 GMT

I did a man on ncftp, but I can't find something similar to "wget -c"? Is their an option for ncftpget to continue the interrupted download? Thanks.

Re: wget file size limitation

Stuart Browne — Fri, 31 Jan 2003 03:59:26 GMT

If you do 'man ncftpget', you should see the simple option of:

ncftpget host /local/path /path/to/file

That will retrieve that file. By default, it will attempt to resume.

You can also use FTP URL's:

ncftpget ftp://host/path/to/file

to do the same thing.

Re: wget file size limitation

K.C. Chan — Fri, 31 Jan 2003 17:20:00 GMT

Ok, by default it will resume the download. Does this mean if I interrupt the download, like control-c out of it. Then at a later point start the download again, will nfcftpget start the download where I left off or re-download the file from start? Or does it mean, it will resume the download on a per session basis (only re-sume download if you lost connetion and you haven't quit out of the session). Could someone shed some lights into this? Thanks.

Re: wget file size limitation

Stuart Browne — Sat, 01 Feb 2003 06:56:41 GMT

Ever thought of just trying it instead of asking first?

No, it's not session based. Yes, if you break out of it, it will continue where it left off.

Re: wget file size limitation

Stuart Browne — Sat, 01 Feb 2003 07:01:01 GMT

Ever thought of just trying it instead of asking first?

No, it's not session based. Yes, if you break out of it, it will continue where it left off.

Re: wget file size limitation

david_69 — Thu, 20 Feb 2003 18:54:02 GMT

what file system are you trying to write to?
Does the filesystem type support large files?
(Ext2 and Ext3 do not support files larger than 2GB).
You might try scp, or netpipes, netcat, or nfs.

Re: wget file size limitation

Stuart Browne — Thu, 20 Feb 2003 22:23:45 GMT

Umm, david, ext2 & ext3 most certainly do support files larger than 2GB.

The 2GB file-size limitation was a kernel limit, not an FS design limit.

The late 2.2 series, and 2.4 series of kernel support up to 2TB file sizes.