Filesize limitations using ux2dos

[Charlie Shore]

When attempting to execute the ux2dos command on HP UX 11 created files that are > 2 gb, I'm getting an error indicating that the execution of ux2dos cannot find the input file. Files less than 2 gb are found and processed without any problems. Is anyone aware of a input file size limitation when using the ux2dos command or if there are undocument parms to deal with large files?

Thanks

[Dave Hutton]

It may be the file you are creating is having issues. Unless you have largefiles turned on the filesystem you are creating a new file it will fail.

If you don't have largefiles turned on, your limited to 2 gigs in file size.

[Robert-Jan Goossens]

Charlie,

have a look at this doc.

http://www11.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c00936500-2

Title: dos2ux and ux2dos do not support large files greater than 2gb
Document ID: emr_na-c00936500-2

Regards,
Robert-Jan

[Hein van den Heuvel]

Hmmm,

Can you hide the large file in a pipe?
Does this work?
# cat dosfile | dos2ux | cat > unixfile ?

Given to volume of data to fix up, maybe you want to consider revisiting the source and/or target of this data.

Maybe the source can be tought not to add the extra carriage-return at line end. For example by using ASCII mode for an FTP.

Maybe the target can be tought to ignore the extra character?

And maybe you can just roll your own filter, possibly with perl:
For example

perl -pe 's/\r$//; last if /^\032$/' dos > unix

[Steven Schweda]

On some UNIX(-like) systems, large files are
invisible to programs built without
large-file support. This would seem to be
the case here.

A more tolerant approach, which would have
been useful here, would be to go ahead and
try to use the file, and wait for seek/tell
operations to fail (invisibly). That
approach can cause file corruption when a
seek wraps around the 2GB limit, leaving the
program looking at the front of the file when
it should have been somewhere in the middle.
Which, I assume, is why some vendors chose to
fail at the open, before any damage is done.

One solution is to get the source code, and
build the program with large-file support.
For a program like this, which (I'd guess)
does no seek/tell operations, it should be
pretty easy.

If the program can be used as a filter, which
would also seem to be true here, then
redirecting standard input and/or output can
work around a limitation like this. That is,
a command like:

ux2dos < in_file > out_file

may work where this:

ux2dos in_file > out_file

would fail, if in_file is large.

As usual, showing the actual commands you
used, and their actual output, might have
been helpful here.

[Arturo Galbiati]

Hi,
to bypass this you can use:

# In lieu of 'dos2ux'
perl -pi.old -e' s!\r\n!\n!;s!\032!! if eof' file.in

# ...and in lieu of 'ux2dos'
perl -pi.old -e 's!\n!\r\n!s;END{print "\032"}' file.in

Origianl file will save as .old

HTH,
Art

[Torsten.]

I'm just curious about this:

"dos2ux, ux2dos â convert ASCII file format"

Do you have **ascii** files larger than 2GB?

Is this any kind of a DB dump?

[Hein van den Heuvel]

Arturo wrote>> # In lieu of 'dos2ux'
perl -pi.old -e' s!\r\n!\n!;s!\032!! if eof' file.in

Word of warning... this will NOT produce the same results as dos2us.
I believe that perl I showed in a reply a day is also subtly different.

The problem here is that the real dos2ux stops at the first ^Z, not just at eof.
The problem with mine it that it does not output all chars befor the ^Z.
CLoser:
perl -pe 's/\r$//; if (/(.*?)\032/) {print $1."\n"; last}' x.dos > tmp.tmp

Also.. iirc 'eof' is a little expendsive in that it gets the potential eof byte, checks it, and ungets it.

Hein.