topic Re: Filesize limitations using ux2dos in Operating System - HP-UX

Filesize limitations using ux2dos

Charlie Shore — Wed, 30 Jan 2008 16:18:06 GMT

When attempting to execute the ux2dos command on HP UX 11 created files that are > 2 gb, I'm getting an error indicating that the execution of ux2dos cannot find the input file. Files less than 2 gb are found and processed without any problems. Is anyone aware of a input file size limitation when using the ux2dos command or if there are undocument parms to deal with large files?

Thanks

Re: Filesize limitations using ux2dos

Dave Hutton — Wed, 30 Jan 2008 16:20:10 GMT

It may be the file you are creating is having issues. Unless you have largefiles turned on the filesystem you are creating a new file it will fail.

If you don't have largefiles turned on, your limited to 2 gigs in file size.

Re: Filesize limitations using ux2dos

Robert-Jan Goossens — Wed, 30 Jan 2008 16:24:45 GMT

Charlie,

have a look at this doc.

http://www11.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c00936500-2

Title: dos2ux and ux2dos do not support large files greater than 2gb
Document ID: emr_na-c00936500-2

Regards,
Robert-Jan

Re: Filesize limitations using ux2dos

Hein van den Heuvel — Wed, 30 Jan 2008 16:43:15 GMT

Hmmm,

Can you hide the large file in a pipe?
Does this work?
# cat dosfile | dos2ux | cat > unixfile ?

Given to volume of data to fix up, maybe you want to consider revisiting the source and/or target of this data.

Maybe the source can be tought not to add the extra carriage-return at line end. For example by using ASCII mode for an FTP.

Maybe the target can be tought to ignore the extra character?

And maybe you can just roll your own filter, possibly with perl:
For example

perl -pe 's/\r$//; last if /^\032$/' dos > unix

Re: Filesize limitations using ux2dos

Steven Schweda — Wed, 30 Jan 2008 16:45:27 GMT

On some UNIX(-like) systems, large files are
invisible to programs built without
large-file support. This would seem to be
the case here.

A more tolerant approach, which would have
been useful here, would be to go ahead and
try to use the file, and wait for seek/tell
operations to fail (invisibly). That
approach can cause file corruption when a
seek wraps around the 2GB limit, leaving the
program looking at the front of the file when
it should have been somewhere in the middle.
Which, I assume, is why some vendors chose to
fail at the open, before any damage is done.

One solution is to get the source code, and
build the program with large-file support.
For a program like this, which (I'd guess)
does no seek/tell operations, it should be
pretty easy.

If the program can be used as a filter, which
would also seem to be true here, then
redirecting standard input and/or output can
work around a limitation like this. That is,
a command like:

ux2dos < in_file > out_file

may work where this:

ux2dos in_file > out_file

would fail, if in_file is large.

As usual, showing the actual commands you
used, and their actual output, might have
been helpful here.

Re: Filesize limitations using ux2dos

Arturo Galbiati — Thu, 31 Jan 2008 10:11:22 GMT

Hi,
to bypass this you can use:

# In lieu of 'dos2ux'
perl -pi.old -e' s!\r\n!\n!;s!\032!! if eof' file.in

# ...and in lieu of 'ux2dos'
perl -pi.old -e 's!\n!\r\n!s;END{print "\032"}' file.in

Origianl file will save as .old

HTH,
Art

Re: Filesize limitations using ux2dos

Torsten. — Thu, 31 Jan 2008 10:17:13 GMT

I'm just curious about this:

"dos2ux, ux2dos â convert ASCII file format"

Do you have **ascii** files larger than 2GB?

Is this any kind of a DB dump?

Re: Filesize limitations using ux2dos

Hein van den Heuvel — Sun, 03 Feb 2008 05:17:02 GMT

Arturo wrote>> # In lieu of 'dos2ux'
perl -pi.old -e' s!\r\n!\n!;s!\032!! if eof' file.in

Word of warning... this will NOT produce the same results as dos2us.
I believe that perl I showed in a reply a day is also subtly different.

The problem here is that the real dos2ux stops at the first ^Z, not just at eof.
The problem with mine it that it does not output all chars befor the ^Z.
CLoser:
perl -pe 's/\r$//; if (/(.*?)\032/) {print $1."\n"; last}' x.dos > tmp.tmp

Also.. iirc 'eof' is a little expendsive in that it gets the potential eof byte, checks it, and ungets it.

Hein.