Using Large Files (>2GB) with Perl

[Nate Arnold]

Greetings,

I am having some difficulty working with files larger than 2GB with Perl 5.8.5 (which I installed via a depot from the HP-UX Porting and Archive Center). I don't have any problems manually moving files with FTP nor do I have problems loading these files into Oracle but when I try to use Perl things fail. These scripts same scripts work successfully with any file that is smaller than 2GB. I suspect that this problem is related to how Perl was compiled but I am not sure.

Thanks

[Steven E. Protter]

I doubt its a perl problem.

Its more likely a problem with the filesystem.

if its not set up for largefiles fsadm must be used to covert it.

A newly created filesystem needs to have this set up explicitly

newfs -F vxfs -o largefiles

SEP

[Nate Arnold]

If I can manually FTP files larger than 2GB in size wouldn't that mean that the largefiles is enabled? My /etc/fstab files mounts that filesystem with largefiles enabled so I don't think that is it.

/dev/clarvg/clarity /clarity vxfs delaylog,largefiles 0 2

[Nate Arnold]

Here is the output from the fsadm command:

zeus:/root # fsadm -F vxfs /dev/clarvg/rclarity
largefiles

[Steven E. Protter]

Yes it does mean largefiles is enabled.

What exact perl command are you having issues with?

A code snippet would help.

SEP

[HGN]

Hi

The command you have used shows that it supports largefiles, this might be related to perl, what are you trying to execute in perl.

Rgds

HGN

[Nate Arnold]

Unfortunately the Perl code is owned by the application vendor and I am not able to give you any thing without their permission. There are telling me that I need to recompile Perl with the following flag:

-D_FILE_OFFSET_BITS=64

According to the documentation on the HP-UX Porting and Archive Center their version of Perl was compiled with the above flag removed with the comment that it wasn't needed.

I really don't want to have to go through the trouble of compiling when I don't even know if that is the problem.

I can say that these scripts have worked fine for years with the same files on a Tru64 server.

[Noel Miranda]

Look at the following snippet from
www.xav.com/perl/lib/Pod/perlhpux.html

Using Large Files with Perl
---------------------------
Beginning with HP-UX version 10.20, files larger than 2GB (2^31) may be created and manipulated. Three separate methods of doing this are available. Of these methods, the best method for Perl is to compile using the -D_FILE_OFFSET_BITS=64 compiler flag. This causes Perl to be compiled using structures and functions in which these are 64 bits wide, rather than 32 bits wide.

There are only two drawbacks to this approach: the first is that the seek and tell functions (both the builtin version and the POSIX module's version) will not correctly function for these large files (the offset arguments in seek and tell are implemented as type long). The second is that any extension which calls any file-manipulating C function will need to be recompiled using the above-mentioned -D_FILE_OFFSET_BITS=64 flag. The list of functions that will need to recompiled is: creat, fgetpos, fopen, freopen, fsetpos, fstat, fstatvfs, fstatvfsdev, ftruncate, ftw, lockf, lseek, lstat, mmap, nftw, open, prealloc, stat, statvfs, statvfsdev, tmpfile, truncate, getrlimit, setrlimit

[H.Merijn Brand (procura]

I'm pretty sure that flag is already on, since it is made default for HP-UX builds long ago.

You can check that with the -V (capital V) option:

a5:/u/usr/merijn 102 > perl -V | grep 64
use64bitint=undef use64bitall=undef uselongdouble=undef
cc='cc', ccflags =' -Ae -D_HPUX_SOURCE -Wl,+vnocompatwarnings -I/pro/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 ',
a5:/u/usr/merijn 103 >

or for builds created with GNU gcc:

a5:/u/usr/merijn 103 > gperl -V | grep 64
use64bitint=undef use64bitall=undef uselongdouble=undef
cc='gcc', ccflags ='-mpa-risc-2-0 -D_HPUX_SOURCE -DDEBUGGING -fno-strict-aliasing -pipe -I/pro/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
a5:/u/usr/merijn 104 >

Or even shorter:

a5:/u/usr/merijn 104 > perl -V:ccflags
ccflags=' -Ae -D_HPUX_SOURCE -Wl,+vnocompatwarnings -I/pro/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 ';
a5:/u/usr/merijn 105 > gperl -V:ccflags
ccflags='-mpa-risc-2-0 -D_HPUX_SOURCE -DDEBUGGING -fno-strict-aliasing -pipe -I/pro/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64';
a5:/u/usr/merijn 106 >

Enjoy, Have FUN! H.Merijn

[Nate Arnold]

Unfortunately I don't see the

-D_FILE_OFFSET_BITS=64

flag when I run a perl -V.

zeus:/clarity/data # perl -V |grep 64
use64bitint=undef use64bitall=undef uselongdouble=undef
zeus:/clarity/data # perl -V:ccflags
ccflags='-D_HPUX_SOURCE -mpa-risc-2-0 -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE';

[H.Merijn Brand (procura]

Then you probably have the HP porting center's distribution. They deliberately remove that flag (read http://hpux.connect.org.uk/hppd/cgi-bin/wwwtar?/hpux/Languages/perl-5.8.6/perl-5.8.6-ss-11.11.tar.gz+perl-5.8.6/HPUX.Install+text ):

--8<---
* hints/hpux.sh:
- Remove -D_FILE_OFFSET_BITS=64 from compiler flags (not needed).
-->8---

Any way, you're most likely safe there.

I'm currently talking to the porters over there to get to a situation that reflects more, and hopefully better, defaults.

Enjoy, Have FUN! H.Merijn

[Nate Arnold]

I received a compiled version of Perl that contains the flag for largefiles from our application vendor.