Oracle - 'HP-UX error 27: file too large'

[Timothy Czarnik]

Hey all,

We got this output on our Oracle server that we were looking to put into production tomorrow, and I'm wondering if anyone out there could help me determine where this is coming from:

Mon Jun 10 07:38:09 2002
KCF: write/open error block=0x5215a online=1
file=13 /u02/oradata/prd1/WIT_CARE_DATA06_01.DBF
error=27072 txt: 'HP-UX Error: 27: File too large
Additional information: 336218'
Automatic datafile offline due to write error on
file 13: /u02/oradata/prd1/WIT_CARE_DATA06_01.DBF
KCF: write/open error block=0x792a9 online=1
file=12 /u02/oradata/prd1/WIT_CARE_DATA05_01.DBF
error=27072 txt: 'HP-UX Error: 27: File too large
Additional information: 496297'
Automatic datafile offline due to write error on
file 12: /u02/oradata/prd1/WIT_CARE_DATA05_01.DBF
KCF: write/open error block=0x54721 online=1
file=10 /u02/oradata/prd1/WIT_CARE_DATA03_01.DBF
error=27072 txt: 'HP-UX Error: 27: File too large
Additional information: 345889'
Automatic datafile offline due to write error on
file 10: /u02/oradata/prd1/WIT_CARE_DATA03_01.DBF
KCF: write/open error block=0x738c7 online=1
file=9 /u02/oradata/prd1/WIT_CARE_DATA02_01.DBF
error=27072 txt: 'HP-UX Error: 27: File too large
Additional information: 473287'
Automatic datafile offline due to write error on
file 9: /u02/oradata/prd1/WIT_CARE_DATA02_01.DBF

Any help will be appreciated greatly!

Tim

[Tom Geudens]

Hi,
Does your filesystem allow for largefiles (> 2Gb)? You can check this with
#fstyp -v
If f_flag is not 0, your filesystem supports largefiles. If not ... that's probably what the problem is. Activate largefiles on your filesystem.

Regards,
Tom

[harry d brown jr]

Tim,

What OS level are you running?

And like Tom mentioned, check to make sure you have largefiles turned on.

live free or die
harry

[Timothy Czarnik]

Tom,

I have verified that the filesystem supports largefiles:

/var/adm/cron # fsadm /u02
fsadm: /etc/default/fs is used for determining the file system type
largefiles

Thanks though!

TIm

[Timothy Czarnik]

Harry,

Sorry, I should have mentioned this.


We are running Oracle 8i on HP-UX 11i. We experienced this issue a week or 2 ago, and after that I went through the patches with a fine toothed comb, so I don't believe its going to be patch related (as Oracle and HP have both looked at our installed patches and have agreed that they looked ok).

I found a response in the forums when doing a lookup of 'error 27' that seems interesting. It brings up the idea that the /var/adm/cron/.proto file may be playing a part in this because Oracle is stopped and started via OmniBack. I don't know what this .proto file does, and am looking into this now. Anyone have any info on this?

As always, thanks a ton for your help!

Tim

[Tom Geudens]

Hi,
Well, if the problem is Omniback related I'll have to pass ... but just on the off-change that you did overlook this (probably not :-). Do the tablespaces auto-extend ? And is there room in the filesystems left ?

Like I said, probably not something you overlooked, but I thought I'd mention it.

Regards,
Tom

[Timothy Czarnik]

Tom,

The table spaces do not auto extend, and there is plenty of room in the filesystems.

Tim

[Kawah Cheung]

Hi,

Have you checked the ulimit? See attached file.

Regards,
Kawah

[Jeff Schussele]

Hi Timothy,

Kawah may be on to the problem here. Definitely check the ulimit for the user in question & remember that ulimit is in 512-byte blocks...so a 4Mb ulimit is only 2Gb size. May want to set an 8Mb ulimit size.

Rgds,
Jeff

[Jeff Schussele]

Hi (again) Timothy,

Another thing to consider is Oracle version...if this is 8.0.5 or earlier, they may support >2Gb DB filesizes but do NOT support exports > 2Gb. You have to compress.

Rgds,
Jeff

[Volker Borowski]

Hi Tim,

this looks like someone mounted your filesystems correctly upon installation and created everything allright, but did put the wrong entries into /etc/fstab.

I assume you rebooted the box and after the remount of the filesystems with the entries from fstab nothing works anymore.

Unmount filesystems, edit vfstb and add the "largefiles" option and remount.

Hope this helps
Volker

[Timothy Czarnik]

Hey all,

Here is where we are sitting. All filesystems were created correctly as largefile filesystems and mounted properly.

I am now concentrating on the file /var/adm/cron/.proto. This file is used when you run an at/cron job, and I believe it is also used when OmniBack runs a pre/post exec job. This passes some paramaters to the job, one of which is the ulimit value - $l (dollar el). With the .proto file unchanged, we created a 1 line script as oracle to pre-allocate a file greater than 2 Gb:

prealloc /u02/oradata/timtest 2300000000

When run from the command line, this works. WHen run via the 'at' command, this fails with the output sent to email:

prealloc: File too large

When we change the /var/adm/cron/.proto file to set the value of ulimit to unlimited:

/var/adm/cron # cat .proto
# @(#)B.11.11_LR
cd $d
ulimit unlimited
umask $m
$<

this script works. It looks as though we may be experiencing the same bug that Steve Steel mentions is this ITRC posting:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x37d791ccb36bd611abdb0090277a778c,00.html

If anyone has any additional info or ideas surrounding the file /var/adm/cron/.proto, I would love to hear them.

Tim

[Jeff Schussele]

Hi (again) Tim,

IF the ulimit of the user submitting the at job is large enough, you shouldn't have to set ulimit in the .proto file. Which user "owns" the script pid?
Also is the FS in which the submitting user is sitting set for largefiles? Where is the script owner sitting when the the script submits the at job - may want to cd to a *known* largefiles FS?
You may want to consider submitting the job to a *specific* queue (d-y) which means you'll have to specify a .proto.[d-y] file & explicitly specify the absolute time [[CC]YY]MMDDhhmm [ . ss ] to execute the job.
But if setting unlimited ulimit in the .proto file works.....why bother? You just need to be aware the ANY at job can create ANY filesize.

HTH,
Jeff

[Timothy Czarnik]

Jeff,

That was the interesting thing. Before we changed the .proto file, I inserted some calls to the /usr/bin/ulimit -a command to see what the exact ulimit was of the user that was attempting to run the jobs. In every single case, the system reported that the oracle user had an unlimited ulimit. However, whenever we ran jobs through at/cron/OmniBack, they failed with the 'file too large' error, which shows that the value of ulimit, while reported to be unlimited, was not unlimited. This seems to be a bug in the way the value of ulimit is reported. Once we changed the .proto file to specifically force the ulimit to be unlimited, we no longer experienced the error.

We will be leaving the .proto file the way it is, with the value of ulimit set to unlimited.

Tim

[Timothy Czarnik]

Hey all,

Just some final follow up.

As the user oracle, from the command line, I issue the command ulimit and get the response 'unlimited':

oracle on dbphprp1
/home/oracle> ulimit
unlimited

oracle on dbphprp1
/home/oracle>


As a final test of my rantings here, I edited the /var/adm/cron/.proto file to its original state, with the line ulimit $l (dollar el). I then submitted an 'at' job that only called the date command and checked its file in /var/spool/cron/atjobs. With the ulimit in the .proto file set to $l, the ulimit in my at job became this:

cd /home/oracle
ulimit 4194303
umask 22
/usr/bin/date

(these are the last 4 lines of my at job as found in /var/spool/cron/atjobs)

As you can see, the ulimit is now set to 4194303 512 byte blocks, which is 2 Gb. Remember now that previous to this, from the command line, the ulimit command stated that my ulimit was unlimited. I found this very interesting, and so did the next logical step. I edited the /var/adm/cron/.proto file and set the ulimit line to unlimited:

/var/adm/cron # cat .proto
# @(#)B.11.11_LR
cd $d
ulimit unlimited
umask $m
$<

and once again submitted an 'at' job. As expected, now the ulimit in the at job found in /var/spool/cron/atjobs listed an unlimited ulimit:

cd /home/oracle
ulimit unlimited
umask 22
/usr/bin/date

(these are the last 4 lines in the newly created at job).

So, even though the output of the ulimit command, when run from the command line, stated that my ulimit was unlimited, it was actually NOT unlimited when it came down to cron/at jobs (OmniBack pre/post execs included in this), but was set at the 2 Gb file size limit. ONLY by changing the ulimit value in the /var/adm/cron/.proto file was I able to submit a job that was not constrained (cronstrained? heh heh heh) by the 2 Gb filesize limit.

In my personal, professional opinion, this goes beyond the kooky stage, well into the arena of wackiness. Any info/insights are welcome!

Tim

[Kawah Cheung]

Hi,

Just browsing oracle metalink, and apparently, cron has its own environment settings, so what you set for the user has no relevance on cron jobs (apparently). See attached.

Rgds,
Kawah

[Timothy Czarnik]

Kawah,

Ahhh! Validation!

Thanks! :)

Tim

[Wayne Schmidt]

Tim,

We are seeing the same issue on HPUX 11.00 and Oracle 8.1.7.4. When files are going beyond 1gb (I stress one gig), the database is reporting:

*** SESSION ID:(2.1) 2003-12-01 00:57:14.611
KCF: write/open error block=0x10469 online=1
file=16 /ora/laxap01u.plaxsbo.u1000/oradata/plaxsbo/users_indexes_01.dbf
error=27072 txt: 'HP-UX Error: 27: File too large
Additional information: 66665'
Automatic datafile offline due to write error on

This is the second time this error has occured. 1st time on datafile, 2nd on index file.

.proto has ulimit set to unlimited. Oracle environment ulimit set to unlimited. Fstab is mounting the filesystem with largefiles option. mkfs reports largefiles. fstyp f_flag=16.