topic Re: File table full and system down in Operating System - HP-UX

File table full and system down

nill_3 — Mon, 25 Aug 2003 07:27:53 GMT

Hello consultants,

One of our client running HP-UX and Oracle Java application server. The memory reduce continually and system down with syslog of 'file table full'. Without any help by add nfile kernel para to 2000.

Following is part of the system log file:
Jul 31 14:15:24 hpux vmunix: : table is full
Jul 31 14:27:35 hpux vmunix: file: table is full
Jul 31 14:27:35 hpux above message repeats 215594 times
Jul 31 14:27:35 hpux vmunix: file: table is full

Any hints?
Thanks very much.
Regards,
Nill

Re: File table full and system down

Robert-Jan Goossens — Mon, 25 Aug 2003 07:32:31 GMT

Hi Nill,

You have to increase the value of the NFILE kernel parameter. You can use sam to increase this parameter.

Robert-Jan.

Re: File table full and system down

Steve Steel — Mon, 25 Aug 2003 07:34:46 GMT

Hi

Increase nfile to the double of now.
reboot.

Then you can use glance to determine which process which has the most files open.

Steve Steel

Re: File table full and system down

nill_3 — Mon, 25 Aug 2003 07:37:38 GMT

Hi Robert,

Thanks for your info.

My client has increased the kernel parameter from 900 to 2000. It seems not many help. And as my understanding, file table full may not result the Operating System down. Maybe some other reason cause this.

Any thing we can do to troubleshooting it?

Regards,
Nill

Re: File table full and system down

Michael Tully — Mon, 25 Aug 2003 07:41:07 GMT

Firstly you should work out if is a formula or value.

The easiest way is:

# kmtune -l -q nfile

If is is a formula you'll probably find it is tied heavily in with 'maxusers', so this would be the parameter that would need to be increased, not 'nfile' directly.

Re: File table full and system down

Robert-Jan Goossens — Mon, 25 Aug 2003 07:51:18 GMT

Hi,

http://www5.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000064128770

The error message vmunix file: table is full, means the maximum number of open files has been exceeded. Check above doc.

Check your /stand/system file if your new value has been set.

Mine is,

db1:/stand# strings system | grep nfile
nfile (15*NPROC+2048)
db1:/stand# strings system | grep nproc
nproc ((MAXUSERS*3)+64)
db1:/stand# strings system | grep maxusers
maxusers 200

((((200 x 3) + 64)) x 15) + 2048) = 12008

Hope it helps,

Robert-Jan.

Re: File table full and system down

nill_3 — Mon, 25 Aug 2003 08:33:24 GMT

Hi Steeve & Michael,

your info are very helpful and I will try.

But it is steel amazing file table full will cause OS down. Bugs maybe :-)

Regards,
Nill

Re: File table full and system down

sdip — Mon, 25 Aug 2003 10:35:39 GMT

Nill,

2000 is too small for Oracle Java application servers. I would advise for 30000. You can check the NFILE utilization by sar -v command. It would be better if you can post existing kernel parameters then we may advised you more proactively.

Dip

Re: File table full and system down

nill_3 — Tue, 26 Aug 2003 01:14:56 GMT

Hi Dip,

Thanks. I will collect the kernel para from customer later.

Regards,
Nill

Re: File table full and system down

Bill Hassell — Tue, 26 Aug 2003 01:55:40 GMT

Just a note: HP-UX, like all flavors of Unix, is critically dependent on files in order to run. Every time a file is opened, an entry must be made in a table in the operating system. If all the file entries are occupied, not even root can login because login needs to open several files, so yes, HP-UX will appear to be down. This is not a bug, it is a feature.

Now in the above error message, it as been repeated 215,000 times in one minute and that is definitely not normal. Some process is trying to open a file continuously rather than terminating normally. A simple shell script created by a novice user (does not have to be root) could devastate your system in this manner. Or it may just be java threads going crazy because of a programming error. Idf changing nfile to 30,000 (and regenerating the kernel and reboting) does not help, you definitely have a runaway program or script that needs to be terminated.

Re: File table full and system down

nill_3 — Tue, 26 Aug 2003 03:01:56 GMT

Bill,

You definitely have conviced me. We will try enlarge the kernel para first. If not work, it seems the problems is some program are trying to open files crazy. Then we try to use glance find which process are doing this.

Thanks,
Nill