File table full and system down

[nill_3]

Hello consultants,

One of our client running HP-UX and Oracle Java application server. The memory reduce continually and system down with syslog of 'file table full'. Without any help by add nfile kernel para to 2000.

Following is part of the system log file:
Jul 31 14:15:24 hpux vmunix: : table is full
Jul 31 14:27:35 hpux vmunix: file: table is full
Jul 31 14:27:35 hpux above message repeats 215594 times
Jul 31 14:27:35 hpux vmunix: file: table is full

Any hints?
Thanks very much.
Regards,
Nill

[Robert-Jan Goossens]

Hi Nill,

You have to increase the value of the NFILE kernel parameter. You can use sam to increase this parameter.

Robert-Jan.

[Steve Steel]

Hi


Increase nfile to the double of now.
reboot.

Then you can use glance to determine which process which has the most files open.


Steve Steel

[nill_3]

Hi Robert,

Thanks for your info.

My client has increased the kernel parameter from 900 to 2000. It seems not many help. And as my understanding, file table full may not result the Operating System down. Maybe some other reason cause this.

Any thing we can do to troubleshooting it?

Regards,
Nill

[Michael Tully]

Firstly you should work out if is a formula or value.

The easiest way is:

# kmtune -l -q nfile

If is is a formula you'll probably find it is tied heavily in with 'maxusers', so this would be the parameter that would need to be increased, not 'nfile' directly.

[Robert-Jan Goossens]

Hi,

http://www5.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000064128770

The error message vmunix file: table is full, means the maximum number of open files has been exceeded. Check above doc.

Check your /stand/system file if your new value has been set.

Mine is,

db1:/stand# strings system | grep nfile
nfile (15*NPROC+2048)
db1:/stand# strings system | grep nproc
nproc ((MAXUSERS*3)+64)
db1:/stand# strings system | grep maxusers
maxusers 200

((((200 x 3) + 64)) x 15) + 2048) = 12008

Hope it helps,

Robert-Jan.

[nill_3]

Hi Steeve & Michael,

your info are very helpful and I will try.

But it is steel amazing file table full will cause OS down. Bugs maybe :-)

Regards,
Nill

[sdip]

Nill,

2000 is too small for Oracle Java application servers. I would advise for 30000. You can check the NFILE utilization by sar -v command. It would be better if you can post existing kernel parameters then we may advised you more proactively.

Dip

[nill_3]

Hi Dip,

Thanks. I will collect the kernel para from customer later.

Regards,
Nill

[Bill Hassell]

Just a note: HP-UX, like all flavors of Unix, is critically dependent on files in order to run. Every time a file is opened, an entry must be made in a table in the operating system. If all the file entries are occupied, not even root can login because login needs to open several files, so yes, HP-UX will appear to be down. This is not a bug, it is a feature.

Now in the above error message, it as been repeated 215,000 times in one minute and that is definitely not normal. Some process is trying to open a file continuously rather than terminating normally. A simple shell script created by a novice user (does not have to be root) could devastate your system in this manner. Or it may just be java threads going crazy because of a programming error. Idf changing nfile to 30,000 (and regenerating the kernel and reboting) does not help, you definitely have a runaway program or script that needs to be terminated.

[nill_3]

Bill,

You definitely have conviced me. We will try enlarge the kernel para first. If not work, it seems the problems is some program are trying to open files crazy. Then we try to use glance find which process are doing this.

Thanks,
Nill