kernel table "file" is full

[Jeremy Shirling]

I recieved this error through openview's telalert "kernel table "file" is full" on one of my webservers. I found some info on IRTC relating to the "sar -v" command.
My question is this: can anyone tell me if this alert is related to this output (it is listed below). 2)is it something to be overly concerned about since, as of right now, it is a single incident?

Here is the sar -v output (the overflow happened at 14:00):

1:50:00 N/A N/A 238/2580 0 2876/2876 0 2003/4760 0
11:55:00 N/A N/A 237/2580 0 2876/2876 0 1993/4760 0
12:00:01 N/A N/A 238/2580 0 2876/2876 0 1997/4760 0
12:05:00 N/A N/A 237/2580 0 2876/2876 0 2036/4760 0
12:10:00 N/A N/A 237/2580 0 2876/2876 0 1983/4760 0
12:15:01 N/A N/A 239/2580 0 2876/2876 0 2021/4760 0
12:20:00 N/A N/A 237/2580 0 2876/2876 0 1982/4760 0
12:25:00 N/A N/A 236/2580 0 2876/2876 0 1993/4760 0
12:30:00 N/A N/A 241/2580 0 2876/2876 0 2078/4760 0
12:35:00 N/A N/A 242/2580 0 2876/2876 0 2073/4760 0
12:40:00 N/A N/A 235/2580 0 2876/2876 0 1984/4760 0
12:45:00 N/A N/A 238/2580 0 2876/2876 0 2033/4760 0
12:50:01 N/A N/A 237/2580 0 2876/2876 0 2013/4760 0
12:55:00 N/A N/A 241/2580 0 2876/2876 0 2025/4760 0
13:00:00 N/A N/A 241/2580 0 2876/2876 0 2061/4760 0
13:05:00 N/A N/A 234/2580 0 2876/2876 0 2076/4760 0
13:10:00 N/A N/A 237/2580 0 2876/2876 0 1980/4760 0
13:15:00 N/A N/A 240/2580 0 2876/2876 0 2041/4760 0
13:20:00 N/A N/A 247/2580 0 2876/2876 0 2106/4760 0
13:25:00 N/A N/A 240/2580 0 2876/2876 0 2013/4760 0
13:30:00 N/A N/A 247/2580 0 2876/2876 0 2060/4760 0
13:35:00 N/A N/A 246/2580 0 2876/2876 0 2095/4760 0
13:40:00 N/A N/A 237/2580 0 2876/2876 0 2034/4760 0
13:45:00 N/A N/A 243/2580 0 2876/2876 0 2053/4760 0
13:50:00 N/A N/A 240/2580 0 2876/2876 0 2044/4760 0
13:55:01 N/A N/A 250/2580 0 2876/2876 0 4175/4760 0
14:00:00 N/A N/A 243/2580 0 2876/2876 0 2871/4760 375
14:05:00 N/A N/A 243/2580 0 2876/2876 0 2111/4760 0
14:10:00 N/A N/A 248/2580 0 2876/2876 0 2092/4760 0
14:15:00 N/A N/A 244/2580 0 2876/2876 0 2078/4760 0
14:20:00 N/A N/A 251/2580 0 2876/2876 0 2089/4760 0
14:25:00 N/A N/A 245/2580 0 2876/2876 0 2044/4760 0
14:30:00 N/A N/A 250/2580 0 2876/2876 0 2152/4760 0
14:35:00 N/A N/A 243/2580 0 2876/2876 0 2075/4760 0
14:40:01 N/A N/A 246/2580 0 2876/2876 0 2214/4760 0
14:45:00 N/A N/A 247/2580 0 2876/2876 0 2108/4760 0

Thanks in advance.
Jeremy

[Patrick Wallek · Accepted solution]

It sounds like your nfile parameter may need to be increased.

Check /var/adm/syslog/syslog.log and see if you see and "file table full" messages there. If so, that points directly to nfile being too small.

[A. Clay Stephenson]

While you might think the ninode limit is being hit, based upon your sar -v output, it's really not a problem because, assuming almost all of your filesystems are vxfs, inode allocation is dynamic. You almost certainly hit the "nfile" limit. This could be caused by a runaway condition is some of your CGI code or extremey heavy demand. Although your current usage is less than 50%, it must rise considerably at some times. Nfile is a cheal (memory-wise) paramater so if you see values approaching your current setting (4760) then you should probably increase it. Typically, nfile, is coupled to a formula based upon the pseudovalue "maxusers" among other things that attempts to produce reasonable values for reasonable systems -- whatever that is. I would directly plug a numerical value in if you see that you are approaching the current setting.

[Jeff Carlin]

If it continues to happen, you may need to bump up nfile in the kernel. It could have been just a fluke with one of your processes though. Keep an eye on it.

[Jeremy Shirling]

Thank you all for your quick responses.
I will definately keep an eye on this.


Jeremy