Too many open files : FIFO

[Macleod]

Hi
We are running Tomcat which after running for few days start generating 'Too Many open files exceptions'.. We ran lsof when this problem occurs and found a huge number(about 1000) of FIFO files.
Can you provide some info on what these FIFO files are ? Why do they get created ? And basically how to fix this problem.

Thanks in advance.

[Biswajit Tripathy]

FIFO (first in first out) are named pipes and used
for interprocess communication. They can be created by mkfifo(3C) library call.


- Biswajit

[Bill Hassell]

This may be a normal error message that occurs when a program needs an unusually high number of files opened at the same time. The kernel parameter: maxfiles controls this limit and is usually 60. However, this can be increased with ulimit or inside the program with setrlimit up to maxfiles_lim which defaults to 1024 in the kernel. I would change maxfiles_lim to 2048 and see if the problem goes away. A program that needs thousands of files open at the same time is suspect, but perhaps you have 2000 web connections active at the same time? Note also that nfile, the global file count for the whole system may come into play after a while.

[Macleod]

we have tune the file limit to 4096, still happen the same problem. And we only have 10-20 connection at one time. It looks the FIFO can't release correctly, how to solve it ?

[Bill Hassell]

Call your programmers. Tomcat is probably dong what it was told to do which may the result of Java code or just application bugs. Nothing you can do as a sysadmin except to shutdown and restart the web server.

[Macleod]

Same application in other Platform(Solaris , RH Linux) don't have this problem. So we don't know how to find the root cause and solve it.

[Bill Hassell]

Are the corresponding parameters on the other platforms set higher? If you run lsof on the other platforms, how many files do they have open? If it is significantly less then you probably have a bug in the HP-UX version. Start with patches, especially threads and networking, Apache/Tomcat and Java.

Is the kernel parameter maxfiles=4096 or just maxfiles_lim=4096? If maxfiles is lower, use ulimit to bump the ulimit -n value to 4096 and then start Tomcat. If that works OK, change the startup script for Tomcat to add the ulimit value permanently. As mentioned, programs can call setrlimit to accomplish this but you may not have any control over the code.

[Macleod]

We have use ulimit to set nofiles to 4096, so Tomcat will generate "Too many open files" exception after over 4096.

In other system, I only can see 1~5 FIFO files.
But in HP-UX will over 1000, it looks the process never release FIFO files. Anyway, I will focus on the patch.

Thanks