java error (Too many open files (errno:24))

[Mark Peiffer]

Ever since we upgraded to the new java (1.4.2.12 for the DST change) on our HP-UNIX boxes we have been experiencing this error about once every 1 to 2 days: 4/18/07 3:54 PM webroot: Servlet error
java.io.FileNotFoundException:/u08/networks/de01sbr2/niku/clarity/webroot/ui/evolution2/images/WdgError.gif (Too many open files (errno:24))

The above error is from the app-orion.log file. The application we use with Orion is Clarity (Niku). The current band-aid fix is to stop and restart the application daemon. After working with the vendor (Niku Clarity) and looking around on the internet we were told to increase the MAXFILES on the HP-UNIX systems. We increased it from 2048 to 4096. This has not fixed the issue. Now the vendor is telling us that we need to set MAXFILES to unlimited. The HP-UNIX admin said this can not be done. Is this true? I'm not even sure that's the problem. This all started with the DST patch. The java home has been rebuilt and still we have this error. Is it possible that the java is not “releasing” file descriptors? Has anyone else seen this error?

[Mark Peiffer]

In case anyone needs this info for the above issue:

$ uname -a
HP-UX de01sbr1 B.11.11 U 9000/800 2917260238 unlimited-user license
$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 3802048
stack(kbytes) 392192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 4096

[A. Clay Stephenson]

The errno you are getting is definitey EMFILE (too many per-process files) as opposed to the ENFILE (too many system-wide files). I assume that you increased both the hard and soft maxfiles kernel tunables (maxfiles and maxfiles_lim). You can test this with:
ulimit -nS # soft
ulimit -nH # hard

you can't set this to unlimited but it can be increased well above your current value although nfile (the system-wide limit) will probably also need to be increased. One way to tell if files are being closed and file descriptors are being re-used is to run tusc against the executable. The open system call returns the current file descriptor. If a close is done then an open should return the lowest available file descriptor. If your file descriptors simply increase without ever reusing lower number then you have an application problem.

[Bill Hassell]

Your vendor was only half right. The parameters are MAXFILES and MAXFILE_LIM which are system-wide limits for each process. A properly written and documented program for Unix will:

1. Inform the sysadmin how many files will be needed, especially when more than a few dozen files will be opened.

2. If the limit is reached (errno 24) then the program will report the current number that are open and the desired additional files that are needed.

3. If the files include network ports, these numbers will also be reported.

4. The program will call setrlimit to increase the current value of MAXFILES.

Most likely few if any of the above design features were included in your programs. Notice that I said that the program can increase MAXFILES? The setrlimit system call should always be used by such programs. Sure, you could set MAXFILES to 4096 or even 195,000 or some other ridiculous number but that parameter is system-wide. Now every user on the system can write a 20 line C program and open all the file descriptors available on your system (the NFILE kernel parameter) and dent root the ability to login.

That's why there are two paramaters: MAXFILES, which every user gets, and MAXFILES_LIM which is the hard limit for all processes. MAXFILES is usually set to 100 or 200 and MAXFILES_LIM set to a reasonably large number. Then problem programs are limited but properly written prgrams can increase their limit as required.

In case you program(s) do not use setrlimit, you can add a wrapper script around the program to use ulimit to adjust the MAXFILES value on behalf of the program. The admin is correct: there is no kernel parameter unlimited for the file descriptor count. You can raise it with ulimit -n but only to MAXFILES_LIM.

[Mark Peiffer]

Is TUSC equivalent to LSOF?

I can't find either on the box. I will have to ask the admin to install one.

[A. Clay Stephenson · Accepted solution]

No, lsof and tusc are both very useful and very distinct tools. You can get them from any of the HP-UX Porting Centre's. Lsof is used to list open files and tusc is used to trace every system call of a process. You will see every open(),write(),read(),close(),socket(), .... system call along with the value returned and the arguments supplied to the system call.

http://hpux.its.tudelft.nl/hppd/cgi-bin/search?package=on&description=on&term=lsof

http://hpux.its.tudelft.nl/hppd/hpux/Sysadmin/tusc-7.8/

[Bill Hassell]

If you can troubleshoot the Java code, start here:

http://www.hp.com/products1/unix/java/?jumpid=go/java

Definitely look at the measurement and optimization tools. Also check that the code is not in runaway mode with thousands of threads.

[Mark Peiffer]

I want to thank you for your help. I had LSOF installed and now can see the list of descriptors. I noticed a disturbing repeating trend (same node same name) of descriptors. Does this point to Java or the Application (weblogic) using the Java (or both)?

[Peter Nikitka]

Hi,

I saw this behaviour some time ago at Solaris and it was the application, which did not release filehandles correctly.

mfG Peter

[Bill Hassell]

Java code, especially prepackaged routines, have consistently shown big problems in handling garbage collection and file handles. I don't why Java is so notorious in this area but all sysadmins agree that once Java starts running on your system, CPU usage jumps way up, threads can jump way up and file handles are massive. If you wrote the code, then look at the Java Developer's webpage, specifically and HPJMeter and HPJtune:

http://www.hp.com/go/java

[Matti_Kurkela]

Based on the file names like".../resource/projmgr_nls_cs.properties", here's my initial guess:

A Java software component named something like "projmgr" is using several .properties files, probably for localization or language selection.

The trouble is, instead of reading the properties files once while the application is deployed or when the application server (Weblogic?) is started up so that each projmgr thread will use the same properties, each thread will read the .properties files independently. If the content of those .properties files does not change very often, this is wasteful and just silly.

If the developer of the projmgr component is competent with Java application servers, he/she is likely to have a major "duh" moment when seeing that attachment of yours.

If that does not happen, the developer might need to increase his/her knowledge about the handling of "properties". The Java language and the Weblogic server both have built-in means to handle cases like this in a sensible way: they just need to be used correctly by the application code. If you need to be able to make any changes the properties files take effect while the program is running, that can apparently be arranged too. So there is no need to re-invent this particular wheel.

(I'm not a Java programmer, but we had a similar problem once...)

MK

[Mark Peiffer]

I just want to thank you guys for your help. After using the LSOF command to collect data the vendor agreed there was a problem and has opened a bug-fix request. They also said another company is experiencing the same issue.

You guys were quick and knowledgeable!

[Mark Peiffer]

I just want to say “thank you” for your help. After using the LSOF command to collect data the vendor has opened a bug-fix request. They also said another company is experiencing the same issue.

You guys were quick and knowledgeable!