Too many open files error

[Shivkumar]

Dear Sirs/Madam;
We are getting errors in weblogic managed server log file as "java.net.SocketException: Too many open files". We are running WebLogic 7.x on HPUX 11i.

We are thinking to increase the value of maxfile_lim from 2048 to 12,288. Will this fix the too many open files issue ?

Thanks,
Shiv

[Pedro Cirne · Accepted solution]

Hi,

Concerning open files you have:

-maxfile_lim which sets the hard limit for the number of files a process is allowed to have open simultaneously

-maxfiles which sets the soft limit for the number of files a process is allowed to have open simultaneously

-nfile which defines the maximum number of files that can be open simultaneously, system-wide, at any given time

Probably you error is due to nfile parameter, run you program and simultaneously, collect the following and post the output:

#sar -v 5 10

Enjoy :)

Pedro

[morganelan]

If you have exceptions pointing to having too many open files, you might have an operating system resource problem.file defines the maximum number of open files. The maxfiles value is the soft-file limit per process, and maxfiles_lim is the hard-file limit per process.I think you must try first to see the impact of that value you provide.

[morganelan]

On HP we can also use the performance-monitoring tool Glance to analyze the total number of files open when running the WebLogic Server:

COMMAND PID USER FD TYPE DEVICE
SIZE/OFF NODE NAME
in.telnet 29705 root 2u inet
0x30002808fd8 0t76 TCP
aaaaabbbb:telnet->
abcdef.bea.com:3886 (
ESTABLISHED)


This example shows a socket in a CLOSE_WAIT state:

COMMAND PID USER FD TYPE DEVICE
SIZE/OFF NODE NAME
java 545 weblogic 24u IPv4
0x30002a4cea8 0t0 TCP
abcd:7001->xyz. com:12345
(CLOSE_WAIT)

As long as the socket remains in this state, its associated file descriptor will exist. If many sockets are in this state, then the process can run out of file descriptors. TCP sockets enter in this state when its peer (other side of the connection) sent a FIN. No timeout can be set to force the closure of these sockets, so they are just waiting for the local application to call the close() function.

[Shivkumar]

Pedro Cirne;

Here is out of the $ sar -v 5 10 command:
$ sar -v 5 10

HP-UX jasmine B.11.11 U 9000/800 08/19/05

08:14:09 text-sz ov proc-sz ov inod-sz ov file-sz ov
08:14:14 N/A N/A 128/8192 0 1263/90000 0 1528/90010 0
08:14:19 N/A N/A 128/8192 0 1263/90000 0 1532/90010 0
08:14:24 N/A N/A 128/8192 0 1263/90000 0 1533/90010 0
08:14:29 N/A N/A 128/8192 0 1263/90000 0 1532/90010 0
08:14:34 N/A N/A 128/8192 0 1263/90000 0 1531/90010 0
08:14:39 N/A N/A 128/8192 0 1263/90000 0 1531/90010 0
08:14:44 N/A N/A 128/8192 0 1263/90000 0 1530/90010 0
08:14:49 N/A N/A 128/8192 0 1263/90000 0 1533/90010 0
08:14:54 N/A N/A 128/8192 0 1263/90000 0 1533/90010 0
08:14:59 N/A N/A 128/8192 0 1263/90000 0 1533/90010 0

Thanks,
Shiv

[Pedro Cirne]

Hi,

Did you have the error ""java.net.SocketException: Too many open files" when collecting sar statistics?

From the sar statistics you post, it seems that you don't have any problem with nfile, 1533 out of 90010 as maximum.

Do you have Glance?

For tunning Java, you also have a very good tool you can get from:

http://www.hp.com/products1/unix/java/java2/hpjconfig/index.html

Enjoy :)

Pedro

Pedro

[Mel Burslan]

Shiv,

Increasing maxfiles, maxfiles_lim or nfile kernel parameters will not do much help to you at this point. if you look at the one before the last column of the sar output, it tells you your files utilization is nowhere near the limit.

Did you talk to BEA support ? What did they say about this error message. Is this coming form a home developed java module or something BEA provided under Weblogic ? As you may or may not know, not all the error messages actually mean what the real error is. So, talk to the app support people first is my suggestion.

[Bill Hassell]

From your sar -v listing, you have many times over the number of files and processes (nfile=90,010 and nproc=8,192) needed--at least at the moment you ran the sar command. Was this sar measurement made when the system was having problems? If so, your nfile and nproc values are excessively high. BUT your ninode setting is far, far too large. ninode controls the number of cache entries for HFS filesystems. You probably exactly one: /stand. Change ninode to 2048 and leave it alone.

So the error message isn't coming from nfile. Next is maxfiles and maxfiles_lim. Changing maxfiles_lim will not change anything if the Java code and/or WebLogic fails to call setrlimit() to change the current setting for maxfiles. So I would change maxfiles to 8192 and also change maxfiles_lim to 8192. Remember, this parameter is for a SINGLE program that wants to open more than 2048 files at the same time--a very unusual situation.

What about maxuprc? If all the processes run under a particular user, say weblogic, then the default of 75 processes is far too low and may need to be bumped up a lot. Check this with:

ps -ef | sort | cut -c-78

This collects all the processes that run under the same user together so you can count them. If maxuprc is 75 and any of the processes for a single user exceed 65-70, then maxuprc is way too low.

And since Java and WebLogic can use a lot of threads, you'll have to adjust max_thread_proc and nkthread.

[Shivkumar]

Bill Hassell;

Is it recommended that maxfiles_lim should be equal to or greater that maxfiles ?

Thanks!
Shiv

[Ranjith_5]

Hi Shiva,

It is not mandatory that you have to keep both the values same. These are quite often set too low. Bump maxfiles and maxfiles_lim to 200 if they're not already 200, and higher on Web and file servers (2000 or above ). Bump them up if you encounter "Too many open files" program failures. For 11.11, maxfiles_lim was one of the new â dynamic tunables,â which means
you can bump it with the kmtune command...


on my machine, which is a web server the values are as follows.

maxfiles 2048
maxfiles_lim 60000

Regards,
Syam

[Yogeeraj_1]

hi shiv,

Bill Hassell has already answered that question at the followin URL:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=909129


You can set the values incorrectly (maxfiles greater than maxfiles_lim) and the kernel silently enforces the lower limit. maxfiles is simply a starting point, normally set quite low (60 to 120) for ordinary users. Power programs that really need hundreds of files open at the same time will call setrlimit() to change the limit higher. maxfiles_lim is the absolute limit for the number of open files in a single process. Or you can change the limit (just for your shell environment) using ulimit -n.

NOTE: ksh does not implement all the options in ulimit (no options at all for the last 19 years) but with a recent patch now adds ulimit -a and the options listed, but not ulimit -n. Now as far as the actual values: you really have programs that open as many as 2000 files at the same time? Not one at a time, but all of them open at once? The Universe database system can be setup for something like that (it uses thousands of files) but *every* instance of the database program requires that many aditional file handles. If you run 100 copies with 2000 files open at the same time for each process, your nfile kernel parameter must be increased by 20,000.

Mnay, many vendors publish rather simplistic instructions like: maxfiles=maxfiles_lim=2048, probably created by a newbie sysadmin that couldn't figure out which parameter to change and just took a whack at all of them. There is a specific purpose in having maxfiles smaller than maxfiles_lim. By making them both large, any user can write a stupid program that loops around creating thousands of files, eventually using all the kernel's file descriptor slots. At that time, no one login (including root!) or do anything new. It's called a denial of service.


hope this helps!
kind regards
yogeeraj

[Shivkumar]

Bill Hassell;

Couple of more questions:

We have 2 different settings on the different servers such as 2048, and 12288 ?

Do you feel 12288 too high and 2048 is too less?

Thanks!
Shiv

[Bill Hassell]

There is no way to now without asking the WebLogic vendor. You can certainly set the maxfiles and maxfile_lim to 12000 and see what happens, but if it works OK, I would demand an explanation from WebLogic why their programs open so many files. I can't imagine the complexity of code that must keep track of so many files at the same time. NOTE: open files are also network sockets, but the same questions exist. Any application that fails to document massively large kernel parameters needs a reevaluation as to suitability in a production environment. I would also be concerned about support if this information is difficult to obtain.

Setting the maxfiles and maxfiles_lim values extremely large do not affect the kernel's size. These 2 values are simply fences that limit program requests. Indirectly, an application that needs a large number of files will indeed affect the nfile value (and thus, the kernel's size)

[Shivkumar]

Bill Hassell;

Most of the product vendor support folks are not unix gurus and not aware of how reputed unix such as hpux
works in great depth.

We are using weblogic cluster which uses 4 managed servers in each cluster and each managed server is basically a jvm (in our case jdk131_09). I was thinking to count the number of open file connection by using the command lsof - p pid (of the jdk131_09).

I have planned to take the sum of all opened files connection for all jvm. Should this total number(i.e. sum) not increase not increase maxfiles or maxfiles_lim ?

Thanks for continued explanation and help.

Shiv

[Bill Hassell]

You wrote:

"Most of the product vendor support folks are not unix gurus and not aware of how reputed unix such as hpux works in great depth."

This is very scary. I would be seriously worried that your vendor has never really tested WebLogic on HP-UX. Now I know that WebLogic is widely used on HP-UX so you need to escalate your question to WebLogic support management, asking for either senior support people who understand basic Unix kernel parameters (maxfiles and maxfiles_lim are ecerywhere in the Unix world), or ask management to get the testing lab involved.

The question about the number of files needed by the application is NOT a Unix question, it is an application question. If the application writers don't know, I would be very concerned about whether this will work in your environment or what you will do if there is a problem and app support cannot help you.

Yes, you can count files by hand using lsof and that will give you an idea of what is happening. Here's the story on maxfiles versus maxfiles_lim. maxfiles is a starting point and is generally set low, perhaps 50 to 150. That way, some programmer that makes a mistake in a program and tries to open millions of files will fail at maxfiles. But a programmer that understands basic Unix environments will be familiar with the Unix system call: setrlimit() With this call, the programmer can set the maximum number of open files in the current program to a much larger value. So the programs that need extra files opened can request the larger number with setrlimit. Like all system calls, setrlimit will return an error code if the limit cannot be set as high as needed.

So I would set maxfiles to about 50-150 and set maxfiles_lim to 12000 and see what happens. If the application fails, use lsof to watch the programs that open lots of files. If they fail, call WebLogic and ask why their programs do not use setrlimit(). You can help troubleshoot their code by calling setrlimit yourself before starting the WebLogic code. The method is to use ulimit -n (only available in the POSIX shell). Find out where the WebLogic code is started, then (assuming it is started in a script) insert ulimit -n 5000 to increase the maxfiles value to 5000 for the programs.

Or you can simply set maxfiles = maxfiles_lim and set both to 12000. Not a good sysadmin technique but it may get you past the problem that shoud have been answered by your vendor.

[Shivkumar]

Bill Hassell;

I didn't mean to say that WebLogic vendor has never tested the WebLogic Product on HPUX. Of course they must have done
it.

But i have come to know with my own experience that it is not necessary that whoever works at vendor support area should
be knowing hpux in great detail. I believe that hpux and aix are regarded as world leader in the unix world in their respective areas. I have not escalated the current issue with weblogic vendor yet.

I also believe that having product knowledge and unix operating system expertise at a time are considered a
rare skill set. Unix takes years to master it. Such kind of skill sets are available with few people like you.

Thanks,
Shiv