Too many open files

[Praveen Bezawada]

Hi
We are running a java application which after running for few hours start generating 'Too Many open files exceptions'.. We ran lsof when this problem occurs and found a huge number of STSO files with the name /dev/tcp->tcpm->ip. The count of these files is increasing all the time and they never seem to get closed. Slowly reaching the maxfiles_lim limit when it starts to give these exceptions. I looked for some information for these files but could only find that it's a stream between the tcp and ip layer.
Can you guys provide some info on what these files are ? Why do they get created ? And basically how to fix this problem.
Thanks in advance.
...BPK...

[Steven E. Protter]

These applications need to open a lot of datafiles.

Consider increasing nfiles maxfiles_lim.

Perhaps collect some performance data(script attached) while running this. You may see some high water hits that will guide you through what kernel parameters you need.

SEP

[Praveen Bezawada]

Hi
We have set maxfiles and maxfiles_lim has 15000 !!! and lsof shows 15067 files open.. Out of which over 12000 are /dev/tcp->tcpm>ip files.
Based on our appl design, we use Java NIO , there is no way we can go that far as we have hard check at 10000 after which the application drop closing sockets.
I think some where some thing is going wrong causing these files to open and not close.
May be too much load !!!

...BPK...

[doug mielke]

Could this be a connection problem? Is the java only a local app? If so, could there be something wrong in a domain / hostname setting, causing the connection to be broken in 'mid-stream'?

[Helen French]

Read this document (TKB #KBRC00012980):

http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000069403094

I checked the bug database in Sun site, but couldn't find the document there. May be it needs a developer support agreement to view it.

[Ron Kinner]

Does
netstat -an
show a lot of connections in FIN_WAIT_II?

Ron

[Praveen Bezawada]

No these connection are not accounted for in the netstat -an

[Michael Steele_2]

a) What I/O ports involved?

b) Got processes waiting on I/O that never's fulfilled?

c) Did you measure the activity on these ports?

# lsof -i tcp:### (* Where ### is the port number *)

# lsof -i udp:###

[Michael Steele_2]

Try this :

http://forums1.itrc.hp.com/service/forums/parseCurl.do?CURL=%2Fcm%2FQuestionAnswer%2F1%2C%2C0xd2781cc6003bd6118fff0090279cd0f9%2C00.html&admit=716493758+1068749052266+28353475