TCP Setup: Socket() failed: Too many open files

[Sudeesh]

Dear Experts,

Could you pls tell me what parameter I should tune to get rid of messages like this in syslog:

Server[10607]: 499259: TCP Setup: Socket() failed: Too many open files

Thanks in Advance

Sudeesh

[James R. Ferguson]

Hi:

You need to evaluate 'maxfiles' or possibly 'nfile'.

Use 'kctune' (>= 11.23) or 'kmtune' (11.11) to see your current kernel settings.

Better than a textual message would be the actual 'errno' value or mnemonic, such as ENFILE (#23) or EMFILE (#24). The later just so happens to match "too many open files" which points to 'maxfiles'.

Regards!

...JRF...

[whiteknight]

Sudeesh,

tune up the nfile kernel parameter.

WK

[Jeeshan]

Hi Sudeesh

to view the number of open files

#sar -v 5 5

[Kapil Jha]

Generally these parameters are big enough to handle various application.
what are values of current nfile??
You amy check using lsof wchi process had these open connectiona and try to recycle that processes.
If this happenns again then probably you need to change the kernel paramater.
BR,
Kapil

[Fabien GUTIERREZ]

you sould consider looking to maxfiles_lim value, if you're using 11.23 it's quite easy with kcusage if you are using 11.11 you can use lsof -p pid to see open ressources and be able to count them
this could comes from a java program not closing some descriptors in it's

[Sudeesh]

Thank you for your replay.

This is a socks jumpstation and lot of network connections go through this box. This is the only message I get from syslog. I dont have any error numbers. When this situation arise, socks authentication stops working. Then we switch the cluster pkg to alternate node to get working.

[Sudeesh]

Current kernel values:

maxfiles 512
nfile 12259

maxfiles_lim 1024

[rick jones]

As suggested you could try an lsof on the socks server(s) to get their open file counts. You could also do something similar via glance, and also check the global file descriptor limit.

Another option would be to obtain and use tusc against the socks server binaries - it will show you the actual errno's for the socket() calls being made, and will even give you (IIRC) the mnemonics.

Other thoughts:

The documentation for the socks server may explain the error message.

If "Too many open files" is a common error message from strerror (iirc) then it may be possible to reverse map that to an errno and so knowledge of whether it is nfile or maxfile/maxfile_lim being hit.

[James R. Ferguson]

Hi (again):

> Rick: If "Too many open files" is a common error message from strerror (iirc) then it may be possible to reverse map that to an errno and so knowledge of whether it is nfile or maxfile/maxfile_lim being hit.

In fact, this is how I concluded that 'maxfiles' is the likely culprit:

# grep -i "too many open files" /usr/include/sys/errno.h
#define EMFILE 24 /* Too many open files */

Then, examining the mapages for 'socket(2)' yields:

" [EMFILE] The per-process descriptor table is full."

Regards!

...JRF...

[Sudeesh]

Thank you for all the suggestions.

I am closely monitoring the server now with lsof and netstat o/ps.

Will assign you points shortly


Sudeesh

[Sudeesh]

I worked with response center and finally concluded this as an issue with application. As a workaround increased maxfiles and maxfiles_lim and added monitoring for these values so that I can restart the application when it is close to the max limit

[Sudeesh]

sa