Sockets ..

[rleon]

My app team wrote:

The behavior we see is a steady increase in open files (lsof -p | wc -l) until it reaches the current max of 2048. At that point, the process must be bounced. It looks like the "open files" are sockets (from looking at lsof -p ).

There are two servers .. servera and serverb. Same config. So we thought. There where some kernel changes made to sync the systems up but that did not change the issue.

I checked ndd settings and can not find anyting related to socks.

Am I missing something?

[A. Clay Stephenson]

It would really help if there was a question here but I think the answer is that you are hitting the kernel tunable maxfiles --- which also applies to sockets. Man 2 socket for details. The answer may to increase maxfiles (which may also require an increase in maxfiles_lim) but it may also be to fix your code. It sounds as though you are calling socket() many times without doing corresponding shutdown()'s or close()'es.

[Steven E. Protter · Accepted solution]

Shalom,

maxuprc

Maximum number of files per user. Default is 75, thats easy to hit.

nfile
nproc
maxfile_lim

SEP

[Sandman!]

The application should issue a shutdown(2) socket system call to close the current connection alongwith the socket descriptor before it opens a new one? If that is not the case then the app team needs to modify their code accordingly.

[rleon]

maxfiles_lim is set to 2048.
But here is the kicker .. it is set to 2048 on both servers. But one still have the issue with the sockets and one doesnt.

same hardware, same build image, same kernel parameters.

[A. Clay Stephenson]

There are 2 limits for the same value that do the soft and hard limits respectively, maxfiles and maxfiles_lim. Maxfiles_lim should be > maxfiles. The application actually hits maxfiles and that will result in errno on socket being set to EMFILE. Well-written applications will then attempt to increase maxfiles (up to a maximum of maxfiles_lim). In any event, you can't simply compare two servers unless their loading is almost identical.

[rick jones]

One possible translation:

Our application is leaking file descriptors.

(sockets are file descriptors just as squares are rectangles)

The suggestion to call shutdown() is probably on the right track, but insufficient - shutdown() does not actually free the file descriptor. You have to call close() for that.

So, some tusc'ing of the server application to see what system calls it is making, and if it is properly responding to say read returns of zero (indicating client close) would seem to be inorder.

If the client's aren't closing on their own accord then perhaps the server application requires a means to cull older connections on its own.