Server hangs after running for 2 to 3 days

[vsridhar]

I am facing a problem on HPUX ia64 11.23 platform. I have a server running on HPUX machine . The server hangs after running for two to three days.There are large number of CLOSE_WAIT connections observed on the server after the server when hang occurs. The server fails at accept() call with the error:"No Buffer Space available". I have tried changing the TCP parameters on the system but it didn't helped me in resolving the issue.

Any help in this regard would be very helpful for me.

[TTr]

How many of these do you have? Some CLOSE_WAITs can be tolerated and eventually they time out. If you are certain that the server hangs because of the CLOSE_WAITs, first check for any patches but in most cases this is the result of a misbehaving application. Find out which application (and it must be a very active application) is leaving these behind without disconnecting properly and see if there is an application fix. If you can't fix the application, search around for doing some more ndd tuning. These forums have a lot of ndd references.

[Steven E. Protter]

Shalom,

This is an application problem.

See about fixing the application, and closing obviously dead process with kill command issued by root.

SEP

[vsridhar]

Thank you responding to my query.

I have verified the netstat information and found that the connections from the server application are in CLOSE_WAIT state for a very long time. I have PHNE_35117 patch available on the machine. I have tried ndd tuning for TCP parameters available in the forum but was not successful. Please let me know what kind of application fixes would help me.

[TTr]

There is no application fix that we can give you. Which application is it? You need to contact the application vendor and tell them about the problem. They may give you a config setting within the app or a patch for the app.

[Shailendran V Naidu]

Hi Sridhar,
Looks like the application is opening up a Socket and not closing it.

Ex: In java

you can use the - Socket.setSoLinger()

to close the connection as soon as application ends.

This is purely programming.

[vsridhar]

We observe that when the server application was killed abnormally it leaves behind the sockets in CLOSE_WAIT state and and when the same application is launched immediately at the same port, it goes into hanged state.

[rick jones]

CLOSE_WAIT means the remote TCP endpoint has indicated it will be sending no more data by sending us a FINished segment. While this is then a 100% usable simplex (unidirectional) connection, 99 times out of 10 it means we are now waiting on the application to close the connection at this end as well.

While there may be a timer kludge in HP-UX networking, or even the unlikely possibility of a bug in the stack, chances are lots of CLOSE_WAITs are a bug in the application.

If there are enough of them, the application will bump-up against the per-process limit for open file descriptors and will be unable to open any more files or accept any more connections.

However, I would not have expected the accept() calls to start failing with ENOBUFS - I would have expected EMFILE. While it seems an odd error to return another 99 times out of 10 :) an ENOBUF on accept() means the remote client has given-up on the connection before the application has gotten around to calling accept().

[Krishnan Seshadri]

Hi Sridhar, Strangely enough we at Liz Claiborne is battling an exactly same issue with no avail yet from OS/Server side. The application we are using is for EDI, and the vendor is INOVIS (may be you have the same one??)

We migrated from PARISC a week ago, and this version of application is not supported on ITanium yet, so decided put through ARIES. As you have experienced we are in the same boat every two or three days, and the sympotom is indeed when there is heavy influx of transmissions.

We have been advised by a very informative JAVA resource from HP Support that the version of java 1.4.2.11 is the culprit, and have recommended 1.4.2.11a, which is the patched version. Did you hear any such info? Which version of Java you are running.