topic Re: CLOSE_WAIT Sessions in Operating System - HP-UX

CLOSE_WAIT Sessions

KapilRaj — Thu, 12 Jun 2003 08:21:47 GMT

Hi,

I am facing excessive close_wait sessions on my apache server and tomcat. Is there any specific reason to it?.

Is there any tcp tunable parameter which would forcefully close such sockets ?. I am suspecious abt the application (apache & tomcat) and my application team is working on it.

What i really need to know is, Is there anything a SysAdmin can do to overcome close_waits sessions ?.

Thanks in advance,

Kaps

Re: CLOSE_WAIT Sessions

Massimo Bianchi — Thu, 12 Jun 2003 08:27:04 GMT

Hi,
there are many tunable paramenter, you can get a list of there with the tool

ndd -h supported
ndd -h unsupported

will shot you the list of parameters.

HTH,
Massimo

Re: CLOSE_WAIT Sessions

Bill McNAMARA_1 — Thu, 12 Jun 2003 08:31:29 GMT

see the notes about ndd dynamic parameters here:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xc38585079106d71190050090279cd0f9,00.html

also a good explaination of the behaviour is discussed.

Later,
Bill

Re: CLOSE_WAIT Sessions

Michael Steele_2 — Thu, 12 Jun 2003 11:01:44 GMT

I think what you're looking for is a way to avoid rebooting blocked tcp ports after they change to a CLOSE_WAIT state, and the 'tcp_discon' is the parameter in 'ndd' used for this. Like so:

ndd -set /dev/tcp tcp_discon
tcp_discon

This thread provides the complete procedure with some warnings:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xa100237a4bc6d611abdb0090277a778c,00.html

A sysadmin can also obtain a copy of 'lsof' and issue this command to further evalutate the problem:

lsof -i tcp:####
lsof -i udp:####

Where #### is the port number in the CLOSE_WAIT state.

'lsof' is an extremely powerful trouble-shooting tool that should be in every sysadmins tool box. Obtain it here:

http://hpux.cict.fr/hppd/hpux/Sysadmin/lsof-4.64/

Re: CLOSE_WAIT Sessions

KapilRaj — Fri, 13 Jun 2003 06:20:44 GMT

Have you guys ever experienced close_wait prob with apache server ?. If yes what was the fix applied ?. As per our application team , the packets going through apache at our site is heavily loaded , Could that be a reason ?.

Cheers .......... Kaps

Re: CLOSE_WAIT Sessions

Ron Kinner — Fri, 13 Jun 2003 13:33:35 GMT

What is the device on the other end? WinNT has a bug in the tcp/ip stack which causes it to mess up the close if a packet gets lost. Usually it leaves you stuck in FIN_WAIT_II and them in LastAck tho.
There is a patch for it but you have to beg MS for it. They never bothered to release it. I have it if you want to try it. It's down at the bottom of my geneology website labeled TCP/IP Patch. http://www.geocities.com/heartland/9670/

You might want to check your network to make sure you are not dropping packets. That's how you usually get stuck in odd states.

Also make sure you have the latest ARPA patches. One of them gives you the ability to ndd the fin_wait_II hold time and it may have other virtues too. We are talking HPUX 11 or better aren't we?

Ron

Re: CLOSE_WAIT Sessions

rick jones — Fri, 13 Jun 2003 16:54:47 GMT

CLOSE_WAIT is the state into which a TCP connection transitions when it has received a FIN (graceful connection close request) from the remote. In this state, TCP is waiting for the application to call close() or shutdown(). If this file descriptor was shared between processes, it is waiting for the last one to call close() - ie the reference count to go to zero.

As for "excessive" I suspect that an HP-UX system could handle thousands and thousands of them without problem. The only problem would be the loss of file descriptors in the applciations, and even if you use the massive tcp_discon kludge (which one should almost _never_ use...) the socket (file descriptor) will still be allocated until the application calls close. So, you still need to get the application to call close()...

If the problem is application restart, then the fix is to make sure the application is setting SO_REUSEADDR before trying to bind().

Re: CLOSE_WAIT Sessions

W.C. Epperson — Fri, 13 Jun 2003 17:16:57 GMT

Rick's right about what close_wait is there for, but not all applications and tcp kernel code act quite right.

Is this occurring over SSL? Does the SSL handshake look broken? Like mod_ssl is trying to shake hands with a client, without receiving a client helo?

Re: CLOSE_WAIT Sessions

W.C. Epperson — Thu, 03 Jul 2003 11:32:58 GMT

There's an Oracle 9IAS bug related to hung close_waits under Apache, with a related patch.

It appears there's a related bug in Top Hat under Apache.

Since there was no follow up to previous post, I presume the problem spontaneously disappeared.