CLOSE_WAIT Sessions

[KapilRaj]

Hi,

I am facing excessive close_wait sessions on my apache server and tomcat. Is there any specific reason to it?.

Is there any tcp tunable parameter which would forcefully close such sockets ?. I am suspecious abt the application (apache & tomcat) and my application team is working on it.

What i really need to know is, Is there anything a SysAdmin can do to overcome close_waits sessions ?.

Thanks in advance,

Kaps

[Massimo Bianchi]

Hi,
there are many tunable paramenter, you can get a list of there with the tool

ndd -h supported
ndd -h unsupported


will shot you the list of parameters.

HTH,
Massimo

[Bill McNAMARA_1]

see the notes about ndd dynamic parameters here:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xc38585079106d71190050090279cd0f9,00.html

also a good explaination of the behaviour is discussed.

Later,
Bill

[Michael Steele_2]

I think what you're looking for is a way to avoid rebooting blocked tcp ports after they change to a CLOSE_WAIT state, and the 'tcp_discon' is the parameter in 'ndd' used for this. Like so:

ndd -set /dev/tcp tcp_discon
tcp_discon

This thread provides the complete procedure with some warnings:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xa100237a4bc6d611abdb0090277a778c,00.html

A sysadmin can also obtain a copy of 'lsof' and issue this command to further evalutate the problem:

lsof -i tcp:####
lsof -i udp:####

Where #### is the port number in the CLOSE_WAIT state.

'lsof' is an extremely powerful trouble-shooting tool that should be in every sysadmins tool box. Obtain it here:

http://hpux.cict.fr/hppd/hpux/Sysadmin/lsof-4.64/

[KapilRaj]

Have you guys ever experienced close_wait prob with apache server ?. If yes what was the fix applied ?. As per our application team , the packets going through apache at our site is heavily loaded , Could that be a reason ?.

Cheers .......... Kaps

[Ron Kinner]

What is the device on the other end? WinNT has a bug in the tcp/ip stack which causes it to mess up the close if a packet gets lost. Usually it leaves you stuck in FIN_WAIT_II and them in LastAck tho.
There is a patch for it but you have to beg MS for it. They never bothered to release it. I have it if you want to try it. It's down at the bottom of my geneology website labeled TCP/IP Patch. http://www.geocities.com/heartland/9670/

You might want to check your network to make sure you are not dropping packets. That's how you usually get stuck in odd states.

Also make sure you have the latest ARPA patches. One of them gives you the ability to ndd the fin_wait_II hold time and it may have other virtues too. We are talking HPUX 11 or better aren't we?

Ron

[rick jones]

CLOSE_WAIT is the state into which a TCP connection transitions when it has received a FIN (graceful connection close request) from the remote. In this state, TCP is waiting for the application to call close() or shutdown(). If this file descriptor was shared between processes, it is waiting for the last one to call close() - ie the reference count to go to zero.

As for "excessive" I suspect that an HP-UX system could handle thousands and thousands of them without problem. The only problem would be the loss of file descriptors in the applciations, and even if you use the massive tcp_discon kludge (which one should almost _never_ use...) the socket (file descriptor) will still be allocated until the application calls close. So, you still need to get the application to call close()...

If the problem is application restart, then the fix is to make sure the application is setting SO_REUSEADDR before trying to bind().

[W.C. Epperson]

Rick's right about what close_wait is there for, but not all applications and tcp kernel code act quite right.

Is this occurring over SSL? Does the SSL handshake look broken? Like mod_ssl is trying to shake hands with a client, without receiving a client helo?

[W.C. Epperson]

There's an Oracle 9IAS bug related to hung close_waits under Apache, with a related patch.

It appears there's a related bug in Top Hat under Apache.

Since there was no follow up to previous post, I presume the problem spontaneously disappeared.