process has bean killed, but the tcp connection still established for long time.

[张磊2008]

In hp-unix, java program.
This process connection to a server(deploy on linux). After I kill this process, I can see the tcp connection still established by netstat.
$ netstat -a |grep 8850
tcp 0 0 10.1.70.21.51091 10.1.70.142.8850 ESTABLISHED
tcp 0 0 10.1.70.21.58809 10.1.70.142.8850 ESTABLISHED
tcp 0 0 10.1.70.21.53875 10.1.70.142.8850 ESTABLISHED
tcp 0 0 10.1.70.21.56228 10.1.70.142.8850 ESTABLISHED
tcp 0 0 10.1.70.21.53196 10.1.70.142.8850 ESTABLISHED
tcp 0 0 10.1.70.21.50048 10.1.70.142.8850 ESTABLISHED
tcp 0 0 10.1.70.21.54080 10.1.70.142.8850 ESTABLISHED
tcp 0 0 10.1.70.21.62420 10.1.70.142.8850 ESTABLISHED
tcp 0 0 10.1.70.21.53533 10.1.70.142.8850 ESTABLISHED
tcp 0 0 10.1.70.21.62902 10.1.70.142.8850 ESTABLISHED

In the server,
tcp 0 0 *:8850 *:* LISTEN
tcp 209136 0 mppServer2:8850 10.1.70.21:54080 ESTABLISHED
tcp 208243 0 mppServer2:8850 10.1.70.21:53875 ESTABLISHED
tcp 232024 0 mppServer2:8850 10.1.70.21:53533 ESTABLISHED
tcp 235446 0 mppServer2:8850 10.1.70.21:53196 ESTABLISHED
tcp 235364 0 mppServer2:8850 10.1.70.21:62420 ESTABLISHED
tcp 233258 0 mppServer2:8850 10.1.70.21:50048 ESTABLISHED
tcp 235013 0 mppServer2:8850 10.1.70.21:51091 ESTABLISHED
tcp 0 0 mppServer2:8850 10.1.70.21:56228 ESTABLISHED
tcp 0 0 mppServer2:8850 10.1.70.21:59553 ESTABLISHED
tcp 235567 0 mppServer2:8850 10.1.70.21:62902 ESTABLISHED

In client, when I user lsof -i |grep 8850 I get nothing.
I can't find this process by ps -ef

Any suggestion will be appreciate.

[Robert-Jan Goossens]

Hi,

Have a look at this document from ITRC Tech Database.

http://www11.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c01003700-3

Title: Terminate a socket without terminating a process
Document ID: emr_na-c01003700-3

and this thread

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1246981

Regards,
Robert-Jan

Ps.to list open ports with lsof use;
# lsof -i tcp:8850

[likid0]

It's normal that if you kill -9 a proccess it can't close the connection.

also take a look at:

tcp_keepalive_interval:

Interval for sending keep-alive probes.

If any activity has occurred on the connection or if there is
any unacknowledged data when the time-out period expires, the
timer is simply restarted. If the remote system has crashed
and rebooted, it will presumably know nothing about this
connection, and it will issue an RST in response to the ACK.
Receipt of the RST will terminate the connection.

If the keepalive packet is not ACK'd by the remote TCP, the normal
retransmission time-out will eventually exceed threshold R2,
and the connection will be terminated.

With this keepalive behavior, a connection can time-out and
terminate without actually receiving an RST from the remote TCP.
[10000, 10*24*3600000] Default: 2 * 3600000 (2 hours)

[rick jones]

Are you certain about that kill -9 business? I don't recall seeing that behaviour before.

That lsof shows no process is concerning. I would have initially asked if perhaps there had been a fork() and so more than one process had a reference to the connection.

Once all processes are gone, the connection should no longer be in the ESTABLISHED state. It should, IIRC transition rather immediately to the FIN_WAIT_1 state or later, with a sequence that would look like:

ESTABLISHED
FIN_WAIT_1
FIN_WAIT_2
TIME_WAIT

So, assuming all I've postulated above is accurate, that you still have connections in ESTABLISHED and no processes with references to same, you have a situation that calls for excercising the support contract and ringing the Response Center.

The ndd tcp_discon stuff would just be kludging over a bug that needs to be reported through official channels and fixed.