Tcp port can not be released

[aena]

On my rp4440 11i server, port 10000 is used by a process. After stoping the process, the port stays on use:
bcn19:root>netstat -a |grep 10000
tcp 0 0 cdmbd.10000 lebl2.1592 CLOSE_WAIT
tcp 0 0 *.10000 *.* LISTEN

When checking which process uses it, I get no answer:
bcn19:root>lsof -i :10000
bcn19:root>

I guess that the problem comes from the connection from lebl2 server blocking the port. This should not happen, if TCP pile and UX kernel worked correctly.

¿How could I release this port, besides from restarting the server?

[harry d brown jr · Accepted solution]

read: http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xe0d97680e012d71190050090279cd0f9,00.html

live free or die
harry d brown jr

[aena]

That one helped to close the remote connection, but I can not stop the LISTEN on the port. Ehe remote server opens a new connection after killing the existing one.

[jpcast_real]

It sounds me strange that the lsof does not say anything . Sometimes the lsof does not present the port number and present the name of the service which is using this port.

netd 1175 root 28u inet 0x6318d840 0t0 TCP *:recserv (LISTEN)
inetd 1175 root 29u inet 0x6318d9c0 0t0 TCP *:swat (LISTEN)
inetd 1175 root 31u inet 0x6318dcc0 0t0 TCP *:omni (LISTEN)
inetd 1175 root 32u inet 0x6318de40 0t0 TCP *:hacl-probe (LISTEN)
inetd 1175 root 34u inet 0x6319f280 0t0 TCP *:hacl-cfg (LISTEN)
sendmail 1450 root 4u inet 0x5fecd1c0 0t0 TCP *:smtp (LISTEN)


Can you make an lsof |grep TCP ?????

[Todd Whitcher]

There must still be a Process using that port.

Use lsof to review the port and see what PIDs are active on it.

You can get it here:

http://ftp.cerias.purdue.edu/pub/tools/unix/sysutils/lsof/binaries/hpux/

Here is an example

lsof -i tcp:23
losf -i udp:2049

[harry d brown jr]

Todd, he has lsof.

Is port 10000 define in /etc/services ???

grep 10000 /etc/services.
*********************************
HOLD the PHONE (Presses, whatever)!!!

port 10000 is :

console 10000/tcp # MC/System Environment console multiplexor


YOU should not be using port 10000 for an application.

live free or die
harry d brown jr

[Todd Whitcher]

Oops

[aena]

lsof |grep TCP shows a lot of processes, but the process on port 10000 does not appear:
bcn19:root>lsof |grep TCP
inetd 824 root 5u inet 0x509c2340 0t0 TCP *:ftp (LISTEN)
inetd 824 root 6u inet 0x509c24c0 0t0 TCP *:telnet (LIST
EN)
inetd 824 root 8u inet 0x509c27c0 0t0 TCP *:login (LISTE
N)
inetd 824 root 9u inet 0x509c2940 0t0 TCP *:shell (LISTE
N)
.
.
.

We are not using Multi Computer (MC) software. There should be no conflict between applications.

Any further ideas

[Todd Whitcher]

Try lsof w/ the -p option to use port #'s instead of names on the server that shows that tcp port.



ex.
root@gator/HAdumps/TOOLS>./lsof_4.63 -i -P |more

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 1020 root 5u inet 0x2500540 0t0 TCP *:22 (LISTEN)
inetd 1185 root 4u inet 0x278be80 0t0 UDP *:49165 (Idle)
inetd 1185 root 5u inet 0x278b180 0t0 TCP *:21 (LISTEN)
inetd 1185 root 10u inet 0x2500440 0t0 UDP *:518 (Idle)


You can also use ndd to look for your connections.

This is an ugly output, it doesnt show up good in the ITRC forums. Just make your display WIDE so you see all the columns. The second to last are localport / foreignport. You want to look for your port in the lport column, its in hex.

ndd -get /dev/tcp tcp_status |more

TCP dst snxt suna swnd cwnd rnxt rack rwnd rto mss [lport,fport] state
03030380 :: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 03000 00536 [9,0] TCP_LISTEN
03030080 :: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 03000 00536 [7,0] TCP_LISTEN
03030680 :: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 03000 00536 [13,0] TCP_LISTEN
03030980 000.000.000.000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 03000 00536 [220,0] TCP_LISTEN
03030c80 000.000.000.000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 03000 00536 [21f,0] TCP_LISTEN
030383c0 000.000.000.000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 03000 00536 [1e87,0] TCP_LISTEN
030380c0 000.000.000.000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 03000 00536 [17e0,0] TCP_LISTEN

You have to convert the port numbers from hex to decimal. Look in the lport/fport field.

So for 10000 look for lport 2710

[Todd Whitcher]

Let me break down the first column for you since it doesnt show up so well

Your concerned w/ the [9,0] TCP_LISTEN
fields at the end.

So if I take this string:

TCP dst snxt suna swnd cwnd rnxt rack rwnd rto mss [lport,fport] state
03030380 :: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 03000 00536 [9,0] TCP_LISTEN

My local port is 9 and foreign port is 0
and its in a Listen State:

So if you find your port you can disconnect it w/ ndd also.


This is an only writable option of ndd.
It allows to flush any connection regardless of it's state.
This was formerly used to get rid of FIN_WAIT2 state connections. (now tcp_fin_wait_2_timeout is available)
Also connections in the state CLOSE_WAIT could be removed.

To give an example:


ndd -get /dev/tcp tcp_status | grep -e state -e TCP_FIN_WAIT_2
TCP dest snxt suna swnd cwnd rnxt rack rwnd rto mss [lport,fport] state
00000001015d1468 015.043.232.202 361e667e 361e667e 00008000 000005b4 06dda013 06dda013 00008000 24000 01460 [ca16,9e9] TCP_FIN_WAIT_2*
0000000102b6a468 015.140.008.076 4853f49a 4853f49a 00000000 00006000 4852f36d 4852f36d 00008000 04063 04096 [2ab,c4f6] TCP_FIN_WAIT_2*

To flush one of those connections you only need to take the first column of the tcp_status line and use it for the tcp_discon option.

ndd -set /dev/tcp tcp_discon 0x00000001015d1468

[harry d brown jr]

There's a conflict if port 10000 is not commented out in /etc/services. If you don't comment it out, inetd will listen on that port.

live free or die
harry d brown jr

[aena]

bcn19:root>ndd -set /dev/tcp tcp_discon "0xc31028e8"
bcn19:root>ndd -get /dev/tcp tcp_status|grep 2710
00000000c31028e8 000.000.000.000 13d8cb64 13d8cb63 00000
000 00000000 00000000 00000000 00000000 03000 00536 [2710,0] TCP_LISTEN
bcn19:root>

It stays up

[Todd Whitcher]

There is some application binding to that port even when you disconnect it. Typically lsof catches it. I've seen older versions of lsof miss these types of things.

Try going out to Purdue University and getting the latest one. I posted the link earlier.

Todd

[rick jones]

Simply being in /etc/services is meaningless - inetd will only listen on things configured into the inetd.conf file.

If inetd were indeed listening on port 10000 then lsof should show it.

If this process forks during its lifetime, I'll bet that one or more of the processes remains, perhaps in a zombie state, and didn't close some FD's after the fork call. 99 times out of 10 CLOSE_WAIT means that either apps have not properly responded to a remote "no more data" indication (read return of zero on the socket) or have other references (eg processes) to the socket lying about.

Any use of ndd to terminate endpoints should be considered a kludge at best. In desparate situations, kludges are necessary, otherwise find root cause and fix that instead.

[aena]

Eventually found the root cause. A process that uses TIBCO does not close correctly the transport and rvd daemon keeps thinking that the port is used. Restarting rvd daemon solves it for the moment, but changes will have to be made to the code.

Anyway, I guess lsof should have showed rvd as the owner of the socket.

Thanks for you help