netstat -a hangs for a while on certain connections - how to

[Frank de Vries]

I have a situation where something is wrong with a connection , it shows up in netstat, but really we cannot even ping the client,
but is shows as ESTABLISHED.

Furthermore the netstat -a command itself
hangs for about a minute before it manages to
display this connection.

What I would like to know , how can I prevent
this and is there a command to remove such
connections.

when I do a netstat -a tcp 0 0 tcp 0 0 picnew.piclisrv oradb1.3708 ESTABLISHED
tcp 0 0 picnew.piclisrv oradb1.4361 ESTABLISHED
tcp 0 0 picnew.piclisrv bevwbxbxc02240.1049 ESTABLISHED
tcp 0 0 picnew.piclisrv picnew.59228 ESTABLISHED
tcp 0 0 picnew.piclisrv bevwbxbxc02261.1078 ESTABLISHED
[root@orasrv1:]/root<>>>

So this last one bevwbxbxc02261.1078 ESTABLISHED shoul not be there and netstat
hangs before it for a minute.

By the way I cannot see them with lsof -i tcp,
which seems to confirm the theory that they
became fake connections.

My first priority is to get hold of a way to
so that these fake connections do no longer show up in netstat and netstat can run
fast and smoothly.

Secondly I would like to know how to prevent this.

Regards,

[Mobeen_1]

Frank,
These processes you are referring to seem to me like defunct processes.

The only way you can get rid of them is by trying to use a "kill -9 pid" and if they don't vanish then reboot your server.

I would how ever urge you to have some patience and see how others in this forum respond before following anything.

We used to have this issue on AIX quite some time back and we were doing what i advised you

regards
Mobeen

[Frank de Vries]

How do I find out the pid from netstat -a ?
Is there a way to link up the process tables
with network socket connections on netstat ?

[Saravanan_15 · Accepted solution]

Dear Frank,
You can clear that particular connection by the following ways.First we have to convert that particular connection into hexa, for that take the netstat o/p using (to take the connection addr. in terms of ip address),

#netstat -na |grep -i established

Note down the Local & Foreign Address which you want to clear.

#LOCAL=
#FOREIGN=
#HEX=`echo ${LOCAL}.${FOREIGN} | awk -F "." '{printf "%02x%02x%02x%02x%04x%02x%02x%02x%02x%04x\n",$1,$2,$3,$4,$5,$6,$7,$8,$9,$10}'

Then, you can disconnect the connection using ndd utility by specifying this hexa no.

#ndd -set /dev/tcp tcp_discon_by_addr ${HEX}

Hope this will help you.

Regards,
Saravanan M

[Ermin Borovac]

Does netstat -an run any faster?

Can you please check how long it takes to do 'nslookup bevwbxbxc02261' and nslookup on the IP address it returns?

Does 'lsof -i tcp:1078' show this connection?

[Frank de Vries]

Thanks geeky stuf from Saravan,
I will certainly try that and hopefully
that will give me an instrument to
track and eliminate these bandit process
before the become a pain - Thanks S. :) :)

Concerning lsof see my comment in
opening message.

I seem to have find the following
relationship:
All the pc names that we
see with netstat -a cannot be resolved
with nslookup. Dns issue ?

I do not understand how that can be,
where is the name comming from
(some buffering ?)

It's weird.Hope one of you knows more.

[Frank de Vries]

S.

It helps to have more info and stuff,
but I am stil back to square one.

I have legitimate reasons to use
the ndd -set /dev/tcp tcp_discon_by_addr value

However I fail to see how to exactly
interprete the value part of the command.

I use
ndd -set /dev/tcp tcp_discon_by_addr 0ad6109905ef0ad609c3049d
and the response I get is:
operation failed, Invalid argument

The 2 tcp-endpoint that I need to terminate
are:
tcp 0 0 10.214.16.153.1529 10.214.9.195.1181 ESTABLISHED

Anyone see the idiosyncrosie in my
finishing of the ndd command above ?

[rick jones]

If netstat -a takes a long time for some connections, it suggests delays in resolving the names from the IP addresses of the connection (or service names from port numbers).

A connection can appear in netstat -a and not appear in lsof output. Lsof will only show those endpoints with a coresponding socket. However, a connection with no socket will only appear in netstat -a and perhaps ndd /dev/tcp tcp_status.

A connection in ESTABLISHED that does not appear in lsof output could possibly be a connection that has been ESTABLISHED, but for which the server application has not called accept(). If it persists for a "long time" it suggests that there is some issue with the server application.

The tcp_discon stuff should be considered a MASSIVE KLUDGE and SHOULD NOT be used except in EXTREME CASES.

I would suggest running the netstat -a command under tusc. Between that, and comparing the performance of netstat -an you will likely see resolution delays. Be sure to add the timestamps to the tusc command, and might as well make it verbose so you can see the bind/connect calls on the SOCK_DGRAM sockets involving port 53 for DNS lookups.

[Senthil Kumar .A_1]

Hi frank,

I agree with rick, ndd needs to be used cautiously..

But alas if u need to use it do as directed..

Proto Recv-Q Send-Q Local Address Foreign Address (state)

tcp 0 0 143.26.128.95.1019 143.26.128.97.2049 CLOSE_WAIT


Convert "Local IP", "Local Port", "Remote IP" and "Remote Port" to hexadecimal format. In this example it will be:

"Local IP" 143.26.128.95 0x8f1a805f
"Local Port" 1019 0x03fb
"Remote IP" 143.26.128.97 0x0f1a8061
"Remote Port" 2049 0x0801


Terminate the corresponding TCP connection using ndd command. The argument will be the 24 character strings that contains hexadecimal representation of "Local IP", "Local Port", "Remote IP" and "Remote Port".


ndd -set /dev/tcp tcp_discon_by_addr "0x8f1a805f03fb0f1a80610801"

[jerry1]

I know this is an old post but someone
may want to know why.

I had the same problem. Turns out that
one of the DNS servers was shutdown and
even though that server was listed as
a secondary in /etc/resolv.conf. It caused
the server to have slow performace.
I took the nameserver line out of resolve.conf
and everything started working fast again
on all my HP servers. This problem only
seemed to affect HP boxes. Not the Sun boxes.
On my hp-ux 10.20 boxes, dtgreet would
not startup with the downed dns server ip
entry in /etc/resolv.conf.
I do not know about 11.0 or 11.11 servers.
they do not have bitmap monitors.

Bad coding I say.