Weird sockets

[Rob Saunders]

My client sockets are ending up in a SYN_SENT state when the server process is restarted.

Both client and server processes are local to the same machine (a 6CPU N-class 4GB RAM).

I have 120 clients attempting to connect to the server after a server restart and they all result in the sockets being left in this state.

maxfiles is currently set to 200 (but I can't recreate the problem locally on a similar system where maxfiles is much lower). I don't believe that this is a problem with the number of sockets...

Any ideas anyone?

TIA

Rob

[Jeff Schussele]

Hi Rob,

One thing to always remember about UNIX is that it's a file-based OS. *Everything* is a file, and that includes devices and sockets as well as *real* files.
maxfiles @ 200 is low if you have an app that spawns/accepts a lot of connections. So I'd bump mafiles as well as maxfiles_lim to a value at least 50% higher than the anticipated high water connection mark for that app.

Another thing to investigate is dropped connections (network) and/or crappy client TCP stacks (read M$). If clients have their/hose their connection, you can be left in such a state as you're seeing & then the socket's not closed & the *file* remains open.

My $0.02,
Jeff

[U.SivaKumar_2]

Hi,

maxfiles , nfiles will only come in to picture only in UNIX DOMAIN SOCKETS not in tcp communication.

Since you have lot of SYN_SENT , indicating that you have not received SYN_ACK back from the server.

That may be because of a busy server or may be because of connection drops due to tcp backlog queue overflow.

#netstat -s | grep queue

If the output is

0 connect requests dropped due to full queue

Then your backlog queue size is ok. But if you have connection requests drops. Do this

#/usr/sbin/ndd -set tcp_syn_rcvd_max 2000

Make entries in /etc/rc.config.d/nddconf file

TRANSPORT_NAME[]=tcp
NDD_NAME[]=tcp_syn_rcvd_max
NDD_VALUE[]=2000

ndd -set /dev/tcp tcp_connec_request_max 2048

Make entries in /etc/rc.config.d/nddconf file

TRANSPORT_NAME[]=tcp
NDD_NAME[]=tcp_connec_request_max
NDD_VALUE[]=2048

Restart the server once to clear current SYN_SENT
regards,
U.SivaKumar

[U.SivaKumar_2]

Hi,

maxfiles , nfiles will only come in to picture only in UNIX DOMAIN SOCKETS not in tcp communication.

Since you have lot of SYN_SENT , indicating that you have not received SYN_ACK back from the server.

That may be because of a busy server or may be because of connection drops due to tcp backlog queue overflow.

#netstat -s | grep queue

If the output is

0 connect requests dropped due to full queue

Then your backlog queue size is ok. But if you have connection requests drops. Do this

#/usr/sbin/ndd -set tcp_syn_rcvd_max 2000

Make entries in /etc/rc.config.d/nddconf file

TRANSPORT_NAME[]=tcp
NDD_NAME[]=tcp_syn_rcvd_max
NDD_VALUE[]=2000

ndd -set /dev/tcp tcp_connec_request_max 2048

Make entries in /etc/rc.config.d/nddconf file

TRANSPORT_NAME[]=tcp
NDD_NAME[]=tcp_connec_request_max
NDD_VALUE[]=2048

Restart the server once to clear current SYN_SENT

regards,
U.SivaKumar

[rick jones]

A couple of corrections.

*) maxfiles _can_ come into play with TCP sockets - however, if there are sockets associated with TCP endpoints in the SYN_SENT state then is is rather unlikely that a process (maxfiles, maxfiles_lim) or the system (nfile) has run-out of file descriptors.

*) The ndd tunable is tcp_conn_request_max

*) SYN_ACK makes it looks like a TCP state :) It is more commonly (iirc) or perhaps more accurately presented as SYN|ACK since it is a short-hand for a TCP segment that has the SYN bit set and the ACK bit set, and having both those set in C-speak would be SYN|ACK.

*) Given that the default for tcp_syn_rcvd_max is 500, it is unlikely that is being overflowed. The default for tcp_conn_request_max is merely 20, making it a more likely candidate. It is also necessary for the server application to make a listen() call with a suitably sized backlog parm as the actual queue length will be the minimum of tcp_conn_request_max and the parameter to listen().


*) Connections in SYN_SENT should remain there no longer than tcp_ip_abort_cinterval milliseconds (ndd) (yep, cinterval for "connection" to distinguish it from tcp_ip_abort_interval, which is for established connections)

The suggestion to check the netstat statistics is a good one - the queue that would be full would be on the server. On the client(s) the stat of interest would be the tcp retransmission timeouts. It is entirely possible that when the server is restarted, it cannot initially deal with the flood of all the clients trying to re-establish their connections at the same time

ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_netstat.txt

[Rob Saunders]

Thanks for the help so far - it's really appreciated. Okay, I've checked out the tunables and they look okay when compared against similar systems that don't exhibit this problem. The netstat -s | grep queue statistic does show that connection requests are being dropped though. So, busy local tcp? Is there a way in which I can determine what application (if any) is responsible for the bulk of the traffic? nettl?

Both client and server are co-hosted, but obviously tcp connects can come from internal and external sources. I may be looking for the proverbial needle in the haystack here, but I'm trying to understand why this system demonstrates the problem, yet this is the only system that does it in a large number of similar systems.

Can anyone help here?

Thanks again

Rob

[Ron Kinner]

netstat -a |grep tcp

will show all of your tcp connections. Should be obvious who is doing the talking and what protocol they are using. -an will run faster but then it shows only the IP addresses and port numbers.

Ron

[rick jones · Accepted solution]

Connection drops due to full queue mean a busy application. TCP is just fine and dandy :) (ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_netstat.txt)

That other systems with the same ndd settings do not exhibit the problem does not exonerate the ndd settings :) It only means that those systems are either faster or more lightly loaded. I'd still try upping tcp_conn_request_max to something like 1024.

As for figuring-out which app - you could use nettl or ou could use tcpdump. If you download and use tcpdump, you would want to use a filter expression that only showed TCP segments that had the SYN bit set in the header. You would then look at the port number(s) to see which apps are getting the connection requests. Those that are not keeping-up with the rate of connection requests will have SYN's retransmited - you can tell that a SYN segment is a retrans by the local/remote IP, local/remote port, and sequence numbers being the same.

If you do not want to wade through the tcpdump trace by hand, you might try running it through tcptrace and then xplot. Then you will get nice little graphs showing "R"s in red for retransmitted segments - althoughyou will likely get separate xplot files for each "connection" (four-tuple of IP and port numbers).