SYN_RCVD messages !!

[Ivan Azuara]

Hi,

I'm lossing the connection with mi Application Server (Bea Weblogic) throught 9001 port. When i check the netstat status, i receive the next information:

tcp 0 1 130.10.16.4.9001 130.20.180.44.4783 SYN_RCVD
tcp 0 1 130.10.16.4.9001 130.20.180.44.4785 SYN_RCVD
tcp 0 1 130.10.16.4.9001 130.20.218.40.4874 SYN_RCVD
tcp 0 1 130.10.16.4.9001 130.20.180.44.4780 SYN_RCVD
tcp 0 1 130.10.16.4.9001 130.20.180.44.4819 SYN_RCVD
tcp 0 1 130.10.16.4.9001 130.20.218.40.4876 SYN_RCVD
tcp 0 1 130.10.16.4.9001 130.20.180.44.4777 SYN_RCVD
tcp 0 1 130.10.16.4.9001 130.20.180.44.4798 SYN_RCVD
tcp 0 1 130.10.16.4.9001 130.20.180.44.4781 SYN_RCVD

I was searching in the web an i found the next information about the SYN_RCVD message:

A half open TCP connection is stored in the TCP Half Open Queue in the state "SYN_RCVD". It should take less than one second for a TCP connection to transition from the "SYN_RCVD" state to the "ESTABLISHED" state. The only time you will ever see a connection in the "SYN_RCVD" state is during a SYN flood.

Some body has an idea ?

Thank's in advance !!

[Maxim Yakimenko]

Hi
130.20.180.44 and 130.20.218.40 - a these addrs are known to you? Are they are able to be flooders?
Possibly your server cannot process incoming connections, i.e. it can not call accept() system call on them so they hang in incoming connections queue for long time

[Ivan Azuara]

Yes, these ip address are from known clients, but how can i verify if these clients have the flood property ?.

Thank's in advance!

[Matti_Kurkela]

This *may* be a flood, or it may also be an error in the application: if the application makes the Weblogic Application Server lock up or spend a long time in garbage collection, this is what happens.

Another explanation might be that the system's performance is too low for the current load.

Has your application ever been performance tested? This is usually done by using another program on one or more other servers to send requests to your application at a pre-selected speed, or even as fast as possible. The amount of completed requests is then monitored. The testing program needs to be configured to create exactly the same kind of queries that your application users normally do.

Another thing to do is profiling: running an application with profiling enabled slows it down (sometimes a lot!), but it produces information that can be used to find out the parts of the code that the application is spending most of its execution time. This helps a lot in analyzing and fixing performance problems.

With Java applications, you can also take a "thread dump" at any time. You can send a "kill -QUIT" to the Java Virtual Machine: it won't stop the JVM, but it causes the JVM to print out the status of all the threads the JVM is running at the moment, including the name of the method each of the threads is running in.

For example, Bea Weblogic has a concept of "execute threads". At least the earlier versions had a limited number of them: if all of them are tied up in e.g. database queries, the Weblogic server cannot accept any new connections until at least one execute thread becomes free. The result might look just like your case.

In our internal development projects we have generally noticed this: if an application has not undergone performance testing during development, it may very well contain design mistakes and/or inefficient programming techniques that limit its performance. It may be very difficult to correct those mistakes afterwards without scrapping at least part of the design an re-implementing.

[Ivan Ferreira]

Your clients may be trying and trying the conection, but the connection is not ESTABLISHED, this could be a routing problem or a firewall filtering the return packets.

[rick jones]

SYN_RCVD means we have received a SYNchronized from the remote, and we've sent a SYN|ACK to the remote and are awaiting an ACKnowledgement of our SYN from the remote.

The remote's ACK of our SYN _should_ be automagic in the remote's transport, but _could_ I suppose be delayed waiting to give the remote client application a chance to send a request to be piggybacked with the ACK.

Presumably, we will keep retransmitting our SYN until the remote ACKs or we hit the tcp_ip_abort_cinterval setting. You may see an increase in TCP retransmission timeouts recorded in netstat -p tcp output without corresponding increases in data segments retransmitted. Of course, if there are general packet loss problems between the server and the clients, you will see data segments being retransmitted too.

Some "general goodness" things to check:

*) That pings have no losses between the client(s) and the server

*) Check the lanadmin statistics on the server and see if it is in half-duplex and recording _late_ collisions. That _could_ be an indication of a duplex mismatch - search the archives talking about autoneg for more on that one.