We had an issue yesterday where Apache reached MaxClient limit on all our web servers. Though we couldn't determine the coz at that time, in theory we think it happened because of a ACL issue between our web server and app server.
Because web server(through mod_jk connector) wasn't able to talk to app server and the syn packed got dropped on the firewall, the sessions kept increasing and ultimately maxed out.
I want to know if we have a Linux parameter which can be tuned for the time between a syn request and establish. Can we lower that time between those two state?
A TCP connection has no timed delays in the transition from SYN_SENT to ESTABLISHED. This transition can happen as soon as the listening endpoint sends the SYN+ACK packet.
Without the SYN+ACK packet, the transition cannot happen at all: this is because the SYN and SYN+ACK packets are used to determine the initial TCP Sequence Numbers, which must be known by both endpoints or the connection won't work. You cannot establish a connection with something that does not answer at all: no amount of tuning and tweaking can make this happen.
If your mod_jk cannot talk to the application server in a reasonable time, the only possibility is to time out and let Apache generate an error response to the client, ending the client-side request and freeing the Apache client slot.
This can be set in mod_jk workers.properties file, using the worker..socket_timeout and worker..socket_connect_timeout settings.
By default both settings are zero, which means no timeout at all: mod_jk will wait forever for the application server's answer. Of course this causes the client-side sessions to pile up, until the MaxClients limit is reached.
If either of these timeouts are reached, mod_jk will try to re-establish the connection and try again one more time (the number of retries is adjustable too). If the retry attempt fails too, mod_jk will return an error to Apache. At that point, Apache will generate an error page, terminating the client's HTTP request and thus freeing up Apache's client slot.
