Connections dropped due to a full queue

[BFA6]

Hi All,

I have an L1000 running HPUX 11.0. When running netstat -p tcp, I have 922 connect requests dropped due to full queue. 153 of these have been over the weekend.
The tcp_conn_request_max parameter is currently set to 4096.

Suggestions welcome on how to fix this.

Regards,

Hilary

[Steven E. Protter]

This thread seems relavent Hilary:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=21889

I would also add that you may have limits set too low in the kernel. I'll admit I'm not sure which ones.

SEP

[Ron Kinner]

I wonder if adding more memory would help? Your input queue is filling up faster than the process handling the queue can handle. Making the queue longer which is what setting tcp_conn_request_max does is just treating the symptom and not the problem. It's actually self defeating because the clients get impatient and start trying to start new sessions so you get more entries in the queue.

You need to speed up the process in order to handle the load so more memory or on some boxes more processors or perhaps the program needs to be cleaned up or perhaps you need to add a second box to help with the load.

This is assuming that these are all legitimate customers and not some sort of denial of services attack. Should you be getting heavy traffic on the weekends?

Ron

[Con O'Kelly]

Hi Hilary

To me the value of 4096 already seems pretty high for tcp_conn_request_max. Obviously you could increase this value further using ndd but as Ron said this doesn't treat the sympton.

I've attached a Network doco written by Rick Jones(HP) that mentions this issue. As you'll see this issue can be due to the application itself rather than the tcp_conn_request_max being too small. I'd try to identify any applications that might be causing this issue.

Cheers
Con

[rick jones]

Keep in mind that there are _two_ things that affect the length of a given listen queue. The first is tcp_conn_request_max. The second is the value the application passes in the listen() call.

The length will be the _minimum_ of the two.

So, if you have tcp_conn_request_max at 4096, and still have connection drops due to full queue, my first thought would be that one or more applications running on that system has not set a very large listen() backlog.

The other possibility is that the tcp_conn_request_max setting was made _after_ the application(s) in question were started - changes to tcp_conn_request_max will not be "retroactive" to existing listen endpoints.

Finally, it could just be the case that your system/application was hit with a connection request load that it just couldn't handle.

153 connection requests dropped out of how many total connections established over the weekend?

[Con O'Kelly]

Just thought I'd attach Rick's other doco on ndd tunables.
Its an excellent resource for understanding ndd tunables.

ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_ndd.txt

Cheers
Con

[James A. Donovan]

If these are ssh connection attempts that are being dropped, you may want to look at increasing the value of MaxStartups in you sshd_config file.

MaxStartups
Specifies the maximum number of concurrent unauthenticated
connections to the sshd daemon. Additional connections will be
dropped until authentication succeeds or the LoginGraceTime
expires for a connection. The default is 10.

Alternatively, random early drop can be enabled by specifying the
three colon separated values ``start:rate:full'' (e.g.,
"10:30:60"). sshd will refuse connection attempts with a probability of ``rate/100'' (30%) if there are currently
``start'' (10) unauthenticated connections. The probability
increases linearly and all connection attempts are refused if the
number of unauthenticated connections reaches ``full'' (60).

[James A. Donovan]

Just took a look at the openssh code (version 3.7.1p2/sshd.c), the listen() call sets backlog to 5. So if these are ssh connections, that may be the source of your trouble...

[BFA6]

Hi Jim,

We don't use ssh.
This problem has just started to occur over the last month or so. As far as I'm aware nothing has changed in the application, there is not a vast increase in the amount of users. This started to happen after a virus attack which was flooding the network, but this may be a red herring.

This box dropped over 100 connections last night due to a full queue.

Would you recommend that I drop the tcp_conn_request_max parameter back down, as I believe the default is only 20 on an 11.00 system?

Regards,

Hilary

P.S. I will assign points later, but have a disk issue now on another server (it never rains but pours!)

[rick jones]

The only thing dropping tcp_conn_request_max might do for dropped connections would be to increase their rate :)

to know how big the problem is one has to compare the rate of dropped connectoins to the total connections established over the same interval.

[Ron Kinner]

Get a copy of tcpdump and let it look at the traffic. You may find that one of your clients is indeed infected and is attacking your server.

Ron