connect requests dropped due to full queue

[George_231]

A third party application (BMC Patrol) running on one of my server has problem with the agent freezing and not responding to requests.
When I run “netstat –p tcp” command I get the bunch of output. I am copying the last few lines here:
4680 connection requests
16044 connection accepts
20724 connections established (including accepts)
23711 connections closed (including 3295 drops)
1413 embryonic connections dropped
3653768 segments updated rtt (of 3653768 attempts)
3101 retransmit timeouts
473 connections dropped by rexmit timeout
205 persist timeouts
9 keepalive timeouts
4 keepalive probes sent
1 connection dropped by keepalive
55562 connect requests dropped due to full queue
1398 connect requests dropped due to no listener

As suggested on this forum, I increases the value of two tunables. I'll paste them below:

TRANSPORT_NAME[0]=tcp
NDD_NAME[0]=tcp_conn_request_max
NDD_VALUE[0]=8192
#
# Changed the default value of tcp_syn_rcvd_max to 2000 (was 500)
TRANSPORT_NAME[1]=tcp
NDD_NAME[1]=tcp_syn_rcvd_max
NDD_VALUE[1]=2000

I still have:
55562 connect requests dropped due to full queue
and in addition:
1398 connect requests dropped due to no listener

What are you thoughts and suggestions?
Thanks in advance.

[D Block 2]

netstat -s
shows some stats. run again in 5 minutes to perform a delta check or comparision.

are you patched up-to-date ?

get on your local connected switch (cisco switch) and have networking verify how many packets are dropped ? determine the percentage of dropped packets in the switch.

are you using a Gige network card on the server? run lanadmin and look at the stats on the network card. Verify your settings: speed negotiation, etc. If you have a Gige card, the Flow-Control stats is reporting that the switch is backing up or dropping packets. might be something to look at in your investigation.

check the system log for possible errors.

good luck,
T

[Steven E. Protter]

Shalom,

On the face of the message, it would appear the card is getting more traffic than it can handle.

I'd expect to see something elsewhere though, in dmesg or /var/adm/syslog/syslog.log

Perhaps fire up cstm,mstm or xstm and run some hardware diagnostics.

If the switch is set to the same speed as the NIC card, your network team might find some problems too.

SEP

[whiteknight]

George,

I'm more concerned with your current Arpa Patch and Stream Patches, because it fixed a lot of TCP issues.

PHNE_35351 11.11 cumulative ARPA Transport patch
PHNE_34131 11.11 Cumulative STREAMS Patch
or
PHNE_35766 11.23 cumulative ARPA Transport patch
PHNE_34788 11.23 Cumulative STREAMS Patch

Please get the patches updated.

WK
please assign points

[rick jones]

The so called "listen queue" - what is overflowing when you se connections dropped due to full queue will be the _minimum_ of tcp_conn_request_max and what the application passes-in to the listen() socket call. So, once you have tcp_conn_request_max above what the application is setting, any further increase in tcp_conn_request_max will have no effect.

The thing you really need to do is figure-out why the agent is freezing and make that go away. Tweaking listen queues is only treating a symptom.

Now, wrt your specific netstat output. The numbers are the same after your tuning because the statistics do not reset except at boot. So, there were no additional drops after the tuning, or you happened to very coincidentally get the same stats after a boot, or happened to cut-and-paste the wrong set of stats.

If you want stats over an interval, consider saving two snapshots of netstat stats to files and run them through beforeafter:

netstat -s -p tcp > before
sleep 60
netstat -s -p tcp > after
beforeafter before after > delta
more delta

where you can get beforeafter from:

ftp://ftp.cup.hp.com/dist/networking/tools/

[George_231]

I did reboot the server twice for applying different set of patches the latter being the STREAMS patch recommended in this thread.
In short:
1. I have applied the recommended patches
2. Increase the values of tcp_conn_request_max to 8192
3. Increased the value of tcp_syn_rcvd_max to 2000
I still have the "connect requests dropped due to full queue" with rapidly growing count.

I found out from netstat -an that I have packet loss of 10/14/18% on a certain network monitored by Patrol. I'll involve the network staff to troubleshoot.

[rick jones]

Have you actually tweaked any of the _application_ settings? Again, just setting tcp_conn_request_max alone is not enough if the application is calling listen() with a smaller value for the backlog parameter.

[Denver Osborn]

If your clients disconnect due to timeouts caused by a full listen queue, tuning the tcp_conn_request_max higher won't do anything for the app. When your app starts it's more than likely going to set backlog to some other defined value. Check your apps config files or vendor's support site for anything re: listen backlog or listenq

-denver