Slow performance of tcp on OC-48 and understand netstat -s and

[MohitAnchlia]

We have 2 data centers which are connected using OC-48. 2 data centers are 300 miles apart and we have plenty of bandwidth available. Still we are seeing connection hangs and slow sends and received. I was looking at netstat -s output on receiver and this is what I see. There are 2 outputs below taken around 5-10 mts apart on the receiver. Could someone please help me understand the output in detail specially in "Tcp Ext" section. I see lots of DSACKS, delayed acks etc. but not sure how to tell what could be the problem. We are on Linux OEL 5 u 2
--

# date
Sun Mar 7 10:12:56 PST 2010
[root@qsc sock]#
[root@qsc sock]# netstat -s
Ip:
15511440 total packets received
423 with invalid addresses
0 forwarded
0 incoming packets discarded
15510946 incoming packets delivered
17203686 requests sent out
Icmp:
18 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
destination unreachable: 18
18 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 18
IcmpMsg:
InType3: 18
OutType3: 18
Tcp:
67659 active connections openings
88 passive connection openings
67577 failed connection attempts
7 connection resets received
66 connections established
15474225 segments received
17185084 segments send out
6062 segments retransmited
0 bad segments received.
67604 resets sent
Udp:
12033 packets received
18 packets to unknown port received.
0 packet receive errors
11932 packets sent
TcpExt:
44 TCP sockets finished time wait in fast timer
8 packets rejects in established connections because of timestamp
1869 delayed acks sent
1154 delayed acks further delayed because of locked socket
Quick ack mode was activated 313 times
154284 packets directly queued to recvmsg prequeue.
45603979 packets directly received from backlog
111214808 packets directly received from prequeue
1866874 packets header predicted
79007 packets header predicted and directly queued to user
7200996 acknowledgments not containing data received
3904657 predicted acknowledgments
1128 times recovered from packet loss due to SACK data
Detected reordering 1 times using FACK
Detected reordering 92 times using SACK
Detected reordering 124 times using time stamp
1128 congestion windows fully recovered
8022 congestion windows partially recovered using Hoe heuristic
0 TCP data loss events
967 fast retransmits
5095 forward retransmits
313 DSACKs sent for old packets
6062 DSACKs received
6 connections reset due to unexpected data
1 connections reset due to early user close
IpExt:
InMcastPkts: 766
OutMcastPkts: 652
InBcastPkts: 24069
OutBcastPkts: 1
[root@qsc sock]#
[root@qsc sock]#
[root@qsc sock]#
[root@qsc sock]#
[root@qsc sock]#
[root@qsc sock]#
[root@qsc sock]# date
Sun Mar 7 10:18:18 PST 2010
[root@qsc sock]# netstat -s
Ip:
16132584 total packets received
423 with invalid addresses
0 forwarded
0 incoming packets discarded
16132088 incoming packets delivered
17870510 requests sent out
Icmp:
18 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
destination unreachable: 18
18 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 18
IcmpMsg:
InType3: 18
OutType3: 18
Tcp:
67951 active connections openings
88 passive connection openings
67869 failed connection attempts
7 connection resets received
66 connections established
16095221 segments received
17851596 segments send out
6323 segments retransmited
0 bad segments received.
67896 resets sent
Udp:
12082 packets received
18 packets to unknown port received.
0 packet receive errors
11981 packets sent
TcpExt:
44 TCP sockets finished time wait in fast timer
14 packets rejects in established connections because of timestamp
1981 delayed acks sent
1208 delayed acks further delayed because of locked socket
Quick ack mode was activated 314 times
155732 packets directly queued to recvmsg prequeue.
45603981 packets directly received from backlog
111216294 packets directly received from prequeue
1935689 packets header predicted
79008 packets header predicted and directly queued to user
7476769 acknowledgments not containing data received
4084626 predicted acknowledgments
1179 times recovered from packet loss due to SACK data
Detected reordering 1 times using FACK
Detected reordering 92 times using SACK
Detected reordering 124 times using time stamp
1179 congestion windows fully recovered
8288 congestion windows partially recovered using Hoe heuristic
0 TCP data loss events
999 fast retransmits
5324 forward retransmits
314 DSACKs sent for old packets
6323 DSACKs received
6 connections reset due to unexpected data
1 connections reset due to early user close
IpExt:
InMcastPkts: 768
OutMcastPkts: 654
InBcastPkts: 24163
OutBcastPkts: 1