Remote Copy - Excessive TCP Retransmits

[L1nklight]

I have a pair of 7450s. One is located in Site A the other is located in Site B. Geographically these sites are about 8 hours apart by car. Between the two sites we have a 300 Mb/s connection that can burst higher. I believe it can burst to 1 Gbps. The exact number is based on our 95th percentile usage but we rarely exceed 100 Mb/s so the just assume we can burst to 1 Gbps when needed.

The 7450s are 4 node units and we are using the RCIP ports. All 4 nodes have 1 RCIP port configured at both sites.

I have 1 test volume for replication and so far it appears to replicate just fine however I constantly get "Excessive TCP retransmits at <value greater than 10%> on node <0-3>." I have tried a few things like limiting the connection speed down to 100 Mb/s but I haven't had much luck. Looking at the switch ports there doesn't appear to be any scaling CRC errors or Retransmits reported so I have to rule out the issue is between the 7450 and the switch. The link between the sites is a layer 2 link that plugs directly into the same switch and I am not seeing loss between the 2 switches (From Site A or Site B).

What could be causing the alerts? Has anyone seen similar behavior? My bandwidth usage according to my monitors appears fairly low during replication processes on my 7450.

[Dennis Handly]

>these sites are about 8 hours apart by car.

It might be helpful to actually state the distance.  :-)

[L1nklight]

471 Miles, 758 Kilometers.

Round trip ping average is clocking in at 11ms, no loss. Again the connection is a L2 connection that goes directly from site to site. I am told it's an aggregated multivendor MPLS cloud. Metrics have been gathering for longer than 18 months.

[L1nklight · Accepted solution]

Just to kind of bring this to a resolution... we host our 3PARs out of a CoLo provider. The provider links their two data centers together with a 10 gig line. The 10 gig line is linked to our systems VIA a simple layer 2 hand off. It looks, for all intents and purposes, to our equipment like a long cat 6 cable connecting our two remote facilities so there is literally nothing for us to configure or troubleshoot. As long as line protocol is up and we aren't getting any discards or hard errors, the connection appears to be good.

In order to bandwidth cap us, our provider puts a cisco policy map on our interconnect. This is something we were unable to see. A policy map can cap bandwidth but it does it in a really bad way for the 3PAR systems. The bandwidth gets capped by having a bit bucket and when that bit bucket for the policy map gets full, it starts discarding packets causing the source to miss acks and therefore it has to "retransmit."

Now, where it was difficult for us to troubleshoot was observing our own equipment. Our own equipment didn't show any signs of dropping packets or hard errors. The issue never appeared to be a problem with anything in our control. I wasn't really able to make any headway on the issue until I spoke with the engineering team at the hosted provider on how they are limiting bandwidth.