Routing Issue/Speed - Requires different path to and from

[Scott.D]

I am having a really weird issue. It started a few weeks ago after some power outages. My Fiber backbone provider swapped out some Cosco equipment and ever since I am having routing issues. It may not be related but since they changed something and I didn't when the problem started I think it is related to their equipment.

Basically - I have eight sites connected by a Layer 2 fiber network. If you run a throughput test across their fiber it is perfect in every case.

I use an HP 7102 router at 7 sites and a HP 3500 Layer 3 Switch at the central site. All sites going over the fiber are on the same VLAN and same Subnet. I am using RIP on the WAN and each building can talk directly to each building.

I have two sites now that can't talk to my central site at full speed - seems like it is getting throttled back to 10-15 Mbps instead of 50Mbps (my bandwidth allocated by provider over the Fiber).

An example -

If Site C in my Central Site
Site A > C - bandwidth 10-15 Mbps
Site A > B > C - bandwidth 50 Mbps

Now if I set my routing as follows -
Site A > C (Directly from one site to the other)
Site C > B > A (Set the return trip via a different site)
I can achieve full speed.

It is the same for both sites experiencing problem.

I have also tried remove the 7102 Routers and use a VLAN from a HP 4108 and HP 4208 switch to the Procurve 3500 at the Central Site. Same speed issue. If however I place just a dumb switch between their equipment at site C and my Procurve 3500 full speed.

It looks like it is some some of MAC address issue but I have been unable to find anything.

Hope this makes any sense and someone has a suggestion.

Thanks,

Scott

[Olaf Borowski]

Scott,

The 7000 series routes doesn't support any kind of fiber connection so there must be something inbetween. Is your SP providing a copper ethernet drop? I assume that is the case. Have you checked the duplex setting on the interface? Maybe duplex mismatch? If the SP can't tell you what the setting should be (migh be auto), try fixing the duplex at speed on the router side to see if this makes a difference. Also, check the statistics on the particular interface and let us know if there is anything suspicous.

[Scott.D]

Olaf -

Thanks for the reply. Was not clear on the SP side. Their connections are fiber and using Layer 2 switching between buildings. They hand off from the Cisco switch to my 7102 Routers via a 100Mbps Full Duplex port. They require them to be set on my side as well. They do not want Auto.

The strange thing is with both sides set to 100 Full they say the Cisco see it as 100 Half. On Auto on both they seem fine.

Even with that - 5 of the sites are fine, the other 2 read the same but have the speed issue and only if the path to and from are the same.

They have tested from their 100Mbs Full handoff to the other side handoff and the speeds are perfect. I have also attached some laptops and run test laptop to laptop and receive a full 100Mbps between sites. It is only when I am on the other side of the HP equipment the problem starts.

They have no packet loss on their side, nor does my HP. In one case while they were testing Auto vs 100 Full Dplx, they left one of their ports on Auto when mine was on 100 Full Dplx - in that case I had CRC errors everywhere but that was fairly obvious what was happening.

In fact when I start a file transfer they never see their equipment receiving more that the 10-15 Mbps being requested. If I route Site A > C then back C > B > A works perfectly. Full speed.

Stumped - as is my service provider.

Thanks,

Scott

[Dmitry G. Spitsyn]

Hi, Scott !

Check Flow control settings at both ends of links. If not Auto the settings must be consistent.

Good luck,
Dmitry

[Olaf Borowski]

David,

If the Cisco sees it at half duplex, that's a problem. You said when you put a "dumb" switch inbetween, you get the full speed. I suppose, the dumb switch only does 100/half. That would point to a duplex mismatch. I would force both sides to half duplex and rerun the test to determine if that is the problem. What kind of Cisco switch is the 7102 connecting to?

[Scott.D]

Thanks again for the responses all.

I don't see any Flow Control settings on the HP 7102 routers. The other end is the Procurve 3500 switch using

GBIC - no flow control. Also - not problem on 5 of the other sites coming in that port.

The SP has a Cisco Catalyst 3750 metro at each site. Their handoff to my equipment is via a 100Mbps ehternet

port. Their backbone is fiber and is connected to their equipment via a Fiber GBIC. Also - their handoff to my

central 3500 Switch is via Fiber GBIC into my Procurve. That port is set to 300Mbps.

The SP has changed their ethernet handoff to Auto Negotiate and I likeweise switched the 7102 Router to Auto and

they immediately synched up as 100Mbps Full Duplex as they should. Same problem exist though.

My routing is like this. Class C addresses are real but they are no longer Public address. They are all natted

and waiting to be transitioned to private addresses.

Basically â

If I route 207.63.146.0 (Site A) directly to 207.63.144.0 (Site C), speed is slow. If I route from 207.63.146.0

to 207.63.146.0, but create a static route from 207.63.144.0 to 207.63.146.0 via 192.168.101.3, I get full speed.

If I reverse the whole scenario and go from 207.63.144.0 to 207.63.146.0 directly it is slow, if I add a return

route from 207.63.146.0 to 207.63.144.0 via 192.168.101.3, it is slow again.

I can go direct from 207.63.146.0 to any other subnet directly at full speed.

Site A - Subnet 207.63.146.0/192.168.101.7

Gateway of last resort is 192.168.101.1 to network 0.0.0.0

R 0.0.0.0/0 [120/1] via 192.168.101.1, eth 0/2.1
R 192.168.9.0/24 [120/1] via 192.168.101.1, eth 0/2.1
R 192.168.100.0/24 [120/1] via 192.168.101.1, eth 0/2.1
C 192.168.101.0/24 is directly connected, eth 0/2.1
R 207.63.144.0/24 [120/1] via 192.168.101.1, eth 0/2.1
R 207.63.145.0/24 [120/1] via 192.168.101.2, eth 0/2.1
C 207.63.146.0/24 is directly connected, eth 0/1
R 207.63.147.0/24 [120/1] via 192.168.101.8, eth 0/2.1
R 207.63.148.0/24 [120/1] via 192.168.101.3, eth 0/2.1
R 209.7.74.0/24 [120/1] via 192.168.101.5, eth 0/2.1
R 209.7.75.0/24 [120/1] via 192.168.101.6, eth 0/2.1
R 209.174.35.0/24 [120/1] via 192.168.101.4, eth 0/2.1

Site B - Subnet 207.63.148.0/192.168.101.3

Gateway of last resort is 192.168.101.1 to network 0.0.0.0

R 0.0.0.0/0 [120/1] via 192.168.101.1, eth 0/2.1
R 192.168.9.0/24 [120/1] via 192.168.101.1, eth 0/2.1
R 192.168.100.0/24 [120/1] via 192.168.101.1, eth 0/2.1
C 192.168.101.0/24 is directly connected, eth 0/2.1
R 207.63.144.0/24 [120/1] via 192.168.101.1, eth 0/2.1
R 207.63.145.0/24 [120/1] via 192.168.101.2, eth 0/2.1
R 207.63.146.0/24 [120/1] via 192.168.101.7, eth 0/2.1
R 207.63.147.0/24 [120/1] via 192.168.101.8, eth 0/2.1
C 207.63.148.0/24 is directly connected, eth 0/1
R 209.7.74.0/24 [120/1] via 192.168.101.5, eth 0/2.1
R 209.7.75.0/24 [120/1] via 192.168.101.6, eth 0/2.1
R 209.174.35.0/24 [120/1] via 192.168.101.4, eth 0/2.1

Site C - Subnet 207.63.144.0/192.168.101.1 (Central Site to all buildings)

Destination Gateway VLAN Type Sub-Type Metric Dist.
------------------ --------------- ---- --------- ---------- ---------- -----
0.0.0.0/0 207.63.144.32 1 static 1 1
127.0.0.0/8 reject static 0 0
127.0.0.1/32 lo0 connected 1 0
192.168.9.0/24 207.63.144.2 1 static 1 1
192.168.100.0/24 DEFAULT_VLAN 1 connected 1 0
192.168.101.0/24 FIBER-WAN 10 connected 1 0
207.63.144.0/24 DEFAULT_VLAN 1 connected 1 0
207.63.145.0/24 192.168.101.6 10 static 1 1
207.63.146.0/24 192.168.101.3 10 static 1 1
207.63.147.0/24 192.168.101.8 10 rip 2 120
207.63.148.0/24 192.168.101.3 10 rip 2 120
209.7.74.0/24 192.168.101.5 10 rip 2 120
209.7.75.0/24 192.168.101.6 10 rip 2 120
209.174.35.0/24 192.168.101.4 10 rip 2 120

[Olaf Borowski]

Scott,

One important piece was missing. You are NATting. The 7000 series router doesn't support NATting at wire-speed. The number you mentioned seems about right. Between 15-30 Mbit. That means, some networks get NATted and others don't. That may part due to misconfiguration on the 7000 series (ACL that determines what gets NATted or the routing. Without the configs of the routers, not too much I can do.

[Scott.D]

Nope - no NAT internally. Only NAT is at the Internet Gateway which is an ISA server. All addresses inside are real. They are true Class C's in most cases but just not publicly available.

I have open case with HP on this now as well. They are also looking at it. So far they have said the same as everyone else. Setup looks fine and it should be working.

Thanks though,

Scott

[Case Van Horsen]

Scott,

This really sounds like a speed/duplex issue. Are they converting between fiber and copper with an external module? If so, some converters use switches for manual configuration of speed and duplex. They may negotiating one way but be manually forced a different way.

Also, fiber connections do not negotiate speed/duplex settings. They are effectively forced to 100/Full. So the convertors and your equipment should probably also be forced to 100/Full.

Or the converter is damaged.

Just a thought, based on memories an old problem...

casevh

[Scott.D]

Their handoff to my equipment is ethernet - both sides were set to 100Mbps/Full Duplex. If they put their Cisco on Auto/Auto - it was seeing 100Mbps/Half Duplex.

Strange thing again - it was all working and I have 5 other sites - same equipment for them and me that have the same cosmetic issue but work full speed.

The SP did change all their ports to Auto/Auto and mine to Auto/Auto last night and this monring. Both sides agreed on 100Mbps/Full Duplex. Same issue though.

Thanks,

Scott