Remote Mirroring Loosing Traffic?

[Jeff Fern]

Hi all,

I am setting up remote mirroring to dupliacte traffic from our routers to a single point which can be connected to an IDS system. I am monitoring port statistics with Cacti and noticed that things don't quite add up.

The destination switch for the remote mirroring is only being used for this purpose, and the inbound traffic to this switch is 353MB/s. I using 2 interfaces as the destination ports, and they are currently at 50MB/s & 43MB/s. 50 + 43 != 353 (or anywhere close).

A source interface is currently showing 52MB/s in, 27MB/s out, collectively this is 79MB/s and is more than either of the 2 destination ports.

Does anyone have any idea what might be happening? I can't see any indication of dropped packets in the log or interface statistics.

[Tore Valberg]

Hi Jeff

What switch and what firmware are you running?

Tore

[Jeff Fern]

Thanks for the quick reply. The destination switch is a 5406, the source switches are either 5406 or 8212. They are all running k.14.47

[Jeff Fern]

Just to add a bit more information.

I've now disabled all of the remote mirroring except for 1. The soure switch is showing 25MB/34MB (in/out) and the destination switch is showing 63MB in. This is close enough and it looks like the remote mirroring is working correctly to get as far as the destination switch.

The destination port is currently showing 17MB out, which is only ~25% of the expected data.

Cheers,
-Jeff

[Mohammed Faiz]

One thing to check is your interface counter types are consistent in Cacti for each data source.
If I'm seeing strange values generated I usually check that mine are set correctly to "In/Out Bits 64 bit counters" and the "maximum value" for the data source field is also large enough.

[Jeff Fern]

I have checked Cacti, and everything is correct. We automatically add interfaces via a script which uses the exact same parameters for each itnerface.

I have connected the destination port to my laptop and running ethstats I am seeing ~10MB/s-15MB/s

The statistics from the switch port for Rates (5 min weighted average) shows:
Utilization Tx : 01.48 %

This is a 1GB port so 1.48% is around 15MB/s.

Everything seems to be pointing to packets being lost within the switch?!

[Tore Valberg]

Hi Jeff

You might want to Upgrade to K.14.60

http://h10144.www1.hp.com/customercare/library/switches/8212zl/index.aspx?pageTab=5

Fixed in K.14.49:
sFlow (PR_0000015656) â Outbound sampling using sFlow is not functioning.

Hope it helps

Tore

[Tore Valberg]

Sorry about that

I was thinking of the wrong thing :(

Tore

[Gerhard Roets]

Hi Jeff

When you mention "The destination port is currently showing 17MB out, which is only ~25% of the expected data." is this from within the IDS or on the switch'es interface counters ?

Just a quick check. How did you configure the source side of the mirroring ?

How did you set up the destination ? What is the destination port's vlan membership compared to the source port on the source switch ?

The vlan / routed path the traffic follows should all be jumbo capable are you seeing any indication on any inter switch links of giants ( show interfaces all ).

If your IDS system 802.1Q tag aware ?

Sorry for all the questions.

[Jeff Fern]

For the most part, the numbers I'm quoting are based on the graphs produced by Cacti, the 17MB out is from the latest graph (at the time).

I also did a test with 'ethstats' from a laptop connected to the mirrored port, this showed a similar figure.

The commands I've been using are:
[source]
mirror 2 remote ip 192.168.4.183 11832 192.168.4.187
interface Trk4
monitor all both mirror 2
exit

[destination]
mirror endpoint ip 192.168.4.183 11832 192.168.4.187 port A3

The source has several VLANs tagged (no untagged), and the destination is just the default untagged VLAN 1.

The VLAN I'm using to move the data around (with subnet 192.168.4.0/24) is dedicated just for port mirroring, and yes jumbo frames are enabled.

I'm presuming the IDS will be VLAN aware, the evaluation the company is sending is due to arrive next week - I'm trying to get this ready before hand. I have seen reference to being able to remove VLAN tagging and might investigate this however to begin with I just want to get all the traffic to the destination port.

Cheers,
-Jeff