High latency in our internal network: intermittently

[cyberbozzo]

Dear guys,

Since a few weeks, we are experiencing high latency in our internal LAN:
- No VLAN's are in place.
- +/- 50 clients and phones (in total) are connected to our LAN.
- 2 HP 1920G's (they are replaced by HP because of my support ticket: troubleshooting).

The problem is intermittently: outside of business hours there is a little bit of latency: +/- 6 seconds sometimes to hosts on the same switch.
During business it went very well for several hours (+/- 3ms latency on hosts on the same switch), but another hour there is 300/400/500ms latency to hosts on the same switch.

- Loop Detection is enabled on the switches: no loops detected.
- All switches are on the newest firmware.

How can I troubleshoot this? I can work with wireshark, but it's very difficult for me to analyse/interpret this.

Where/how should I start to troubleshoot this?

Kind Regards,
Pieter

[TerjeAFK]

I've never used 1920 switches, but here are a few things I would check:

* Port counters: check error, drop and traffic counters. Are there a lot of errors on some ports? Are there ports that are way above others in traffic?

* How are the two switches connected together? If you have no VLANs then I am guessing that it's just though a regular access port in vlan 1.

* Do you have just these two switches, or is this part of a bigger network?

* How are you measuring the latency? Pinging between clients?

* Have you connected some new clients or phones in the last few weeks that may have defective or buggy NICs?

* Check the switch logs for clues.

* Are all switch ports set to Auto speed/duplex? If so, check the negotiated speed and duplex on all ports.

[parnassus]

You wrote +/- 6 ms and +/- 3 ms...so are you arguing about variation of latencies (Jitter) or are you just saying that the ping response time happens to be respectively below 6 ms and 3 ms (<6 ms, <3 ms) when measured host-to-host within the same switch or between switches in different moments of the day?

Generally, in normal average condition, the RTT (Round Trip Time, the ping delay time) can be well below 1 ms (it can easily be a fraction of 1 ms, let say 250 us or 0.250 ms as example) and there is very low Jitter (variation of Delay) but the OWD (One Way Delay, the delay time calculated between synchronized hosts) can be a little higher depending on Network Topology and number of Hops. Generally, in a scenario similar to your one, the RTT and OWD are practically similar (no Network Topology complexity exists and you're on a LAN with just two Switches interconnected).

As suggested I would check carefully your Network Topology and physical connections NICs/Cables (for testing try to use two hosts where the NICs are OK and link speeds are 1 Gbps) and also your Switches configurations: have a look to their diagnostic informations (you can save Diagonstic Informations and open them with a text editor).

[Vince-Whirlwind]

+/- 6 seconds

300/400/500ms latency

Are both way too high.

First thing I would do is rule out any issue with the hosts themselves. Not sure how you will do that.

Second thing you should monitor the switch CPU and the interface traffic statistics to see if the latency is actually justified by the switch being under some kind of load.

Maybe reduce what is connected to it, perhaps by going in out of hours and spending some hours monitoring it as you gradually re-enable switchports and see what happens.

If not, I suppose you've found out that the 1920 isn't right for you...

[cyberbozzo]

Dear Guys,

Thanks for your suggestions. It's a very difficult problem to troubleshoot: I'm not always onsite (working as an external consultant), and problem is very intermittently. Today I spend another +/- 4 hours on troubleshooting: replaced all the switches (there were some 1810g's in the network too).

On my hypervisor (HyperV) I found a duplicate MAC address........ I've solved this conflicting MAC adress and I'm running a continuously ping to some hosts: they stay behind 1ms all the time (until now).

What do you think guys? Was this my root cause?

When I'm now pinging to hosts in the network (servers, clients, phones, thin clients and soo on), they are all good! But when I ping to my 2 routers (high availability setup with a virtual IP) or to my trusted interface of my firewalls I get latencies till 20ms? Is this normal? Is this just because ICMP has a low priority on these devices?
Also: a ping to the MGMT IP of the switches is going from 1ms until 15ms... Is this normal?

Please let me know, the office is closed the whole weekend, my users starts working again on monday....

Thanks! Kind Regards!

[parnassus]

What does a traceroute done against the VIP (Virtual IP) report?

What do you mean with "...or to my trusted interface of my firewalls..."?

Isn't your VIP (which should be the IP of your HA Default Gateway configuration) the IP Address used as the only Default Gateway for your entire LAN? It's unclear - at least to me - if you have two Routers in HA, both connected to the same ISP or if you have those ones plus an intermediate Firewall between them and the LAN...