HP 1920 - extreme slow VLAN routing

[yurezg2]

i have new HP 1920-24G Switch JG924A, Software Version 5.20.99 Release 1106

there are 2 vlans on it - vlan200 (servers) and vlan201 (workstations)

i want to route between these vlans and assign vlan ip to each of vlan, i can see routing table

all servers have default gateway as vlan200 ip

all workstations have default gateway as vlan201 ip

if i plug some workstation in vlan201 access port, then i have transfer speed between this ws and some server 100kbps-1mbps

if i change this port to vlan200 access port (server vlan), i have transfer speed on the same workstation between ws and server - 100Mbps.

also i have several workstations in vlan201 without this transfer speed problem.

my network topology is a little bigger then i describe here, but i think its not important at this time

have you any idea, why in some cases i have extremely slow network speed?

thanks

[yurezg2]

p.s.: enabling/disabling/deleteing antivirus software on both sides takes no effect

[johnk3r]

Good problem. Have you done the analysis using wireshark / tcpdump?

[yurezg2]

started on both sides, on workstation and server, wireshark shows the same pare Source-Dest (dest-source on the other side) with the same traffic ant layer 2 and at layer 3

vlan 1 -> vlan200 - speed id 100kbps-1mbps

just only changing vlan on access port of workstation (to vlan200, the same as destination server vlan) takes me to full speed

after 2 weeks testing i think its routing software error on l3 switch 1920-24g

also, another WS in vlan1, plugged at the same switch as previous WS, has no transfer speed problem while communitating wuth vlan200

[yurezg2]

i found a lot of records with "TCP segment of a reassembled PDU" in wireshark running n "slow" ws

trying to find any info about it

[Apachez-]

Could you paste current config along with port counters?

Could be faulty cable or interface (either at switch side or client side).

[yurezg2]

files you are asked for are attached

its config of switch that route my subnets

at fact i just began to employ vlans on my old running network, create some vlans and attach some servers / workstations to them

but routing is slow and its no matter, if server and ws attached to the same phisical switch (but different vlans) or to different switches

[Achim_B]

We have exactly the same Problems at our Office. We have 4 different 1920-24g (JG924A) and on all of them the same problems with Layer 3 Routing.
Is here at least one person who has “normal” speed on routed connections thru this Switch?
(on more than 2 VLAN and more than 2 computers)

[eggcurry]

Having this problem too. We have an HP 1920 24G with a small number of VLANS configured. Initially speeds are as you would expect but they drop off significantly after about 24 hours. Rebooting the switch seems to rectify the issue but speeds drop away again to a paltry 1-2 MB/sec.

Our ADSL router can do better than this and is significantly cheaper.

Anyone considering buying one of these ( HP 1920 24G ) please save yourself the hassle. They do not work properly and support is non-existant, just look at this forum.

[Achim_B]

We are folloing some ARP issues at the moment...after deleting the dynamic ARP entrys speed goes up again...

[eggcurry]

So at the moment this switch requires daily intervention just to do what it is meant to. Well done HP.

I'm guessing there is no way to schedule flushing the dynamic ARP table?

[Achim_B]

It is possible to reduce the aging time, but I don`t know if it helps.

we are currently in contact with the HP Support to geht that solved...


ARP is just suspected to be THE Problem...

[mp_AK]

Hey guys,

just wanted to share my experience on this case:

we got two 1920-16G Switches with several VLANS and are also experiencing similar problems.
Copying a big file from one VLAN to another with < 1MB/s.
Inside the same VLAN we can use the full bandwith.

After restarts or manually deleting the dynamic ARP entries, the speed goes up again.

So I can defenitely approve Achims assumption regarding some sort of ARP related issue.

[eggcurry]

Ok I have managed to get a little further.

The ARP table on the HP 1920 24G (JG924A) can only reliably handle 60 entries. Go over this and you will start to experience the loss in performance reported here.

In English, If you have more than 60 devices across all of your VLANS you will have this problem. This is a pretty significant limitation for what is meant to be a layer 3 switch. I also note than this is not mentioned in any of the devices specifications.

It is possible to set the maximum age for ARP entries quite low and this may help mitigate the problem if you are only a few devices over the 60 limit.

To set the maximum ARP age:

1. log on to switch via console or SSH

***next step will invalidate your warranty unless you are doing so under the instruction of an HP engineer***

2. enable the command line mode ( _cmdline-mode on ) with password ( Jinhua1920unauthorized )

3. enter system mode ( system )

4. set the ARP timer ( arp timer aging XXXX ) where XXXX is the value in minutes up to a maximum of 1440

For me 10 minutes seems to work ok. It doesn't get rid of the problem but it does move it about throughout the day and should give preference to those with heavier network usage demands.

If you haven't bought one of these don't.

If I were to write a review of the HP 1920 24G (JG924A) it would be scathing.

We are currently talking to our supplier about returning our 1920 24G as it is not fit for purpose. It was sold as a Layer 3 switch. The specifications issued by HP make no reference to only supporting a maximum of 60 ARP entries (the ARP table can actually hold many more).  If you contact HP support you eventually get this information but they will make you jump through hoops before they admit it.

This is a real shame as I have had excellent experiences with HP's networking devices up until now. Their attitude over this switch is beyond contempt though.

[16again · Accepted solution]

See:

http://h20565.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=7399514&docId=mmr_kc-0130409&docLocale=en_US

My comments:
-On 1920-24G I managed to reproduce this slow routing speed condition
-On 1920-24G there is no syslog event generated when ARP table grows beyond 64 entries.
-On 1920-24G the ARP table can't grow beyond 256 entries, so no CPU based routing for host #257! This condition does generate a  syslog message:"Oct 31 08:25:14 2015 HP1920G %%10TPMB/4/ARP TABLE FULL(t):   1.3.6.1.4.1.25506.2.38.1.2.4.1: ARP table is full and number of items is 256."
-HP doesn't specify IPv6 max neighbor table size,  so we can assume same limits apply. 
Note: On IPv6 number of hosts does not equal to neighbor table size! A host uses multiple addresses (with its link-local and privacy extension addresses)
On 1920-24G I did managed to get this IPv6 event at 256 entries:
"Oct 31 09:11:02 2015 HP1920G %%10TPMB/4/ND TABLE FULL(t):   1.3.6.1.4.1.25506.2.38.1.5.4.1: Neighbor table is full and number of items is 256."
-Switch reboot is seems useless to me , to make neigbors time out earlier use commands like:
 arp timer aging 4    (minutes, default=20)
ipv6 neighbor stale-aging 1    (hours! default=4)
After ARP table size is back within safe limits (60) , hardware speed returns

[yurezg2]

what a shame +1 !

Guys, thanks for reply.

[jbwoodoo]

Hi Folks

I had the same problem in just a small environment, just 40 VM’s and 25 workstations. I did not came across your blog about this issue, so I have struggled over a week on this.

Reading always helps a lot!

I just changed the switch to old Cisco 3750 and everything works fine. On all workstation I got file transfer throughput of more than 30 Mbyte/s between routed Vlans.

So what is this HP 1920 actually for? It says it is layer 3, but partially it is not!  

We are going to give back the switch to the supplier.  

Thanks & Regards,

Boris  

[killergooj]

Hi, All

How can I see current ARP routing usage.
I've same problem with my customer, but my team think problem is not ARP routing resource. How can I discuss this with my team.

Sorry for bad English,

Thank you in advance.

[parnassus]

By generating Diagnostic Information file?

Maybe on that file there will be ARP usage info...not totally sure if reported ARP info will help your team but look at this example (generated by HPE OfficeConnect 1920-48G):

=============================================================
  ===============display arp specification===============
=============================================================
ARP Specification:
Permitted Static ARP Number : 64
Permitted Total ARP Number : 256
Permitted Max Number perIf Configed: 256
Default Permitted Max Number perIf: 256
============================================================
  ===============display arp error slot 1===============
============================================================
Dump Errors in ARP processing:
 Error(s) in initialization: 1
 Error(s) in malloc memory:  0
 Error(s) in func parameter: 207
 Error(s) in Send QueueMsg:  0
 Error(s) in write taskevent: 0

 Error(s) in malloc IPC msg: 0
 Error(s) in malloc RPC msg: 0
 Error(s) in Create IPCGroup: 0
 Error(s) in Join IPCGroup: 0
 Error(s) in Reg WaitList: 0
 Error(s) in Get WaitList: 0
 Error(s) in Set WaitList: 0
 Error(s) in Create Timer: 0

 Error(s) in Send ARP Request: 0
 Error(s) in match static Entry: 0
 Error(s) in Add Arp Entry: 0
 Error(s) in Get Arp Entry: 0
 Error(s) in Delete Arp Entry: 0
 Error(s) in Hitting Arp Entry: 0
 Error(s) in Add Entry For RenewTime: 0

 Error(s) in Create IFCB: 0
 Error(s) in Create Interface Link: 0
 Error(s) in Create Interface LinkNode: 0
 Error(s) in Create Unprobe node: 0
 Error(s) in Create ref Node: 0
 Error(s) in Add Reference : 0

 Error(s) in Slot Insert : 0
 Error(s) in Slot Remove : 0
 Error(s) in IfIndex Insert : 0

 Error(s) in Add Drv Dummy: 0
 Error(s) in Del Drv Dummy: 0
 Error(s) in Add Drv : 0
 Error(s) in Del Drv : 0
 Error(s) in Mod Drv : 0
 Error(s) in Add ArpHost Drv : 0
 Error(s) in Del ArpHost Drv : 0
 Error(s) in Mod ArpHost Drv : 0

 Error(s) in Get Vrf index: 0
 Error(s) in Call IF_GetDataByIfIndex: 0
 Error(s) in Get Device Data: 0
 Error(s) in Get Vlan Data: 0
 Error(s) in Get Lacp Data: 0
 Error(s) in Get Interface Mac: 0

 Error(s) in fast synchronization: 0
 Error(s) in **bleep** Sing: 0
 Error(s) in smoothing process: 0
 Error(s) in Receive wrong HA Msg: 31882
 Error(s) in Process event: 0
 Error(s) in Syn event: 0
 Error(s) in Common: 28533945
===========================================================
  ===============display arp info slot 1===============
===========================================================
ARP Information:
Learning switcher: 1 (1 on, 0 off)
CockSingStatus: 0
ARPCockEndStatus: 0x00000000000000000
ARP DataCollectState:
   ulState: 2 (0 syn, 1 asy, 2 end)
   LinkHeadWaitListHandle: 0
   ListEntryWaitListHandle: 0
   pstArpLinkHead: 0x0
   pstLinkNode: 0x0
   usArpNumSent: 0
ARPSmoothFlag: 0
ARP KeepConsistent:
    WalkState: 0 (1 start  0 end)
    LinkHeadWaitListHandle: 0
    ListEntryWaitListHandle: 0
    pstArpLinkHead: 0x0
    pstLinkNode: 0x0
ARP **bleep** Age:
    WalkState: 0 (1 start  0 end)
    LinkHeadWaitListHandle: 0
    ListEntryWaitListHandle: 0
    pstArpLinkHead: 0x0
    pstLinkNode: 0x0
    pstLinkNode: 0x0
    WalkState: 0 (1 start  0 end)
    LinkHeadWaitListHandle: 0
    ListEntryWaitListHandle: 0
    pstArpLinkHead: 0x0
    pstLinkNode: 0x0
ARP BlackHoleEnable: 0
ARP Default BlackKeep: 0
ARP BlackHoleKeep: 0
ARP ProbeLink Max-Len: 0
ARP Reload Timer: 10000 ms
ARP SendSeq: 0
ARP IPC Queue Len: 0
ARP Detection Information
    Max Vlan ID: 4094
    Max config Vlan number: 1024
    Current configed Vlan number: 0
    Valid check config: 0
    Vlan bit map length(byte): 512
    Vlan bit map address: 0x83d3c820
ARP RateLimit Information
    Default rate limit: 4294967295
    Min rate limit: 5
    Max rate limit: 100
ARP ActiveAck Information:
    Enable: 0
    Current ActiveAck Link Count: 0
ARP SrcMAC ValidCheck: 0
ARP SrcMACFixed Information:
    Fixed Mode: 0
    Current AgeTime: 300
    Current Threshhold: 50
    Current HashNode Count: 0
    Current AttackLink Count: 0
    Current ExcludeMac Count: 0
    Max ExcludeMac Count: 10
    Current DropMac Count: 0
    Max DropMac Count: 200
   Function Form DHCPC: 0x80aee270
   Function Form DHCPR: 0x81535330
   Function Form PORTAL: 0x0
ARP SlotNotifyInfo(0x840d3bd0) slot( )
=================================================================
  ===============display arp statistics slot 1===============
=================================================================
Dump ARP Statistics Data:
All Valid ARP Number: 46
All Static ARP Number     : 0
Unresolved Incomplete Static  ARP Number : 0
Resolved Incomplete Static  ARP Number : 0
Valid complete Static  ARP Number : 0
Invalid complete Static  ARP Number : 0
All Dynamic(unresolved + resolved) ARP Number : 46
Unresolved(Not Valid) Dynamic ARP Number : 0
Resolved Dynamic ARP Number   : 46
Multiport ARP Number          : 0
Permitted Static ARP Number : 64
Permitted Multiport ARP Number : 0
Permitted Total ARP Number : 256
Permitted Max Number perIf Configed: 256
Default Permitted Max Number perIf: 256
Permitted Dynamic ARP Number : 0
ARP Input Counter         : 31672644
ARP Resolved Counter      : 32411
Timer Entering Counter    : 0
Failed Update Fdb Counter : 0
Static Reload Timer       : 0
ARP max ipc queue len     : 0
Last Error Code           : 0
ARP Drop Counter          : 0
ARP Static Item Counter   : 0
ARP Dynamic Item Counter  : 0
ARP Dummy Counter         : 0
ARP ProbeNode Counter     : 0
ARP Probe Request Counter : 0
ARP Mbuf Num              : 0

 Remembering that HPE OfficeConnect 1920 Switch series is Comware 5 operating system based (even if with limited features WRT other more expensive and features rich Comware 5 HPE platforms).