HP Cowmare 5800 Switch performance issue - File transfer is getting verys slow.

[Nitinh]

I have issue with my HP Comware Switch, I configured it as a server form switch and it is in IRF Mode.

Uptime is 157 Weeks.

from last couple of week we are faccing performance issue while doing file transfer from one server to another server. Another critical issue is the backup server is not able to complete its schedule backup due to switch performance as the file transfer is getting to slow.

Dear Cauligues the issue is critical for me. as our backup not completing . We have checked from server end where there is no issue found.  I have checked and replaced all the networking UTP cables if there is Layer one issue. still the problem is there.

I have checked Stp and nothing found. cPU usage is normal and remains constant on 18-20%.

Sharing you the interfaces output and the diagram for information

....

[SERVERFARM-DC-IRF]dis version
HP Comware Platform Software
Comware Software, Version 5.20.105, Release 1808P27
Copyright (c) 2010-2014 Hewlett-Packard Development Company, L.P.
HP A5800-48G Switch with 1 Interface Slot uptime is 153 weeks, 6 days, 4 hours, 54 minutes

HP A5800-48G Switch with 1 Interface Slot with 2 Processors
1024M   bytes SDRAM
4M      bytes Nor Flash Memory
512M    bytes Nand Flash Memory
Config Register points to Nand Flash

Hardware Version is Ver.C
CPLD Version is 003
BootRom Version is 220
[SubSlot 0] 48GE+4SFP Plus Hardware Version is Ver.C
[SubSlot 1] No Module
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

[SERVERFARM-DC-IRF]  dis link-aggregation verbose Bridge-Aggregation 7
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected
Flags:  A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
        D -- Synchronization, E -- Collecting, F -- Distributing,
        G -- Defaulted, H -- Expired

Aggregation Interface: Bridge-Aggregation7
Aggregation Mode: Dynamic
Loadsharing Type: Shar
System ID: 0x8000, 4431-92de-74a3
Local:
  Port             Status  Priority Oper-Key  Flag
--------------------------------------------------------------------------------
  GE1/0/23         S       32768    4         {ACDEF}
  GE1/0/24         S       32768    4         {ACDEF}
  GE2/0/23         S       32768    4         {ACDEF}
  GE2/0/24         S       32768    4         {ACDEF}
Remote:
  Actor            Partner Priority Oper-Key  SystemID               Flag
--------------------------------------------------------------------------------
  GE1/0/23         3       1        1000      0x1   , f8b1-5615-d47e {ACDEF}
  GE1/0/24         2       1        1000      0x1   , f8b1-5615-d47e {ACDEF}
  GE2/0/23         1       1        1000      0x1   , f8b1-5615-d47e {ACDEF}
  GE2/0/24         4       1        1000      0x1   , f8b1-5615-d47e {ACDEF}

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[SERVERFARM-DC-IRF]dis int Bridge-Aggregation 7
 Bridge-Aggregation7 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 4431-92de-74de
 Description: DELL-SERVER-PORT-1-2/23-24
 4Gbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 PVID: 1
 Port link-type: trunk
  VLAN passing  : 1(default vlan), 2-8, 10, 12-13, 20, 30, 35-36, 40, 49-51, 61-64, 66-67, 71, 73, 76, 79, 82, 100, 150, 200, 251
  VLAN permitted: 1(default vlan), 2-4094
  Trunk port encapsulation: IEEE 802.1q
 Last clearing of counters:  Never
 Last 300 seconds input:  111 packets/sec 59155 bytes/sec    0%
 Last 300 seconds output:  179 packets/sec 42289 bytes/sec    0%
 Input (total):  262122668435 packets, 345390017271333 bytes
         261471720753 unicasts, 522579120 broadcasts, 128368562 multicasts
 Input (normal):  262122668435 packets, - bytes
         261471720753 unicasts, 522579120 broadcasts, 128368562 multicasts
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
         0 CRC, 0 frame, - overruns, 0 aborts
         - ignored, - parity errors
 Output (total): 167003154559 packets, 57555230943328 bytes
         157587876010 unicasts, 5146383802 broadcasts, 4257074033 multicasts, 0 pauses
 Output (normal): 166991333845 packets, - bytes
         157587876010 unicasts, 5146383802 broadcasts, 4257074033 multicasts, 0 pauses
 Output: 0 output errors, - underruns, - buffer failures
         0 aborts, 0 deferred, 0 collisions, 0 late collisions
         0 lost carrier, - no carrier
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[SERVERFARM-DC-IRF]dis int gi 1/0/23
 GigabitEthernet1/0/23 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 4431-92de-74de
 Description: DEL-SERVER-CITRIX-01
 Loopback is not set
 Media type is twisted pair
 Port hardware type is  1000_BASE_T
 1000Mbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 Flow-control is not enabled
 The Maximum Frame Length is 10000
 Broadcast MAX-ratio: 100%
 Unicast MAX-ratio: 100%
 Multicast MAX-ratio: 100%
 Allow jumbo frame to pass
 PVID: 1
 Mdi type: auto
 Port link-type: trunk
  VLAN passing  : 1(default vlan), 2-8, 10, 12-13, 20, 30, 35-36, 40, 49-51, 61-64, 66-67, 71, 73, 76, 79, 82, 100, 150, 200, 251
  VLAN permitted: 1(default vlan), 2-4094
  Trunk port encapsulation: IEEE 802.1q
 Port priority: 0
 Last clearing of counters:  Never
 Peak value of input: 120634635 bytes/sec, at 2017-07-15 19:58:30
 Peak value of output: 72072967 bytes/sec, at 2017-01-12 14:38:18
 Last 300 seconds input:  48 packets/sec 9011 bytes/sec 0%
 Last 300 seconds output:  52 packets/sec 16765 bytes/sec 0%
 Input (total):  53931337778 packets, 64062180555913 bytes
         53776670718 unicasts, 90186683 broadcasts, 64480377 multicasts, 0 pauses
 Input (normal):  53931337778 packets, - bytes
         53776670718 unicasts, 90186683 broadcasts, 64480377 multicasts, 0 pauses
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
         0 CRC, 0 frame, - overruns, 0 aborts
         - ignored, - parity errors
 Output (total): 81423608431 packets, 27953377681496 bytes
         78760573285 unicasts, 1410141020 broadcasts, 1249939202 multicasts, 0 pauses
 Output (normal): 81420653507 packets, - bytes
         78760573285 unicasts, 1410141020 broadcasts, 1249939202 multicasts, 0 pauses
 Output: 0 output errors, - underruns, - buffer failures
         0 aborts, 0 deferred, 0 collisions, 0 late collisions
         0 lost carrier, - no carrier
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[SERVERFARM-DC-IRF]dis int gi 1/0/24
 GigabitEthernet1/0/24 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 4431-92de-74df
 Description: DEL-SERVER-CITRIX-02
 Loopback is not set
 Media type is twisted pair
 Port hardware type is  1000_BASE_T
 1000Mbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 Flow-control is not enabled
 The Maximum Frame Length is 10000
 Broadcast MAX-ratio: 100%
 Unicast MAX-ratio: 100%
 Multicast MAX-ratio: 100%
 Allow jumbo frame to pass
 PVID: 1
 Mdi type: auto
 Port link-type: trunk
  VLAN passing  : 1(default vlan), 2-8, 10, 12-13, 20, 30, 35-36, 40, 49-51, 61-64, 66-67, 71, 73, 76, 79, 82, 100, 150, 200, 251
  VLAN permitted: 1(default vlan), 2-4094
  Trunk port encapsulation: IEEE 802.1q
 Port priority: 0
 Last clearing of counters:  Never
 Peak value of input: 78961241 bytes/sec, at 2016-09-05 12:34:58
 Peak value of output: 108269250 bytes/sec, at 2016-08-05 10:31:55
 Last 300 seconds input:  9 packets/sec 4452 bytes/sec 0%
 Last 300 seconds output:  67 packets/sec 18923 bytes/sec 0%
 Input (total):  18872132089 packets, 21354702475267 bytes
         18732730755 unicasts, 120180593 broadcasts, 19220741 multicasts, 0 pauses
 Input (normal):  18872132089 packets, - bytes
         18732730755 unicasts, 120180593 broadcasts, 19220741 multicasts, 0 pauses
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
         0 CRC, 0 frame, - overruns, 0 aborts
         - ignored, - parity errors
 Output (total): 78538778289 packets, 27889603960684 bytes
         76725488869 unicasts, 864233134 broadcasts, 946101032 multicasts, 0 pauses
 Output (normal): 78535823035 packets, - bytes
         76725488869 unicasts, 864233134 broadcasts, 946101032 multicasts, 0 pauses
 Output: 0 output errors, - underruns, - buffer failures
         0 aborts, 0 deferred, 0 collisions, 0 late collisions
         0 lost carrier, - no carrier

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[SERVERFARM-DC-IRF]dis int gi 2/0/23
 GigabitEthernet2/0/23 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 4431-92de-7268
 Description: DEL-SERVER-CITRIX-03
 Loopback is not set
 Media type is twisted pair
 Port hardware type is  1000_BASE_T
 1000Mbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 Flow-control is not enabled
 The Maximum Frame Length is 10000
 Broadcast MAX-ratio: 100%
 Unicast MAX-ratio: 100%
 Multicast MAX-ratio: 100%
 Allow jumbo frame to pass
 PVID: 1
 Mdi type: auto
 Port link-type: trunk
  VLAN passing  : 1(default vlan), 2-8, 10, 12-13, 20, 30, 35-36, 40, 49-51, 61-64, 66-67, 71, 73, 76, 79, 82, 100, 150, 200, 251
  VLAN permitted: 1(default vlan), 2-4094
  Trunk port encapsulation: IEEE 802.1q
 Port priority: 0
 Last clearing of counters:  Never
 Peak value of input: 84898075 bytes/sec, at 2015-09-14 06:09:16
 Peak value of output: 80106569 bytes/sec, at 2017-09-02 10:38:59
 Last 300 seconds input:  11 packets/sec 1886 bytes/sec 0%
 Last 300 seconds output:  26 packets/sec 2619 bytes/sec 0%
 Input (total):  29127657798 packets, 34414889303902 bytes
         28950875286 unicasts, 153371612 broadcasts, 23410900 multicasts, 0 pauses
 Input (normal):  29127657798 packets, - bytes
         28950875286 unicasts, 153371612 broadcasts, 23410900 multicasts, 0 pauses
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
         0 CRC, 0 frame, - overruns, 0 aborts
         - ignored, - parity errors
 Output (total): 3784608986 packets, 795245091952 bytes
         989000886 unicasts, 1578715152 broadcasts, 1213937671 multicasts, 0 pauses
 Output (normal): 3781653709 packets, - bytes
         989000886 unicasts, 1578715152 broadcasts, 1213937671 multicasts, 0 pauses
 Output: 0 output errors, - underruns, - buffer failures
         0 aborts, 0 deferred, 0 collisions, 0 late collisions
         0 lost carrier, - no carrier

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[SERVERFARM-DC-IRF]dis int gi 2/0/24
 GigabitEthernet2/0/24 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 4431-92de-7269
 Description: DEL-SERVER-CITRIX-04
 Loopback is not set
 Media type is twisted pair
 Port hardware type is  1000_BASE_T
 1000Mbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 Flow-control is not enabled
 The Maximum Frame Length is 10000
 Broadcast MAX-ratio: 100%
 Unicast MAX-ratio: 100%
 Multicast MAX-ratio: 100%
 Allow jumbo frame to pass
 PVID: 1
 Mdi type: auto
 Port link-type: trunk
  VLAN passing  : 1(default vlan), 2-8, 10, 12-13, 20, 30, 35-36, 40, 49-51, 61-64, 66-67, 71, 73, 76, 79, 82, 100, 150, 200, 251
  VLAN permitted: 1(default vlan), 2-4094
  Trunk port encapsulation: IEEE 802.1q
 Port priority: 0
 Last clearing of counters:  Never
 Peak value of input: 110123280 bytes/sec, at 2000-07-25 08:00:05
 Peak value of output: 75662036 bytes/sec, at 2017-09-02 10:39:04
 Last 300 seconds input:  5 packets/sec 1145 bytes/sec 0%
 Last 300 seconds output:  14 packets/sec 2449 bytes/sec 0%
 Input (total):  160191547079 packets, 225558246420412 bytes
         160011450144 unicasts, 158840314 broadcasts, 21256621 multicasts, 0 pauses
 Input (normal):  160191547079 packets, - bytes
         160011450144 unicasts, 158840314 broadcasts, 21256621 multicasts, 0 pauses
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
         0 CRC, 0 frame, - overruns, 0 aborts
         - ignored, - parity errors
 Output (total): 3256176028 packets, 917008511885 bytes
         1112820980 unicasts, 1293297520 broadcasts, 847102253 multicasts, 0 pauses
 Output (normal): 3253220753 packets, - bytes
         1112820980 unicasts, 1293297520 broadcasts, 847102253 multicasts, 0 pauses
 Output: 0 output errors, - underruns, - buffer failures
         0 aborts, 0 deferred, 0 collisions, 0 late collisions
         0 lost carrier, - no carrier

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

[parnassus]

Why do you believe slowness is (or could be related to) your IRF setup?

IRF uptime is quite high (it looks like you didn't any maintenance update in at least 3 years!), not necessarily a bad thing but not necessarily a good thing too (for the reason above).

If I were you I will start to diagnose your Backup Server (globally and in details [*]) which is the destination, then the origin(s) and, in between, the IRF...but, despite counters weren't cleared [**], LACP aggregation group 7 shows peak member ports utilizations (Byte per second) quite high ranging - at different times - from 70 to 120 MB per second for traffic leaving/reaching each link.

That's to say that you need to clear counters and start measuring, over time, link aggregation performances to your Backup Server before saying that Switches slow down traffic from a Server to another one.

[*] As example: what Logs report on your Backup Server before, during and after the failed backup operation? what's about filesystem free space? Does your Server have enough free space to let the backup operation to finish? and so on...

[**] See, as example, here (use reset counters interface [interface-type [interface-number]] CLI Command to reset Interface's counters before starting your diagnostic operations).

[peyrache]

Shall you post result of display diag please
Thanks
JY

[VoIP-Buddy]

Hi Nitinh!

There are a couple of things going on here.

First, 153 weeks is way too long without a reboot.  You could have memory leaks going on and that can have an impact on performance.

That said, R1808P27 is 3 years old!  My immediate recommendation is to upgrade to the latest version.  If you were to log in a support case, we would tell you to do that as Engineering will not fix code that old.

So you could stop 2 birds with one stone here... download and install the latest code, and reboot.

The latest code available is R1810P12.  You can find it here... https://h10145.www1.hpe.com/downloads/SoftwareReleases.aspx?ProductNumber=JC101A

David

[parnassus]

Impossible not to agree...