HP 2920 High CPU Usage After Approx. 25 days of runtime

[jeremy00]

I recently installed an HP 2920 as our main Gateway switch. It replaced an aging 2626 that finally gave up. It has been running now for a couple of months but twice now I have had to power cycle it due to high CPU load. After some weeks, so far its been about 25 days between reboots, the CPU will ramp up to 100%. SSH sessions will immidiatly terminate after a connection attempt and the WebUI will not load. The process that eventually consumes the most CPU is TFTP-6.

The only way to talk to it during this is through serial and all commands take some seconds or, depending on the amount of data the command processes (i.e show running-config), can take some minutes to complete. Switching and Routing continue to work but I have not left it in this state for to awful long so I'm not sure if core functionality would eventually degrade over time.

The first time this happened I simply rebooted the switch hoping it was a fluke. This time however I got some process readings before rebooting.

I was monitoring CPU load since the first time this happened so I caught it earlier then the first time. It initially starts with a spike that lasts a few minutes and then settles down, spikes for few minutes and the settles down, although as before initiating an SSH sessions imidiatly closes. Trying to log into the WebUI will get you the login page but imidiatly after entering the username and password, the page will hang (by this I mean a white page with a running cursor and it never loads the UI). If you watch show cpu process while trying to log into the WebUI you will see the CPU will ramp up to 100% with IPsec-3 and TFTP-6 processes consuming half each of the CPU (usually something like 42% / 48% respectively). Closing the Web page will cause the CPU to return to normal levels after some minutes but again trying to log into the WebUI will result in the CPU ramping-up.

I left it alone for most of the day this go around while I was trying to research this issue and found that the CPU will eventually do what it did the first time and ramp up to a constant 100% no matter if you are trying to log into the WebUI or not and that TFTP-6 is again consuming most of the CPU (I did do a show cpu process the first time this happened and it was also TFTP-6 that was consuming the most CPU). After getting some process readings, I did a reload, which took 5+ minutes just for it to ask if I was sure, so I'm good for another 25-ish days.

I'm hoping I might get some advice on what I could check to see why this is happening. Maybe some things I can look at over the course of the coming weeks to see if something strange is going on. Our configuration is pretty basic, our network isn't that complex so this switch isn't even really working all that hard most of the time.

The one major thing I am doing different at my IT managers request is transition to snmpv3 with this switch being the first to have it enabled. I'm currently using PRTG to monitor some basic statistics. I'm currently pulling traffic statistics off of 3 ports, monitoring chassis temp and CPU load. I mention this becuase I know snmpv3 can cause some overhead but I've been going through the release notes and looking for any CPU/memory leak issues related to snmpv3 but havn't seen any. other then that the config is exaclty the same as the old 2626 it replaced.

Please let me know if any other information might be helpful and thank you to anyone in advance who might have some thoughts on this.

Some maybe helpfull info taken during the high CPU issue:

Model: J9729A Aruba 2920 48G POE+ Switch.

show system:

Status and Counters - General System Information

  System Name        : SW1
  System Contact     :
  System Location    : 3rd Floor Server Room

  MAC Age Time (sec) : 300

  Time Zone          : -700
  Daylight Time Rule : Continental-US-and-Canada

  Software revision  : WB.16.05.0007        Base MAC Addr      : 9cdc71-f50b80
  ROM Version        : WB.16.03             Serial Number      : Info Erased

  Up Time            : 25 Days            Memory   -   Total   : 74,637,824
  CPU Util (%)       : 100                             Free    : 44,264,380

  IP Mgmt  - Pkts Rx : 646,039              Packet   - Total   : 6750
             Pkts Tx : 354,758             Buffers     Free    : 5009
                                                       Lowest  : 4922
                                                       Missed  : 0

SW1#

show flash:

SW1# sh flash
Image             Size (bytes) Date     Version
----------------- ------------ -------- --------------
Primary Image       :    16725144 03/27/18 WB.16.05.0007
Secondary Image     :    16713328 12/22/17 WB.16.05.0004

Boot ROM Version
----------------
Primary Boot ROM Version   : WB.16.03
Secondary Boot ROM Version : WB.16.03

Default Boot Image   : Primary
Default Boot ROM     : Primary

SW1#

[parnassus]

Can you share the sanitized configuration portion related to SNMPv3 setup?

I've SNMPv3 enabled and configured on various Aruba/HPE switches, all monitored by a proprietary NMS (HPE IMC) exactly using SNMPv3 with also SSH and Telnet accesses. I never saw such behaviour (although I haven't specifically Aruba 2920 model among them)...what I don't understand is why TFTP-6 (IPv6?) deamon goes so high...does PRTG use TFTP so extensively for monitoring purposes (or for pushing configurations?) so you need it to be enabled Switch side?

[jeremy00]

Thank you for the reply.

Currently we are only evaluating PRTG so it is only doing some basic stuff. I don't have it currently pulling configs or doing anything with TFTP or SSH at all. I only use SSH for management and If I need to pull a config, I currrently do so manually

what I don't understand is why TFTP-6 (IPv6?) deamon goes so high...does PRTG use TFTP so extensively for monitoring purposes (or for pushing configurations?) so you need it to be enabled Switch side?

We are not currently using ipv6. I'll have to dig into the PRTG configuration and see if it is doing something I don't know about with TFTP protocols but no, I don't need the ipv6 version of TFTP.

Note that snmpv2 is still enables since I am still testing/transitioning to v3.

Can you share the sanitized configuration portion related to SNMPv3 setup?

SW1# show run | i snmp
snmp-server community "*************"
snmp-server host 10.xxx.xxx.xxx community "*************"
snmp-server response-source dst-ip-of-request
snmp-server location "My Location"
snmpv3 enable
snmpv3 restricted-access
snmpv3 group managerpriv user "someUser" sec-model ver3
snmpv3 user "someUser"

SW1# sh snmpv3 user

 Status and Counters - SNMP v3 Global Configuration Information

  User Name                        Auth. Protocol   Privacy Protocol
  -------------------------------- ---------------- ----------------
  someUser                SHA              CBC DES

I'm thinking of disabling snmpv3 for a month since it is the only difference between this config and the old 2626 config just to rule it out.

Any other thoughts are appreciated. Thanks again for the reply