Switches, Hubs, and Modems
1752295 Members
4926 Online
108786 Solutions
New Discussion юеВ

Re: Slow/unstable performance on 5300xl

 
Thomas Ingle
New Member

Re: Slow/unstable performance on 5300xl

Hi all. We had the same problem on our network. We run a pair of 5308XL's in our core and a mix of HP 4100gl,4000m,and 2524 switches. We originally had OSPF enabled on the 5300XL's but as we started migrating clients into VLANs we began to notice the occassional drop. The problem became really noticeable when we started installing clients via RIS (Remote Installation Services), or getting any other high traffic applications hitting the network. After some testing I ended up converting to static routes for everything and the problem seems to have gone away (or at least it is intermittent enough that the complaints have stopped). Before removing the OSPF I recorded several instances of dropped connections, long ping times, etc.

We still have our CPU spike up into the 80% and 90% range occasionally but we usually average 15 - 30 % CPU utilization.

NETWORK INFO
~1200 Clients / 25 Servers
Clients spread across 15 VLANS / 20 VLANS total
Two 5308XL Switches as core / layer 3 connected by a bonded pair of gigabit fibers.
Typically not a high traffic network, but has occasional bursts (usually IT related changes / updates / maintenance).
Jeff Brownell
Valued Contributor

Re: Slow/unstable performance on 5300xl

Hello,

I work with the ProCurve escalation team and am attemting to follow-up with the issues noted in this forum post to determine if you've ever found resolution to your respective issues on the 5300xl series. Not knowing your respective environments or issue to any particular depth, I am hoping that you have opened calls with hp/procurve support. If so, you shold have been given a case id for tracking. Could you please provide me with the hp case id's you were given? I will personally track them to see where they are at if you have not found resolution.

Please read through the latest release notes for E.09.03 (ftp://ftp.hp.com/pub/networking/software/59912127-0105b.pdf) and upgrade is if any of your issues appear addressed with this rev. There is also available code (upto E.09.11) that has been release and available which has further fixes. In particular, a fix for asymetric routing leading to elevated CPU utilization is available in E.09.10.

If you are still having a problem and have not opened a case with hp, please do so and provide me that case id. I will ensure it gets the proper attention. Provide which code version you are on.

If you post to this forum, I'll receive an email alerting me to the post...

Regards,
Jeff
IT_7
Advisor

Re: Slow/unstable performance on 5300xl

Hi Jeff,

What you tell us sounds great. I initially started this post, and when I discovered that several other people were dealing with the same problems and had contacted HP, I must admit that I did'nt do so myself. The main reason for this, was that (of what I could read) HP didn't have a clue of what was wrong, so the only thing they could do, was to meet up with some technicians and make some tests on your network, which I'm not interrested in.

Anyway, we are currently running on the 08.53 release and the rpoblem persists. I noticed that a 09.03 release was out, but read the release notes, and it didn't seem to solve this particular problem.
Furthermore it added tons of new features, and since this seems to be a performance issue, I didn't want to add even more load on the switches.

This new release 09.11 you're talking about sounds exciting, but I cannot find it on the download site for the ProCurves. Hasn't it been released yet? I would really like to try it out, since we're now running VoIP on a seperate subnet which requires routing, and it is really vulnerable to performance issues.

Rgds,
Rasmus
Jeff Brownell
Valued Contributor

Re: Slow/unstable performance on 5300xl

Rasmus,

The E.09.11 code has been officially released to support and is expected to be on the WEB approximately April 1st, 2005 (delay from code release to posting to the web). I would strongly urge you to open a call with hp support and reference this post. There are various perf issues noted in this news thread and it is impossible for me to tell if they are all related to the same exact underlying problem. My first impression is that they all are not related specifically to one single issue, but can not say with clarity unless able to reveiw the type of data needed to address a performance issue.

Unfortunately in a large enterprise such as hp, not everyone you speak with is going to be aware of every issue out there with any given product. But you can count on your issue be addressed to your satisfaction should you not feel you are getting the assistance you need. If you dont agree with the advise of the eng tech, just say so and they will get help from peers or their next level of support. If necessary, the case will get to us here in escalations.

In this one particular forum post case, if you open a call and do not feel you are getting the attention you need, please post the case id in this forum and I will follow up with the case owner (of course not all forum posts will be addressed as such. Only the ones where customers are certainly not getting the attention they feel they need). The first recommendation should be the code update to the E.09.11. If the call owner isnt aware of the latest revision, then just send them this post. You need not be concerned about the overhead of the added features since your existing config will be compatible with the newer code. You will not see an increase in overhead from the newer code features unless you enable them. I think LLDP is an exception but this can be disabled if you're concerned. As always, update your secondary flash with the newer code and boot the switch into the secondary flash. This way if something goes wrong, then a simple reboot (defaulting to primary flash) will put you back to the older code rev.

Other types of data relevant to address this performance type issues would be a network map showing your layout, show techs and network sniffer traces of the problem in action, a writeup of all the protocols you are using (which can be inferred from the show tech but not as clearly as your statemnets), as detailed a description as possible of which ports/clients/networks/subnets/etc that are known to be affected and which are not.

Here is the "skinny" of some issues addresses beyond E.09.03 upto E.09.11
├в ┬в Routing - Asymmetrical routing results in high CPU utilization and dropped packets.
├в ┬в XRRP - Different XRRP versions cause excessive event log messages...
├в ┬в QoS ├в DSCP tagging feature not working on routed packets.
├в ┬в Web ├в Java files now compiled using JDK 1.2. JDK 1.1 was used previously.
├в ┬в XRRP - XRRP Automatic Fail Over enhancement.
├в ┬в PIM ├в When PIM is enabled, all multicast streams managed by IGMP can be delayed whenever a Leave is received for any stream.

This forum post has caught the attention of the labs and they are very curious to determine what exactly we are dealing with since we apparently have some un-happy customers (we certainly do not wnat this!). The only way to do that is to have an active call open with hp support for us to to coordinate with the labs on.

At this point it does not "appear" that all issues noted in this post are stemmed from the same underlying issue, but to determine this, the level of technical assistance needed to address these perf issues is not really going to happen via
Carl Bralick_1
New Member

Re: Slow/unstable performance on 5300xl

We had just deployed 4 5300XLs as our backbone in the past week, and were experiencing the issues mentioned above. In our case we discovered that it was a spanning-tree port misconfiguration; the non-edge ports were set to edge. We changed the setting to non-edge and voila! No more errors/disconnects/collisions.

Carl B.
IT_7
Advisor

Re: Slow/unstable performance on 5300xl

We've had this problem before and after RSTP operation.

Re: Slow/unstable performance on 5300xl

Hi all,
at this week i finished the physical connections of our new hp switches, 2x 530x, 3x 28xx. They are all connected by single ge-links or ge-trunks. So at this morning i want to do some quick tests befor i shutdown our cisco switches and switch all our devices to hp. Convergence by MSTP works fine in our littel ring topology, we have no problem.
But the Inter-VLAN-Routing test results were horrible.
We use both sfp-ports on on 530x module and send packet sizes from 64 to 1500 byte from 253 sources to 253 destinations.
Without ACL and QoS we get a max. performance of 5 MBit/s on bidirectional and max. 8 MBit/s on unidirectonal traffic by 50% - 60 % CPU load. If we increase the load, the switch starts to drop packests and the available bandwidth decrease to 1-2 MBit/s by 30% - 35% CPU load.
Switching performances lookes good by 280 MBit/s with 64 byte packets bidir. traffic and 960 MBit/s with 64 - 1500 byte packets bidir. traffic.
So witch mistake i have made in such a simple routing scenario?
FYI we use the current 9.22 image, mstp, gvrp and 256 vlans.

Thanks for any help,
Thilo
r
Carl Bralick_1
New Member

Re: Slow/unstable performance on 5300xl

We finally had to disable spanning tree to clear up the performance problems. We spent 2 weeks troubleshooting with HP Tech Support and they insisted we had a loop. I personally traced each and every fiber strand on our campus to ensure that wasn't so. It is up and running fine now. Prior to turning off spanning tree, I upgraded the firmware to version 9.22 - this had no affect.

Re: Slow/unstable performance on 5300xl

Hi Carl,
to decide between routing or spanning-tree isn`t realy a choice, isn├В┬┤t it?

regards
Thi
Bjorn Tore Paulen
Frequent Advisor

Re: Slow/unstable performance on 5300xl

We did use RSTP in our network. HP claims - at least when there is a problem - that the draft states there can be only 7 switches in a STP domain. We saw that even though nothing was done in the network, our switches recalculated the STP domain every now and then. I actually have one switch in my network that recalculates every second. This, I think, is due to customer's equipment connected to it. However, recalculations should perform every time something is connected/disconnected to the STP domain. Our customers lost their VoIP sessions during recalculation; would be syrup for ~4 sec.

This is a healthy switch, not connected to any other STP enabled equipment. Note the "Time since last change" and "Topology change count":

Rapid Spanning Tree (RSTP) Information

STP Enabled : Yes
Force Version : RSTP-operation

Switch Priority : 32768 Hello Time : 2
Max Age : 20 Forward Delay : 15

Topology Change Count : 9
Time Since Last Change : 35 hours

Root MAC Address : 000a57-04bc00
Root Path Cost : 0
Root Port : This switch is root
Root Priority : 32768


This is a not that healthy switch, running 9.22:

Rapid Spanning Tree (RSTP) Information

STP Enabled : Yes
Force Version : RSTP-operation

Switch Priority : 0 Hello Time : 2
Max Age : 20 Forward Delay : 15

Topology Change Count : 11,127
Time Since Last Change : 1 secs

Root MAC Address : 000e7f-bd3700
Root Path Cost : 0
Root Port : This switch is root
Root Priority : 0



...go figure..