Slow/unstable performance on 5300xl

[IT_7]

Hi,

We have 4 5300xl's in a VLAN setup with routing enabled connected with gigabit fibers.

When e.g. pinging a server on the main switch from a node on one of the other switches AND in another subnet/VLAN the response times are not at all satisfactoriuos (probably spelled that wrong :0) ).

I know that the routing process will slow the traffic down a little bit, and the response times are usually 4-6ms which should be OK.

However, every tenth time (or so) there are times at over 200ms to almost 1sec. I've even experienced drop-outs with a single "destination unreachable" every now and then.

Now my question to you is: Why could this be happening? Within the same VLAN across more switches there are no problems, so it must be something with the routing process. FYI there's no ACL's configured on the implied VLANs and routing is only enabled on the main switch.

Hope one of you have a good suggestion to what this could be caused by.
Thanks in advance,

Rasmus

[Markku Leinio]

We have been experiencing the same problems from the beginning. HP folks here in Finland have been investigating it, but no solution has been found. We are a 9000 people organization (a polytechnic), so I guess HP should be at least somehow interested in solving the case.

We are testing some HP 9300 routers and Catalyst 3550 switches as well, no problems with them in the same topology.

[IT_7]

Will you kindly let my know if you receive any updates on this? Thanks.

Rasmus

[koppendorfer]

i have installed 2 5308xl as core-routers and
20x 2650 as edge switches with 9 vlans over
all switches, running the two 5308 with ospf and xrrp. routing between all vlans is working fine netio test is over 11000kb/s
so no problem there. have updatet all switches with newest software

[Markku Leinio]

We have been using all possible firmwares, also unpublished versions (directly from HP). Firmware versions haven't had very much effect on the problems.

If I hear something new about this, I'll tell here.

[OLARU Dan]

Maybe the default route is the culprit. Take a look at page 16-6 of the attachment; in Table 16-1, last row (Default network route).

It reads: "For the Switch 5300XL Series devices, enter an explicit default route (0.0.0.0 0.0.0.0 or 0.0.0.0/0) as a static route in the IP route table."

Maybe if you type this into your config you'll get rid of the problem.

[Markku Leinio]

We are already using static routes, the network topology is so simple.

[OLARU Dan]

What these guys are saying is that you HAVE TO TYPE the default network route into the config as soon as you start IP routing on the 5300XL series switches.

[Markku Leinio]

If they are STATIC ROUTES, how would you think they appear in the router, if we wouldn't have entered them there?

So, of course we have entered the default route as well.

[OLARU Dan]

Markku,
you can have STATIC ROUTES in your routing table without this necessarily meaning you have a DEFAULT ROUTE. It is very good you have defined a DEFAULT ROUTE, as per HP's recommendations and, yes, I know the DEFAULT ROUTE is one of your STATIC ROUTES.

But I am not sure these clarifications solved Rasmus's problem.

[Markku Leinio]

You are correct. Have you seen any router (other than Internet backbone routers) connected to Internet, having static routes, and not having a default route?-) (Ok, I assumed Internet connectivity (not an isolated private network) here, my mistake on that.)

To stay in the subject, the issue experienced by Rasmus and me is not routing table related. The problem is deeper in the routing engine of Procurve 5300XL. Sometimes it just takes a lot of time to do its task, and that should not happen.

HP engineers brought Catalyst 3550's to our premises as well, for backup devices:-)

[Quartino]

Same problem for us ... since february. No solution from hp

[IT_7]

OK, sounds like this is some major bug in their firmware or whatever.

Markku, since it sounds like HP are actively working at your site, can you ask them when they expect to have a solution? I've send an e-mail to HP, but it is a big company...

Thanks,
Rasmus

[Einar_3]

Rasmus;

Do you experience any Drops TX increasing error count on any interfaces?
We're also investigating a similar error condition on 5300XL with Procurve support.

Regards,
Einar

[IT_7]

As I wrote earlier, when doing a series of pings sometimes I receive a single timeout. Besides that it is just generally slow response times at several hundreds of milliseconds.

[IT_7]

I recently discovered that the back-end device doing all the routing (which seems to be the problem) is running a constant CPU load of 40% to 65%!!!
Any of you guys having this problem, experiencing the same?

I've looked at the traffic monitor in PCM+, but there's not very heavy load on the switch at all.

Running firmware version E_08_42.

[Quartino]

Hi,
proc is running at 15% and the problem is the same on multiple 530X with every os tested(not tested 8_50 yet on our main switches)

cpu load doesn't seem to be the problem

regards

[Paul Vieira]

Hello,

We are having the exact same issues. We have 4 5308xl's as the core with 12 HP2524 switches using 4 vlan's. Network slowness is on and off. We have investigated and changed firmware, adjusted edge devices, looked into PIM, also default routes, etc.. etc.. called HP and they seem to be shocked that there is an issue meanwhile I read the forums and seems this an apparent global issue. HP has no resolve to this.

[randy bentley_1]

5308 problems also. I have found a way to make ours happen at will though. I can connect a computer, another switch, anything with a MAC address. The routing stops working for IP traffic for 5 seconds to maybe 2 minutes.

But it only seems to effect our netware servers, the MS servers don't miss a beat.

[IT_7]

Hi,

Has anyone any experience whether the newest firmware releases for the 5300xl series (08_50 and 08_53) solves this issue?

Rgds,
Rasmus

[Thomas Ingle]

Hi all. We had the same problem on our network. We run a pair of 5308XL's in our core and a mix of HP 4100gl,4000m,and 2524 switches. We originally had OSPF enabled on the 5300XL's but as we started migrating clients into VLANs we began to notice the occassional drop. The problem became really noticeable when we started installing clients via RIS (Remote Installation Services), or getting any other high traffic applications hitting the network. After some testing I ended up converting to static routes for everything and the problem seems to have gone away (or at least it is intermittent enough that the complaints have stopped). Before removing the OSPF I recorded several instances of dropped connections, long ping times, etc.

We still have our CPU spike up into the 80% and 90% range occasionally but we usually average 15 - 30 % CPU utilization.

NETWORK INFO
~1200 Clients / 25 Servers
Clients spread across 15 VLANS / 20 VLANS total
Two 5308XL Switches as core / layer 3 connected by a bonded pair of gigabit fibers.
Typically not a high traffic network, but has occasional bursts (usually IT related changes / updates / maintenance).

[Jeff Brownell]

Hello,

I work with the ProCurve escalation team and am attemting to follow-up with the issues noted in this forum post to determine if you've ever found resolution to your respective issues on the 5300xl series. Not knowing your respective environments or issue to any particular depth, I am hoping that you have opened calls with hp/procurve support. If so, you shold have been given a case id for tracking. Could you please provide me with the hp case id's you were given? I will personally track them to see where they are at if you have not found resolution.

Please read through the latest release notes for E.09.03 (ftp://ftp.hp.com/pub/networking/software/59912127-0105b.pdf) and upgrade is if any of your issues appear addressed with this rev. There is also available code (upto E.09.11) that has been release and available which has further fixes. In particular, a fix for asymetric routing leading to elevated CPU utilization is available in E.09.10.

If you are still having a problem and have not opened a case with hp, please do so and provide me that case id. I will ensure it gets the proper attention. Provide which code version you are on.

If you post to this forum, I'll receive an email alerting me to the post...

Regards,
Jeff

[IT_7]

Hi Jeff,

What you tell us sounds great. I initially started this post, and when I discovered that several other people were dealing with the same problems and had contacted HP, I must admit that I did'nt do so myself. The main reason for this, was that (of what I could read) HP didn't have a clue of what was wrong, so the only thing they could do, was to meet up with some technicians and make some tests on your network, which I'm not interrested in.

Anyway, we are currently running on the 08.53 release and the rpoblem persists. I noticed that a 09.03 release was out, but read the release notes, and it didn't seem to solve this particular problem.
Furthermore it added tons of new features, and since this seems to be a performance issue, I didn't want to add even more load on the switches.

This new release 09.11 you're talking about sounds exciting, but I cannot find it on the download site for the ProCurves. Hasn't it been released yet? I would really like to try it out, since we're now running VoIP on a seperate subnet which requires routing, and it is really vulnerable to performance issues.

Rgds,
Rasmus

[Jeff Brownell]

Rasmus,

The E.09.11 code has been officially released to support and is expected to be on the WEB approximately April 1st, 2005 (delay from code release to posting to the web). I would strongly urge you to open a call with hp support and reference this post. There are various perf issues noted in this news thread and it is impossible for me to tell if they are all related to the same exact underlying problem. My first impression is that they all are not related specifically to one single issue, but can not say with clarity unless able to reveiw the type of data needed to address a performance issue.

Unfortunately in a large enterprise such as hp, not everyone you speak with is going to be aware of every issue out there with any given product. But you can count on your issue be addressed to your satisfaction should you not feel you are getting the assistance you need. If you dont agree with the advise of the eng tech, just say so and they will get help from peers or their next level of support. If necessary, the case will get to us here in escalations.

In this one particular forum post case, if you open a call and do not feel you are getting the attention you need, please post the case id in this forum and I will follow up with the case owner (of course not all forum posts will be addressed as such. Only the ones where customers are certainly not getting the attention they feel they need). The first recommendation should be the code update to the E.09.11. If the call owner isnt aware of the latest revision, then just send them this post. You need not be concerned about the overhead of the added features since your existing config will be compatible with the newer code. You will not see an increase in overhead from the newer code features unless you enable them. I think LLDP is an exception but this can be disabled if you're concerned. As always, update your secondary flash with the newer code and boot the switch into the secondary flash. This way if something goes wrong, then a simple reboot (defaulting to primary flash) will put you back to the older code rev.

Other types of data relevant to address this performance type issues would be a network map showing your layout, show techs and network sniffer traces of the problem in action, a writeup of all the protocols you are using (which can be inferred from the show tech but not as clearly as your statemnets), as detailed a description as possible of which ports/clients/networks/subnets/etc that are known to be affected and which are not.

Here is the "skinny" of some issues addresses beyond E.09.03 upto E.09.11
â ¢ Routing - Asymmetrical routing results in high CPU utilization and dropped packets.
â ¢ XRRP - Different XRRP versions cause excessive event log messages...
â ¢ QoS â DSCP tagging feature not working on routed packets.
â ¢ Web â Java files now compiled using JDK 1.2. JDK 1.1 was used previously.
â ¢ XRRP - XRRP Automatic Fail Over enhancement.
â ¢ PIM â When PIM is enabled, all multicast streams managed by IGMP can be delayed whenever a Leave is received for any stream.

This forum post has caught the attention of the labs and they are very curious to determine what exactly we are dealing with since we apparently have some un-happy customers (we certainly do not wnat this!). The only way to do that is to have an active call open with hp support for us to to coordinate with the labs on.

At this point it does not "appear" that all issues noted in this post are stemmed from the same underlying issue, but to determine this, the level of technical assistance needed to address these perf issues is not really going to happen via

[Carl Bralick_1]

We had just deployed 4 5300XLs as our backbone in the past week, and were experiencing the issues mentioned above. In our case we discovered that it was a spanning-tree port misconfiguration; the non-edge ports were set to edge. We changed the setting to non-edge and voila! No more errors/disconnects/collisions.

Carl B.