Switches, Hubs, and Modems
cancel
Showing results for 
Search instead for 
Did you mean: 

HP3500 losing configuration and poor ping response

SOLVED
Go to solution
Drew_38
Frequent Advisor

HP3500 losing configuration and poor ping response

We've recently installed a 3500yl-24G (J8692A) running latest software K.11.33. We have two problems with it.

1) It frequently loses the ip rip information for a particular VLAN when making a change to the configuration file. In the example attached I have some ip rip commands for VLAN 226. Sometimes when I made a further change to the config file the rip entries vanish. At first I didn't beleive my eyes but it had done this about 4 times now which is annoying as I lose connectivity with the switch!

2) It suddenly responds slowly to pings. Normally I have a ping response of less then 1ms. When I send 100 or more pings (size 16byte) it responds a couple of time with 500 or 600 or 700ms, or even drops altogether. There are only about 50 devices connected via this switch via other HP 26xxx edge switches, so it's hardly overloaded. The previous device we had in this location responded ok so I'm sure there is not a physical problem with the fibre. I should also add we have at 5406zl running K.11.33 in a totally different building and that is also responding in the same poor way to pings. The 53xx and 34xx series switches we have connected elsewhere in a similar fashion don't have such problems.

Anyone else experiencing the same as we are? It seems to me that there is a problem with this release of software.

Thanks in advance for any help;

Drew
29 REPLIES
Matt Hobbs
Honored Contributor

Re: HP3500 losing configuration and poor ping response

With the RIP issue you're seeing, I'd recommend you open a case up with HP support if you can reproduce it easily.

For the slow pings, this is not unusual behaviour as the task that responds to pings addressed to the switches IP runs at a low priority. Normal traffic takes precedence.

I believe if you set a 'management-vlan' type it will improve the ping response reliability, however the management-vlan type is not routable so you have to take that into consideration before using this feature.





Drew_38
Frequent Advisor

Re: HP3500 losing configuration and poor ping response

In reponse to Matt I should have added that users who are on the xxx.yy.226.0 subnet are also complaining about periodic 'slowness' and when they ping a server they are also getting occasional 600-800ms responses when from elsewhere on site 1ms approx is more common. So it's not just the switch which is responding to pings slowly there seems to be a more general issue.

Regarding the rip configuration thanks for the suggestion I'll raise a case with HP.

Would stil be curious if anyone else has these switches and is having problems though.

Drew
Drew_38
Frequent Advisor

Re: HP3500 losing configuration and poor ping response

Hi all just to add further to my own post here. Regarding the rip configuration problem I describe above I have done a little more testing just to try and make sure I'm not going mad.

What is happening is that when I make a change (such as changing the switch IP number) to the 54xx or 35xx from the Menu then the vlan rip information is deleted. When I make the same change from command line it does not delete the rip information. I am guessing the Menu system rather stupidly deletes information which it can't configure itself (such as the rip info).

I've seen this before on the 53xx series: the Menu system is unable to make changes to the configuration properly. Makes me think HP really are not testing their software properly. It's very annoying; I wasted about two hours of my time earlier this week as I had to phyiscally visit a 35xx switch which I'd lost access to due to this.

Do HP offer prizes for people who find bugs? That'll cheer me up!

Drew
Matt Hobbs
Honored Contributor

Re: HP3500 losing configuration and poor ping response

Hi Drew,

HP do offer prizes but I found out today that this bug has already been reported so you don't get the prize! (just kidding about the prizes) If you haven't already done so, contact HP and make them aware of this thread. That way they should be able to get you new firmware quicker once it has been fixed.

Alternatively as you have already found a workaround you could just wait until this bug appears as fixed in the release notes.

Don't forget to assign points to any replies that you receive here.

Matt
Drew_38
Frequent Advisor

Re: HP3500 losing configuration and poor ping response

Matt;

How did you find out the issue with the Menu system had been reported? Not that I mind reporting it as well (which I have done so that might speed up the fix slightly) but if there was a list of outstanding bugs which HP knew about at least we'd not be going mad trying to figure out what the problem is!

I realise HP might not want to publicise what is not 100% correct with their equipment but if I'd known it was an issue I'd not have done the change in the way I did and hence not wasted my time.

I've still an outstanding issue with the slow response users linked via the 3500 are getting, so I won't close this thread yet.

Drew
Mohieddin Kharnoub
Honored Contributor

Re: HP3500 losing configuration and poor ping response

HI
My friend Drew, i have some suggestions !!

1- Why don't you assing an IP Address for the Default-Vlan (Vlan1), since you've enabled IGMP on this vlan (The IGMP feature does not operate if the switch or VLAN does not have an IP address configured manually or obtained through DHCP/Bootp).

2- Have you tried to obtain the Premium license for the 3500 and 5300, so you can enable
IP PIM-dense.

I recommend you to check page 3-21 in this file: ftp://ftp.hp.com/pub/networking/software/6200-5400-3500-Multicast-and-Routing-Jan-06.pdf

Good Luck !!!

Science for Everyone
Matt Hobbs
Honored Contributor

Re: HP3500 losing configuration and poor ping response

Hi Drew,

I found out through a contact at HP. For any new issues like this your best bet is to see if anyone else has reported it here on the forums, otherwise with HP directly.

The issue was only reported a few days before you noticed it, so there is a good chance that if you do speak to HP - the person you speak to may not be aware of it and will need to speak to someone else to confirm it (which is why I suggested mentioning this thread). Otherwise they might have taken you through the whole escalation process which would be unnecessary at this point.

In regards to the second part of your issue (slow response), can you help to narrow it down any further? i.e, When it occurs, how long does it occur for? When it occurs is the CPU usage on the switch higher than normal? Is there anything in the switch event log? If using spanning-tree, are any topology changes occurring? Is there anything out of ordinary when you look through the port counters? Have you tried using anything to monitor the network like PCM+ Traffic Monitor?

Matt
Drew_38
Frequent Advisor

Re: HP3500 losing configuration and poor ping response

Hi all;

Sorry for delays in replies to my query. I do appreciate the attention in here. I've been having to do things like send a show tech to HP in The Netherlands as they are trying to reproduce the problem (even though it's a known bug by HP). That's to do with my first problem with the 3500 menu system.

Anyhow Mohieddin the reason there is no IP number on the default VLAN is that there should be nothing connected to the default VLAN. I put ip igmp on this VLAN on there as that was in when I originally configured and I didn't take it out. We do have a premium license for this device; at one point we did enable pim dense and it didn't make any difference. But it's good to know we're trying the right sort of things.

For Matt some more good questions thank-you. Basically pinging a server from a device connected via the 3500 sending 20 or so pings the normal response of less than 1ms but then in a seemingly random fashion 3 or so pings will suddenly respond with 600, 700 or 900ms, or even drop. So there's bursts of poor connectivity. Pinging the same server from elsewhere does not cause the problem. And I get the same problem from the HP5400 I have in service which is linked via a totally different fibre. There's nothing in the log when this is happening, the CPU is just ticking along at about 13% hardly changes. Port counters no errors. Tried it with spanning tree off and on (we don't normally configure spanning tree on routers).

Not tried PCM+ traffic monitor I can load it up and give it a go. But I suspect it'll not show anything.

Fundamentally the switch is configured in the same way we configure all our layer 3 switches, 5300s, 3400s etc. It's just performing poorly in the way I describe (and so is the 5400 we have).

Is there anyone here who has a layer 3 3500 and it's working fine? I'm curious to know!

Drew
Mike Satterfield
Occasional Advisor

Re: HP3500 losing configuration and poor ping response

Drew, while I didn't experience the config issue this time, I also had similar performance issues with release 11.33 on the 3500yl. In my case it was an upgrade from the 3400cl and 5300. Performance was so bad I had to pull the 3500s out and go back to previous gear.

The network seemed fine until it came under its normal daily load. Then, the ping times went to hundreds of miliseconds in a cyclical fashion. End-users experienced the same cycle with site to site traffic.

Procurve is still looking into the issue.

Mike
Matt Hobbs
Honored Contributor

Re: HP3500 losing configuration and poor ping response

You can add this one to the list as well:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1040132

Is it only routed traffic that is seeing the problem? I'm sure it will be, just want to make sure.

Do you think you could reproduce this in a lab setup? If I can get my hands on a 3500/5400 next week I'll see I can punish it enough to see it.
Drew_38
Frequent Advisor

Re: HP3500 losing configuration and poor ping response

Mike;

Very interesting comments thank-you. glad (if glad is the word) that it's not just me who is experiencing this. This is why it's tricky for me to reproduce it in a test environment as with no load the 3500 performs perfectly ok.

Matt;

Again thanks for spotting the other similar thread. Yes it is only traffic which is routed away from the 3500 which is experiencing slow ping reponses. Having said that I only have one subnet which has any users on it so pretty much everything they connect to is off the subnet.

I'll report this problem to HP then now I think I can construct a job to them which kinda makes sense.

I suspect basically it'll be something they need to sort out in the next software version. Wonder when that might be?

Drew
doubleH
Regular Advisor

Re: HP3500 losing configuration and poor ping response

Hi Guys,

I've worked with Matt before on some other performance issues and he has helped out large. I am having the same sort of issues Drew is describing. I am just starting to migrate on to new HP gear and am not having very good response times in some cases either. I have 2 5400's in the core connected to each other with a 4gig trunk with a 3500 edge device (not in use yet). All are running K.11.33. There have been no spanning tree configuration changes in 14 days.

I don't mean to hijack this thread, but would like to explain my current issue. If you would like I will post to a new thread...

Server 1, HP ML530, W2K Sp4, HP NC7771 Copper Gig Nic, both nic and switch port running 1000/Full. Attached to Core 2, A7

Workstation 1, HP with onboard Broadcom Gig Copper card, WinXP SP2 (no firewall), both nic and switch port are set to auto. Attached to Core 2, D10

If i load iperf on the server and run from the workstation I get 10mbit/sec which is ridiculous. Server is not under much load. If I then take that same workstation connection and plug it into Core 1 the speeds jump up to 250-300mbit/sec. It doesn't make sense to me why I get better speeds going between the 2 cores compared to being on the same core and getting down right nasty speeds. If I run the same test from a W2K workstation on my old network running at 100/Full (forced) I get 85mbit/sec which what I would expect.
doubleH
Regular Advisor

Re: HP3500 losing configuration and poor ping response

Just thought i'd post back. It's 8:30pm EST and everyone has gone home. When running the iperf tool from a workstation on Core 2 (same as server) I am now getting 200+mbit/sec whereas before I was getting 10mbit/sec. Could it be another device that is causing the slow down? I still have better results on Core 1.

Thanks
doubleH
Regular Advisor

Re: HP3500 losing configuration and poor ping response

i did more testing and posted the results to my original thread...

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1036281
doubleH
Regular Advisor

Re: HP3500 losing configuration and poor ping response

i've had this problem with my 5406's and 3500 for a few months now. today we are working on the theory the there is an incompatibility with these switches and PCM+. go ahead and either shutdown the device that is running PCM+ or if you can't just stop the 3 procurve services and you may see your issue go away.
BOVC
Advisor

Re: HP3500 losing configuration and poor ping response

Hi,

Matth Hobbs posted a link to a post that I initially started about performance issues on the 5400zl.

We actually solved the problem, but not the best way :)

It turned out, that when we set up a computer with PCM+ (newest version), after a couple of hours the main switch (which is doing all the routing) exprienced exceptionally poor performance (but only on routed traffic). We pulled the plug on the management server, and restarted the switch - and gone were the problems.

Some days later (and Update 6, instead of Update 5 for PCM) we tried again - with the exact same result!

Again we turned of PCM+, and everything has been working fine ever since.

I called HP about this, but since we were having no problems anymore, I don't think the tech guy took it too seriously. Anyway, I told him that because of this, we would not buy PCM+ (which we had thought of before this).

Rasmus
doubleH
Regular Advisor
Solution

Re: HP3500 losing configuration and poor ping response

my issue has been escalated to the level 3 engineers for a few weeks now. yesterday they were able to reproduce the issue in their own lab like a few of us are experiencing. they are looking at this as a bug in PCM and should be fixed.

in the mean time i have PCM running, but just remove traffic monitoring from all your 5400/3500 and the issue goes away. they mentioned something with sFlow is the issue. at least this way you can still run PCM, just not the traffic monitoring.
Drew_38
Frequent Advisor

Re: HP3500 losing configuration and poor ping response

Thanks all for your recent responses. Can this really be right that it is PCM triggering the problem? How incredible. I really am losing faith in HP and their ability to deliver reliable network equipment. Moreover the (European) engineers who I have been dealing with when trying to report this problem are so way off the mark it's laughable. Last communication we had from them they were trying to blame 3Com switches elsewhere on our LAN (when we don't have any).

Anyhow I'll try and turn off PCM traffic monitoring and see if the problem goes away.

Obviously it isn't actually a problem with PCM, it's more how the 35xx and 54xx series respond to data collection requests from PCM. I love PCM it's great so I'd not be put off buying that.

Drew
doubleH
Regular Advisor

Re: HP3500 losing configuration and poor ping response

i know it sounds weird, but i've been banging my head against the wall for 2 months now. this actually makes sense for what i was seeing and i can reproduce the problem on my end no problem.

if stopping the traffic monitoring deosn't work just shutdown the pcm machine or just stop the 3 procurve services.
Drew_38
Frequent Advisor

Re: HP3500 losing configuration and poor ping response

doubleH just to be clear you are saying that after you stopped the PCM services you had no performance issues with your 35xx and 54xx switches?
doubleH
Regular Advisor

Re: HP3500 losing configuration and poor ping response

correct. it may take a few minutes to kick in. you also may need to reboot your switches, but once they are back up and PCM is not collecting traffice data you should be good to go.
doubleH
Regular Advisor

Re: HP3500 losing configuration and poor ping response

just curious. did this fix your issue?
Drew_38
Frequent Advisor

Re: HP3500 losing configuration and poor ping response

Hi DoubleH; not yet got conclusive information on this yet. We use another traffic management package, InMon, which also could be causing this problem. I am hopeful that the new 35xx software release might have sorted out the problem rather than us having to disable such things as Inmon and PCM+. We've not yet been able to put our 35xx back in service to test it out. I'll post a note when I've got some firm info don't worry!!
Drew_38
Frequent Advisor

Re: HP3500 losing configuration and poor ping response

Hi all;

Just to report good news along the lines suggested by BOVC and doubleH.

I didn't need to go the extent of turning off HP PCM or Inmon as suggested. Having thought about the comments above it seemed unlikely to me it was sflow connectivity but would more likely be snmp which is causing the problem. This was backed up by the fact that in the new 3500 software, K11.41, you can turn sflow off and this didn't improve things. So all I did in the end was change the SNMP access levels to 'Restricted' so that our PCM and Inmon servers (or indeed any other servers which I might not know about) are not able to communicate with the 3500. After about 30 mins or so the ping response returned to normal (consistently 1ms or less) and everyone is happy.

So my conclusion is the 3500 and 5400 series suffer problems when a network management system is trying to interrogate it via snmp. That, of course, is not the fault of HP PCM or similar packages but a problem with the switches. So finally I can tell the HP support people, who have been unable to even grasp the concept of the problems we are facing and keep asking the same questions over and over again, where the problem lies and hope they can somehow grasp the fact that their hardware or software is at fault. The situation is not satisfactory as I do want PCM to manage the 3500 so I will still be pressing for a solution.

Obviously, by the way, new software K11.41 didn't make any difference. Thanks HP for making us wait for a software upgrade which didn't fix anything.

I have to express my thanks to everyone in the forum who has offered advice and help, really you've been beyoned excellent.

Cheers,

Drew