5406zl Error - 00274 chassis: (84) Slot C: Blade Crash detected

[Tony_B]

HP 5406zl switch running software release K.11.63
The following modules installed
module 1 type J8702A
module 2 type J8702A
module 3 type J8702A
module 4 type J8702A
module 6 type J8705A
The module in slot C crashes periodically with the following error
00274 chassis: (84) Slot C: Blade Crash detected

Has anyone else experienced a similar problem with this software and hardware combination?

[Joel Belizario]

Sounds like you have a faulty switch module - your best bet is to log a case with HP support to get it replaced ASAP.

Good luck!

Joel

[Tony_B]

Update on configuration changes made to switch.

I have upgraded the software to K.12.01
After the software update the blade in slot C continued to crash.

I moved all the connections from the blade in slot C to the blade in slot D. As a result the blade in slot D started crashing, with a similar error to the errors previously experienced with blade C.

I moved half the connections from the blade in slot D back to the blade in slot C. None of the blades have crashed since I split the loading across two blades.

Has anyone else experienced blade crashes when a switch blade is fully loaded?

[Joel Belizario]

Now that is rather strange behaviour - I've never seen anything like that personally, I've found the 5400zl series to be pretty robust and stable.

Just out of curiousity, how long has the switch been in use for? Also, it is possible to receive a faulty module now and then from the factory, though having said that it is quite rare in my experience.

[Matt Hobbs]

Are you running any sFlow sampling like PCM Traffic Monitor? If so, try disabling any sampling. It has caused a few weird problems in the past.

[Tony_B]

Joel

The switch has been in service for approximately 6 weeks. The number of user connections has only increased slightly during the time that this switch has been in service.

Having taken half the connections off of the blade in slot C and moved them on the blade in slot D, thus splitting the load across two blades, the issue seems to have gone away. This is however only a temporary workaround.

I still need to get find a permanent solution to the blade crash problem.

Do you have any other suggestions?

[Tony_B]

Matt,

Yes we are running sFlow on this switch as we do run the traffic monitor in PCM+.

I understood from the release notes that came with version K.11.63 of the 5400zl switch software. The problem with sFlow and high utilization on these switches had been resolved. I currently have four other 5406zl switchesâ running K.12.01 and this issue has not occurred on any of those.

Any ideas?

[Matt Hobbs]

Just for troubleshooting I would be disabling the traffic monitor on PCM+ for that particular switch and putting all the connections back onto the single module to see if it re-occurs. If it doesn't reoccur for a few days, enable traffic monitor again and if the problem comes back, you've got your trigger.

Either way, it would be a good idea to open a case with HP at this point in time since you are already running the latest firmware and are seeing a problem.

The sFlow issue that was fixed in K.11.63 was more a performance issue where sampling was causing poor throughput performance.

[Joel Belizario]

From the troubleshooting you've done it sounds like possibly a faulty backplane, especially since you're not experiencing this with any other switch.

Sorry to sound like a broken record but your best bet would probably be to log a case with HP support and get some replacement hardware shipped out to you.

Even if you were to confirm the trigger as Matt suggested, I doubt it would possible to isolate any further without having a spare chassis on hand to test every suspect component.

Good luck!

[Tony_B]

Joel

From the testing I have carried out I would agree with you a faulty backplane looks to be a possibility.

Fortunately I have a 5406zl chassis in stock. I have upgraded the software to K.12.01 then swapped all the blades and power supplied over to the new chassis. The new chassis is running in the same hardware configuration as the old one. With the connections still split across the two blades. It is also running the same software configuration.

I am monitoring the switch to check for any issues. If there are no problems, after 48 hours I will move all the connections back to the blade in slot C.

I have started a case with HP support and get some replacement hardware.

Just as a foot note PCM has picked up that K.12.02 is now the latest software release for the 5406zl chassis. I cannot find this version of the software on the HP web site. I wished to read the release notes for this version of the software.

[OLARU Dan]

If the problem reappears after you fillup blade C, then it's the blade not the backplane.

[OLARU Dan]

Tony,
thanks for the point, mate!
I'm running against Matt here, and any point is welcome ;-)
Cheers

[Tony_B]

OLARU Dan

If as you suggest the problem is with the blade in slot C not the backplane.
How do you explain the following? When all the connections are moved from the blade in slot C to the blade in slot D, the blade in slot D started to crash with the same error messages.

[Tony_B]

48 hours ago the network connections in the blade in slot D were all moved back to blade in slot C. The blade in slot C now has 23 of the 24 ports in use; this is the configuration the blade was in when the blade crashes started to occur.

The blade in slot C has not crashed in the past 48 hours. I am still monitoring the switch to check for any issues. The fault appears to have been with the chassis and not the blade or the software.

[Tony_B]

24 hours after the connections were all moved back on to the blade in slot C the blade crashed again with the same error as before.

Matt

I have done as you suggested now disabled the sFlow on the switch and removed the switch from the PCM+ traffic monitor. I am once again keeping a close eye on this switch.

Any other suggestions?

[Don S]

Regarding the problem with PCM+ and sFlow crashing modules and creating general havoc.. We've been fighting this issue for months, and per HP support, this issue is still not fixed in K.12.01 either. I would in no way continue to run PCM+ on your network.

[Ralph Bean_2]

Would those of you who have a Procurve Case ID (Trouble ticket number) for this crash be so kind as to post it in this forum?

Thanks.

[Tony_B]

Ralph,

Here are my case details.

Reference number 1214695736

[Ralph Bean_2]

Tony, I don't think that trouble call is still open. Are you continuing to work with ProCurve Support regarding this?

[Tony_B]

HP have opened a new case with the following ID 1214859945

[Karl Lasse]

I have had about 5 of these HP5412zls crash like this. The module seems to reboot itself and we loose about 25 seconds. We are getting these crashes on the 8702A and the 8705 modules. I am running K.11.62 code.
Support is trying to figure out if this is software or hardware now.
I am getting the following error:
W 03/30/07 07:42:06 00274 chassis: (7C) Slot L: Blade Crash detected - Available

I 03/30/07 07:42:12 00375 chassis: Slot L Downloading
I 03/30/07 07:42:13 00376 chassis: Slot L Download Complete
W 03/30/07 07:42:22 00274 chassis: Slot L ACL Int status=0x2000000 25=0x80000005
: Task=tDevPollRx Task ID=0x4305d314 IP=0x40087044

Is anyone else getting the error in the Task=tDevPollRx task?
-Karl

[stieven struyf]

We had the same problem on both 5412zl's.

We had it on a slot which was connected to our firewalls. 12 ports and each a different vlan.
although the slot shouldn't be overloaded(max firewall throughput on all interfaces toghether was 300Mb) we decide to spread the interface accross multiple slots.
We also updated to the latest firmware and the slots have been replaced without resolution.
we also noticed an incompat. between our old avaya's and the new procurves(lost packets, high ping response times) we had experts look at it but never found the reason.
(response time suddenly increased after running for a few hours until at some point the modules crashed which had the firewall on it, rebooting the procurve solved the problem for a few hours until the same pattern repeated itself)

Since we removed our old avaya's and moved the firewall links to multiple slots the problem went away(now very stable for about 7 months)

our overall cpu load on the switch was never higher than 12%.

maybe you also have some kind of incompat.
do you notice a pattern like i did before it crashes. any new devices added to the network before it happened?