ProCurve 2626PWR stops responding and needs reboot! Urgent!

[Rodrigo Ordonez]

We have approx. 60 2626PWR switches installed throughout different areas of a campus wide network. Each area has 3-4 2626PWR arranged on a loop using RSTP and then all areas are arranged on a star configuration around an HP5308xl.

The system works great for the most part, but there is one area that has had problems all along. We have replaced 3 of 4 switches already and the problem persists.

What happens is suddenly (it seems out of nowhere) at least one of the switches will stop responding completely (no network traffic through any ports, no SNMP communication, no Telnet access, no WEB access) and the problem can only be solved by rebooting the switch. After powering back up, the switch responds like if nothing ever happened and of course all logging is cleared so there is very little troubleshooting information to analyze.

We've gone as far as replacing the UPS in that rack thinking this could be power related but the problem happened again today.

Any ideas on what can cause these switches to fail in this way? Can it be power? Can it be a network storm? Can it be a faulty device on the network?

Any suggestions will be greatly appreciated!

Thanks,
Rodrigo

[Olaf Borowski]

Rodrigo,

To get more information, configure a syslog server so you don't loose your log. You might not be able to capture exactly what is going on with those switches, but it should give you and indication on what happened before communication stopped. Also, can you still access the RS232 console? Any messages there?

Olaf

[Mohieddin Kharnoub]

Hi

Any switch has a CPU, and if the utilization of this CPU goes very high, then it won't forward any traffic or respond to any communication (i think even the console).

I think its a process that kills this group of the 2600 switches, and i suggest you to enable the logging to a Syslog server, and disconnect the uplink to the core when you get this problem and try to investigate in the logging.

Also, many features now available with the new firmware for these switches H_08_106, and one of them is the BPDU-Filtering.

Try to upgrade the firmware:
http://www.hp.com/rnd/software/j49008106.htm

Good Luck !!!

[Matt Hobbs]

Also, instead of pulling the power from the switch - are you able to just click the reset on the front? This way it shouldn't lose log/crash information.

[Rodrigo Ordonez]

Thank you all for your suggestions. After checking the show tech reports in detail, I noticed that RSTP on one of the switches was not set up correctly and we most likely had a network storm when some connections were disturbed.

It is interesting that the network storm was so bad that it completely killed 3 of the switches (the reset didn't work). There was no network traffic at all on any of these switches and the only way to communicate to them again was to power cycle them.

The suggestion to install a Syslog server is very much appreciated and we will do so immediately. Question: does this only log events logged on the switches Event Log? Is there any way to keep track of port error counters over switch power failures?

Thanks!
Rodrigo