5406zl PS fault light

[synaesthesia]

2 of these units roughly a mile apart.

Both of them at exactly the same time have an amber flashing alert light (POE, power supply 0). Nothing in the logs.

Disabled POE on both switches, rebooted one of them, nothing; no problems.

Any hints? Two faults at exactly the same minute and huge coincidence?

[Helper]

Hi,

Do you have any information about this event in ?

show log -r -a

show system power-supply

show system power-consumption

You shouldn't mix different type of power supply in the chassis, it will work but the fault led will be on.

Relevant to this situation if you are using PoE features you should configure the redundancy mode of the power supply as N+1 and not full : power-over-ethernet redundancy n+1

In the maintenance guide :

The power LED has a simple function: it is on if the Power Supply Shelf is on, and off
if the Power Supply Shelf is off.
During normal operation, the fault LED is off. If it is on, an error has occurred with
one of the EPS ports. Check the EPS port status LED to determine which one is
experiencing the problem.
The power status LED is green if the corresponding power supply is installed and
plugged in. This LED is off if no power supply is installed. If the power supply is
experiencing a problem or if it is not plugged in, the power status LED flashes
orange.
Two LEDs report the status of each EPS port:
 Device connected LED—This LED is on if there is a valid connection to an E5400
zl Switch. It is off if there is no connection.
 Power status LED—This LED is on if the power supply is providing power to a
connected E5400 zl switch. It is off if there is not a valid connection to the
E5400 zl or E8212 zl switch. If the LED is flashing orange, there is a fault
condition on the corresponding power supply.

Bye.

[synaesthesia]

The systems are as they are off the shelf, no additional supplies or anything. Both did it at the exact same time. Case raised with HP 3 days ago, no reply yet...

POE disabled across the board.

DR-Core# show system power-supply
Power Supply Status:
 PS# |   State     |   AC/DC  + V    | Wattage
 ----+-------------+-----------------+---------
   1 | Perm Failed |  AC 220V        | 1500
   2 | Not Present |  -- ----        |    0

   0 /  2 supply bays delivering power.

DR-Core# show system power-consumption
 Slot Power Usage:
 Slot  Module Description                        CurrentPower
 ----- ----------------------------------------- ---------------
 A     HP J9534A 24p Gig-T PoE+ v2 zl Module     22 W
 B     HP J9535A 20p GT PoE+/4P SFP v2 zl Mod    21 W

Obviously won't paste the full log (that was sent to HP anyway) due to size but find it odd that both showed failed at the same moment?

The switches are both otherwise working as intended.

[synaesthesia]

If it makes any difference:

I 02/04/12 23:48:09 00071 chassis: Power Supply failure:  Supply: 1, Failures: 1
W 02/04/12 23:48:09 00576 chassis: 50V Power Supply 1 is Faulted. Failures: 1

on one switch, and the other connected ONLY by fibre, in an entirely different building circa a mile away:

I 02/04/12 23:47:04 00071 chassis: Power Supply failure:  Supply: 1, Failures: 1
W 02/04/12 23:47:03 00576 chassis: 50V Power Supply 1 is Faulted. Failures: 1

No power cuts or brownouts. Well into the night, so no human intervention. It is almost certainly software related in my eyes. Strange one?

[synaesthesia]

Don't suppose there's any ideas from anyone? Tried a hard reboot today (power off, wait, on) as it was the first appropriate time.

HP so far have had a week with my case and all I've had back is a confirmation that a case has been raised. Do they all go on communal holidays or something?

[Richard Litchfield]

The ProCurve switches have a lifetime warranty - why not order replacement PSUs? If each chassis has dual PSUs, you can hot-swap the faulty PSU. With the replacement unit you can do that anyway: put the new one in PS bay 2, remove the PSU in bay 1. You could try reinserting the faulty PSU and see if it still has the fault light.

[synaesthesia]

Well I would, if HP got off their arses and actually looked at the case that was raised at the same time as the first post in this thread. Not amused by their response times, and I'd rather not have to get someone to call them (I can't do it myself); hate going through middle men to satisfy some silly preference for a telephone.

[Richard Litchfield]

The warranty coverage is next business day, advance replacement - once confirmation of the problem/part required is amde, you will receive the replacement the next business day. If you haven't received a phone call or email by now, it definitely needs chasing.

In Asia Pacific, you can follow it up by email to procurve.apjsupport@hp.com with your log number in the subject heading.

[synaesthesia]

Replacements are here... sort of.

One of the power supplies is in fact the bottom paper tray for a printer.

The other - well how do we do a hotswap with no downtime when they don't provide the mains cables?

I think HP need to do some work on their returns procedures :(

(Happy about the speed of things once they've gotten over the 2 week lead time though)

[nhand42]

If only HP honoured the "next day replacement" policy. We're up to two weeks now on the same failure.

Five 5406 switches spread across campus. All ten PSU failed at the same time. They're on different power feeds, different power circuits, and the switches are connected only by fibre.

HP is refusing to replace the PSU. HP claim it must have been a power surge and wants us to pay for brand new supplies, rather than replacing them under warranty.

The symptom is that the PSU continues to supply power to the switch, but it reports itself as failed, and the PoE module reports 0W. Doesn't sound like a power surge to me. Sounds like software.

# sh system power

Power Supply Status:
 PS# |   State     |   AC/DC  + V    | Wattage
 ----+-------------+-----------------+---------
   1 | Perm Failed |  AC 220V         | 1500
   2 | Perm Failed |  AC 220V         | 1500

# sh power-over-ethernet

Internal Power
        1   0W        /Connected - Faulted.
        2   0W        /Connected - Faulted.

[nscapj]

Hi there. My understanding from the whole story is that 10 of them went bust at the same time. Is that correct? And you are running K.14.63.  I could understand one or two going faulty but 10 at the one time? This probability of that is really very small given that everything has been fine for 1-2 years. Could it be a power surge instead that caused the problem?  I am not certain that it is software related. I have never seen PoE power not being delivered being a Software Issue after it has done so for a long time and there has been no code upgrade.  HP will definitely replace all the power supplies if they are deemed faulty but not if there has been an event such as a power surge that has caused the damage. The best thing would be for you to get 1-2 of these power supplies replaced and HP look at the faulty ones to determine root cause.

[synaesthesia]

Wow, that's interesting. Perhaps you should point them directly at this thread then and they can confirm from my case it's not standalone. I'm also thinking software (May have mentioned already).

The replacement power supply that was actually a power supply is in one unit already and is working fine. The other has not alleviated itself magically as a result; I wonder if it's something along the lines of STP (although disabled on ours) or maybe more likely LLDP.

[synaesthesia]

nscapj: It's surprising how many people don't seem to understand the "connected only by fibre" part.

For instance ours are on UPSs which have not reported any unusual electrical activity.

Even if there was, they're nearly a mile apart. If there was any unusual electrical activity that would fry a power supply (or part of it) then the rest of the town just might have noticed problems with their electrical appliances. If not, then there is absolutely something wrong with the PSU's in these units.

The above post states that his campus's units were also connected only by fibre, with the power distribution on different systems entirely. Nothing out of the ordinary for large sites...

For note these are the J9306a POE+ power supply units.

[UKCoolHandLuke]

We too have this same issue although our switches are located in the same cab (x2).  They run fine for a couple of weeks and then out of no where they just report a power supply fault and stop supplying PoE devices.  running latest firmware K.15.06.0008

About to install a VoIP setup in the next couple of weeks too so this needs to be sorted :(

W 03/11/12 00:18:57 00576 chassis: 50V Power Supply 1 is Faulted. Failures: 1

Both fail at the same time (it has happened twice now), after the first time I disabled all PoE except on 4 ports.

[synaesthesia]

Spoiler

If you can, remove both PSU's and let them power down for a little while. Then replug in and see if that clears it. I'm still fairly certain there's a software bug involved.

[intel_killer]

For what it's worth we have also experienced this issue on a few of our switches.  Running K.15.04.0003 when the failures happened.  Thought it was related to power issue but our UPS connected to the same circuit showed no spikes or drops on the line in voltage.  Disconnected power to the "failed" PSU for about five minutes and the error cleared.

[John Gelten]

For what it is worth (I can't remember the details) : we have had some power-related issues with ProCurve E2610 switches a few years ago, and they turned out to be (confirmed by HP) issues with firmware in the PoE-controller of those switches. There has been a specific software-release (available in request only) for the E2610-PoE series that actually erased this controller firmware, and put a new version in it.

I know... E2610 is a very different series from the E5400, but it is the general idea: PoE failures could definitely be caused by firmware-issues. And it is quite imaginable that 'something in your network' triggered this very exotic bug simultaneous at all switches...

[IWM-KMRC]

We've had this issue now on several of our 5406zl, after upgrading to FW release 15.12.0010.

I now upgraded to 15.12.0012 and will report back, if the problem still occurs.

[Simon Clout]

I'd like to add a 'me too' message.

We have around 25 54xxzl's on campus and I've had this on only one switch and it's always the same one! The switch is a 5412zl and has four power supplies installed. I have already replaced two out of the four supplies on the chassis as they too have the amber light showing (and then later failed completely. I've disconnected the offending supply and then reconnected and this clears the amber light. A check a few days later sees that the amber light is back again.