C7000 Degrade alerts

[City_Blue]

we have been geting alert e-mails to say we have lost redundancy on our c7000, but straight after the e-mail we get another to say it is ok.

we have managed to be loged in to the OA at the time of the alerts, however we can only see a warning triangle and a message that say's degraded. we can not see or find what is failing.

we have recently changed all the Power supplies as per the HP advisory, but we are still geting the alerts.

has any one else had somthing simaler?

cheers

[GORDITO · Accepted solution]

Yes, We've had the same problem last week after HP replaced the power supplies in one of our C7000 chassis. You need to check the power sub system.

HP had to replace the OA sleeve and reseat all of the power supplies in our enclosure.

[City_Blue]

thanks for this, will chase HP over the issue

[Jimmy Vance]

Check this Customer Advisory


http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01519680

and also

HP PCC c7000 Power Supply Replacement Program

https://h30090.www3.hp.com/pcc%2DCPS/

[City_Blue]

thanks JIm

but i said in my opening post we have replaced all the units under this programe

[Jimmy Vance]

Sorry, just quickly glanced at your post.

Does the OA syslog say anything?

[City_Blue]

no the log's only show genral activity, we can see some power errors, but we know these are caused by us as they are at the times we tested them, re seating and so on.

very strange

[Neal Bowman]

Do any of your blade servers show excessive power consumption? I had similar situation a few weeks ago, all 6 PS showing they were distributing power, rather than 2 engaged and 4 in Dynamic Power Saving mode. I had 1 BL460c reporting to OA it was using over 43000W of power, triggering the power event.

I had to perform some BIOS/ILO upgrades tp the blade server itself in order for the chassis to revert to a normal state.

Neal

[Scarf And Tea]

we have this problem too. I verified the power supplies and though some of our enclosures have bad PS units, some are OK and have that same mass mailling problem. At this point, I think segregating the OA and ilos from the production network as per http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1243522322162+28353475&threadId=1232146 might be our best shot at resolving the issue. According to HP, the alert mails are caused by loss of communication between the OAs, and these losses could be caused bu the production traffic.

Just my 2 cents...
let me know if it helps.

Scarf

[Scarf And Tea]

Also, we were told by HP that this might have been caused by our power redundancy mode set to N+N so we have changed the settings of one of our enclosures to N+1 and still get spammed...

The behaviour is as follows :

We receive an email stating that the enclosure is degraded.

Thu 5/28/2009 10:41 AM
HP AlertMail-214: (WARNING MINOR) Enclosure Status: Degraded

one minute later we receive an email stating that the enclosure status is now back to OK.

Thu 5/28/2009 10:42 AM
HP AlertMail-215: (NORMAL) Enclosure Status: OK

Then, later we receive another alert stating that the enclosure is degraded and so on ...

Thu 5/28/2009 10:46 AM
HP AlertMail-216: (WARNING MINOR) Enclosure Status: Degraded

Please note that if we reboot the Onboard Administrators, the issue disapears for about 35 days after which it reappears a couple of time per day at first and worsens up to the point where we receive 2 emails every 5 minutes...

[Fredrik Oqvist]

I had the exact same problem.

Try to do this:

Upgrade all blade servers ROM version to 07/10/2009, iLO firmware 1.79 and Onboard Administrator to 2.60.

It worked for me!


Regards

Fredrik

[City_Blue]

we are going to upgrade the OA's and ILO's plus ROM BIOS again next week to see if this fixes the issue it has now spread to all our enclosures.