BladeSystem - General
cancel
Showing results for 
Search instead for 
Did you mean: 

C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

SOLVED
Go to solution
drolfe
Valued Contributor

C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Hi All

 

I'm currently having with ilo disconnects, it seems to happen at random on all g6 servers over two different chassis now
 
I've gone from Static in EPIBA, now dhcp, next I guess is to try static ilo assignment ?
 
As this is more than one C7000 chassis and multiple servers I don't this it's hareware related.
 
I'm thinking this is a firmware issue
 
Any help you could give as this is causing me major headaches.
 
I have some Gen8 (iLO4) servers which have never had any problems with their ilo
 
see below
 
iLO issues.jpg
 
 
20 REPLIES
scharchouf
Trusted Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

send me in private message an ShowAll

I work for HP
A quick resolution to technical issues for your HP Enterprise products is just a click away HP Support Center Knowledge-base
See Self Help Post for more details

drolfe
Valued Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

pm sent
drolfe
Valued Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

It's also worth a mention, that all the blades with the ilo errors still work server side.

 

Blades can be rebooted and startup without error.

 

The only way that I've found to recover the ilo as the web UI is down and I can't ping the ilo IP is to 

 

ssh to the OA and run the command "reset server x" (where x is the blade number 1 - 16) to reset the blades "E-FUSE"

 

Regards, Daniel

 

 

scharchouf
Trusted Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Check the Excel File :

 

You need to Update :

BIOS iLO2

 

you can use the new SPP 2014.02.0

 

I work for HP
A quick resolution to technical issues for your HP Enterprise products is just a click away HP Support Center Knowledge-base
See Self Help Post for more details

drolfe
Valued Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

HI, 

 

Thanks for that, 

 

I'll get everything updated firmware wise.

 

I've got the second chassis enc2 running OA 4.11 and all the g6 blades are running SPP 2014.02.0.

 

I'll monitor to see if the second chassis has any more ilo issues

 

Regards, Daniel

drolfe
Valued Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Hi,

 

I have Chassis 2 fully upto date firmware wise and after a week, 2 blades have lost ilo

 

Please see attached firmware snapshot

 

Regards, Daniel

scharchouf
Trusted Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Bay 1: ProLiant BL465c Gen8 Server Blade Rom Version : A26 06/09/2013 iLO Version :1.30 Jul 18 2013
Bay 2: ProLiant BL465c Gen8 Server Blade Rom Version : A26 06/09/2013 iLO Version :1.30 Jul 18 2013

 

HP VC Flex-10 Enet Module not up to date

 

I send you also a PM because I found that some server's haven't assigned profile

 

 

I work for HP
A quick resolution to technical issues for your HP Enterprise products is just a click away HP Support Center Knowledge-base
See Self Help Post for more details

drolfe
Valued Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

HI,

 

The servers without profiles are new, currently waiting on DAC storage.

 

Also I can upgrade the VC firmware but these two flex10 switches are apart of the overal domain, so doing the VC update on the domain will cause a service interuption to clients, so this can be done but wasn't as easy.

 

I'll get the Gen 8 servers updated, I didn't do these as they aren't having any issues with their iLO4 but if it could have an impact on the overall chassis I'll get it done.

 

So you think the VC version would be causing this issue ?

 

Thanks Daniel

 

TK8
Member

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Hey guys,

I understand this was an old issue but any insight into this will be very helpful

we just got hit with the same problem in our infrastructure. Random WS460 and xw460 G6 blades lost connectivity to iLO.

 

we had to reseat the blade to get it back however just so it doesnt happen again, it will great to know why it may have happened especially that randomly on random blades

 

Thanks

drolfe
Valued Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

HI,

 

Bay Manufacturer Part Number Spare Part Number Firmware Version
1 HP 456204-B21 503826-001 4.21 Apr 12 2014
2 HP 456204-B21 503826-001 4.21 Apr 12 2014


Bay Device Model Firmware Version
1 HP VC Flex-10 Enet Module 4.20
2 HP VC Flex-10 Enet Module 4.20
3 3Gb SAS Switch 2.2.17.0
4 3Gb SAS Switch 2.2.17.0

 

 

I have one chassis fully updated to the latest SPP or newer, OA 4.21, VC 4.20 iLo 2.25, I"m still seeing the issue: please see below from the OA logs, not I've not been logged into the OA at all dince these events have been happening:

 

May 4 03:16:32 OA: Blade 8 found a partner device.
May 4 03:16:32 OA: Blade in bay 8 has been powered on

 

May 4 12:34:04 OA: Management Processor on Blade 9 appears unresponsive.
May 4 12:35:04 OA: Blade 9 has been allocated a default power value of 509W because iLO appears unresponsive.

 

May 7 12:41:40 OA: Management Processor on Blade 4 appears unresponsive.
May 7 12:43:00 OA: Blade 4 has been allocated a default power value of 509W because iLO appears unresponsive.

 

May 8 08:27:55 OA: Blade 4 found a partner device.

May 8 08:27:55 OA: Blade in bay 4 has been powered on

 

May 9 04:25:19 OA: Management Processor on Blade 7 appears unresponsive.
May 9 04:25:34 OA: Blade 7 has been allocated a default power value of 509W because iLO appears unresponsive.

 

I have no idea what could be causing these issues, Ilo network is on an isolated vlan from the rest of my network.

 

Please help

 

Regards, Daniel

drolfe
Valued Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Thought I would reply to say this is still going on.

 

Please see below what I've extracted from the OA system log
 
Please see below what I've extracted from the OA system log

Jul 15 20:29:22  OA: Management Processor on Blade 3 appears unresponsive.
Jul 15 20:29:42  OA: Blade 3 has been allocated a default power value of 509W because iLO appears unresponsive.

Jul 18 22:50:11  OA: Management Processor on Blade 9 appears unresponsive.

Jul 26 05:56:24  OA: Management Processor on Blade 3 appears unresponsive.

Jul 27 05:27:42  OA: Management Processor on Blade 5 appears unresponsive.
Jul 27 05:28:02  OA: Blade 5 has been allocated a default power value of 509W because iLO appears unresponsive.

Jul 27 06:34:37  OA: Management Processor on Blade 6 appears unresponsive.

Jul 28 08:12:48  OA: Management Processor on Blade 9 appears unresponsive.
Jul 28 08:14:28  OA: Blade 9 has been allocated a default power value of 509W because iLO appears unresponsive.

Jul 28 12:13:47  OA: Management Processor on Blade 8 appears unresponsive.

Aug 12 07:03:48  OA: Management Processor on Blade 9 appears unresponsive.
Aug 12 07:05:28  OA: Blade 9 has been allocated a default power value of 509W because iLO appears unresponsive.

 again, the only way to recover these devices is with an E-Fuse reset

 

HP replaced one of the G6 server boards and I haven't seen the issue again

 

ilo was recommended to be updated to 2.20 but the Server Board bios is still I24 05/05/2011

 
Do we think these ilo2 errors were caused but faulty firmware toasting the motherboards ?
 
Looking at this post, he seems to have the same issues as me:
 
 
 
All my current g6 servers where working without issues for years, once I shipped them into the New DC I upgraded all the blades to 2013-2014 firmware and these issues started to pop up. ??
 

 

Oscar A. Perez
Honored Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

When the iLO2 is showing the error, can you communicate to iLO2 from the OS?
Example, sending XML scripts to iLO2 via hponcfg from the OS?



__________________________________________________
If you feel this was helpful please click the KUDOS! thumb below!
drolfe
Valued Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Hi,

No it's totally dead from the OS also, I can't communicate via hponcfg

 

I also can't access the ilo web interface, or ping the ILO ip address from either the network or from the OA itself

Regards Daniel

Oscar A. Perez
Honored Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

I'm assuming these iLO2s were working fine in the past. Do you know what changed in your environment that could have contributed for this issue to surface?

I sent you a PM.



__________________________________________________
If you feel this was helpful please click the KUDOS! thumb below!
drolfe
Valued Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Hi,

 

Yes I replied to your PM :-)

 

If you have some BETA firmware for me to test that would be great!!

 

Regards, Daniel

drolfe
Valued Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Hi,

 

I'm currently testing Beta iLo2 2.26 firmware with the "Watchdog" proccess runnning

 

This "Watchdog" is to check on the hour if iLO2 is ok and if not locally reset the iLo2 ROM locally.

 

This saves me from having to E-Fuse reset blade servers to regain access to iLo.

 

So far so good, it's been around 5 days and I haven't had anymore blades stuck in iLo error but I'll need atleast 2-3 weeks to know for sure.

 

On another note, I can see this "Watchdog" proccess is enabled by defauilt on iLo3 (Bl460c G7) 

 

iLO3_auto_watchdog_restart2.jpg

 

So I guess the question is. Why isn't the watchdog enabled by default on iLo2 ?

 

Regards, Daniel

drolfe
Valued Contributor
Solution

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Hi,

 

This issue has been resolved

 

I'd like to thank Oscar A. Perez for taking the time to provide me with some BETA iLO2 2.26 firmware with the watchdog enabled.

 

Please note, this is BETA firmware and is likely NOT supported by HP so please use at your own risk

 

iLo 2 Firmware 2.26 watchdog enabled BETA

 

Regards, Daniel

Server-Support
Super Advisor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Dan,

 

Did you finally update the VC firmware or not needed for your case?

drolfe
Valued Contributor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Hi,

 

No only the ilo firmware was needed.

 

Regards, Daniel

Server-Support
Super Advisor

Re: C7000 OA4.01 BL460c G6 iLO 2 version 2.22 dropouts

Thanks for the reply back.
Would you be able to share what steps did you took to upgrade the VC firmware from 4.01 to your current latest 4.11 ?

I've never done this before so I'm curious to know how without causing server downtime.