Integrated Lights-out 3 interface status not responding

[sajad mohammed]

Hi,

Have noticed that some of our DL360 G7's report in Systems Management that the Integrated Lights-out 3 interface status is not responding.

iLO works fine though. iLO firmware version is 1.05. Have stopped/restarted all the HP related services and this temporarily clears the message.

Has anyone come across this before ?

[Oscar A. Perez]

This is a known issue. You need to upgrade iLO3 to version 1.15.

[sajad mohammed]

Thanks for your reply. Are there any known bugs/issues with using that firmware version.

[Scott Bonci]

I'm having the same issue. HP has replaced the motherboard and ilo firmware is at 1.15. It only appears at boot but not always.

[Oscar A. Perez]

If this is the random iLO3 unavailable issue during POST but, iLO3 is alive and good then, this issue is fixed in the next iLO3 firmware release 1.16. This release should hit the web early January.

[David Trounce]

How do you reset the ILO when it is unresponsive? My new DL980 G7 shipped with ILO 1.10, Smart Update Manager didn't tell me about 1.15 :-(

I now have the 1.15 flash program, but can't flash since the ILO is unresponsive. hponcfg can't reset it. I don't see how to reset it in SMH either. How do you reset ILO when it is unresponsive?

[David Trounce]

We pulled the power on the server, and ILO 1.10 was responsive after restoring power. We then upgraded firmware to 1.15, which fixed the hponcfg responsiveness issue.

But there is a bug in 1.15, which then prevents the server from booting again. See http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1293668988596+28353475&threadId=1456726

So we downgraded to 1.10, and the server boots again.

But we still only have 4 out of 8 sockets working, per the ILO issue reported at http://bizsupport1.austin.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=4154735&prodTypeId=18964&objectID=c02624820

Pulling power does not fix this, contrary to the recommended workaround. ILO 1.10 and 1.15 don't fix it. So we have a DL980 that works like a DL580 G7 right now. HP first level support has not been trained on DL980, so everything goes to 2nd level support. But no call-back in 4 days. This is despite paying for 6-hour CTR support.

Not great experience so far.

[David Trounce]

According to HP, ILO3 firmware 1.15 is not supported on DL980 G7. Only 1.10 for now. That's because it won't boot with 1.15.

http://www11.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c02655638

But the workaround for the processor issue on DL980 G7 requires ILO firmware 1.15.

http://www11.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c02641201

Catch-22?

[Oscar A. Perez]

If you can, please try the latest 1.16 Beta.


FTP System: ftp.usa.hp.com (15.192.32.78)
Login: tempilo3
Password: G1vmeilo (NOTE: CASE-sensitive)
Last Modified: 10/28/2010 1:22:49 PM
Expire Notice: scheduled for 11/3/2010 12:00:00 AM
Valid Until: 11/4/2010 1:22:49 PM (7 days)
Access Type: read-only

FTP Access: ftp://tempilo3:G1vmeilo@ftp.usa.hp.com/
or: ftp://tempilo3:G1vmeilo@15.192.32.78/

[David Trounce]

I tried the ILO 1.16 beta 7 from the download link. But I get the same results as with 1.15
* Boot sequence still shows processors 5-8 deconfigured
* Boot sequence freezes after displaying inlet temperature

I've verified the eight XNC cables are correctly placed. I've had to go back to ILO 1.10 to allow me to boot with the four sockets active.

Still no response from HP level 2 support. This is a week after reporting the problem.

[Oscar A. Perez]

David,

The issue you are having with your DL980G7 is ROM related. Upgrading iLO3 to 1.15 and later just brought the issue to the surface. There is a new ROM version in the works that contains the fix. Please post your case number here so, I can get second level support to pull it ASAP.

[David Trounce]

Many thanks. My case # is 4624038849. I have had no response from HP level 2 support since opening the case 10 days ago.

I am running the most recent publicly available ROM - P66.

[David Trounce]

Thanks Oscar - an update. The lower XNC board was replaced yesterday, which solved the 4 vs. 8 socket issue. So it wasn't ILO 1.10 that was causing the socket deconfiguration. It just had the same symptoms as the ILO problem.

I'll wait for the new ROM (post P66) before upgrading ILO to 1.15 (or beyond). We have ILO fully configured now, so the ILO 1.10 unresponsive issue during configuration with hponcfg.exe should not be an issue any more. It's primarily an issue during the setup phase.

[sajad mohammed]

Have applied the latest iLO3 firmware update of 1.16 and yet this cleared the message and interface was responding. Then a few days later, reverted back to not responding.

Any ideas?

[NJK-Work]

I am having this problem on EVERY DL380 G7 we own.  It is quite random...although a few servers have this problem every day after I do a reset of the iLO.

I have updated to version 1.25 (latest from HP as of 6/27/2011) and updated to 8.70 support pack.  I still have this issue.  This is really annoying, not because it stops the iLO from working (the iLO itself is fine) but because it is a false positive and you never know when the server is having a REAL problem.  You alway have to check the server when you see the status, because you never know if it is not the usual iLO faux problem but rather a legit problem.

Any updates on this?

Nelson

[Oscar A. Perez]

Nelson,

What OS are you running on the 380s? Is this the 9006 trap issue reporting iLO isn't responding but iLO3 is fine and dandy?

[Chris_Coates]

We are having the same problem with BL460G7 and DL380G7 servers with an ILOE firmware of 1.25. Has anyone heard of a solution yet?

Chris

[MarcSameli]

Hi there

we have the same issue with the ilo3 1.25. does anybody have already a solution?

Thx Marc

[Oscar A. Perez]

I need more details. Is this the OS reporting iLO3 as unresponsive? What OS do you guys have?

[MarcSameli]

This message is coming from the ilo interface, when i connect from the management homepage. the os is windows server 2008 r2 with sp1.

[Oscar A. Perez]

OK, Agents for Windows are reporting iLO3 as not responding. We are working on this issue right now.

[stevenice]

Hello,

Just wondering if there were any further updates on this problem? We've seen it across 20+ DL380 G7 servers running Windows 2008 R2.

Thanks

[Oscar A. Perez]

We are still working on this issue. The issue is very intermittent so, finding the root cause has been nearly impossible.  Some customers have successfully tested a new iLO3 driver 3.4.0.0 Beta (to be officially released in Jan 2012) that contains some improvements but, few other customers have reported that the issue is still present with the new driver. 

We are looking into making important changes to the iLO3 CHIF interface code but, these changes requires extensive QA testing and the plan right now is to have them officially put in an iLO3 release scheduled for April 2012.

[Cederberg]

Hi.

Any update on this? Have this error on BL460c G7 and DL380 G7 ranging from ilo 3 fw 1.10 to 1.26

Remote Insight Agent: The Remote Insight Board/Integrated Lights-Out has detected a controller interface error.

[SNMP TRAP: 9006 in CPQSM2.MIB]

[Oscar A. Perez]

I hope that the new iLO3 CHIF interface code (currently under development) will solve this issue for good. The release is scheduled for April 2012.  I will ask few customers to test a beta as soon as possible.