Unable to establish communication with iLo/RILOE-II

[Josh Catana]

I'm getting the following error whenever I try to run hpconfig:
Waiting for Management processor...
HPONCFG RILOE-II/iLO setup and configuration utility
Version 1.5 (c) Hewlett-Packard Company, 2005

ERROR: Unable to establish communication with iLO/RILOE-II.
Management processor is busy

I've done the following:
Restarted all HP services.
Disabled/Re-Enabled the driver.
Rebooted the server.
Completely replaced the hardware on the server. (swapped the disks into another DL360 G5)

Nothing fixes this. Also, running hpdiags.exe hangs whenever it gets to anything concerning the mainboard (temperature/iLo/fans/power supplies) consuming high amounts of processor resources until it is killed.

System Management Homepage shows all items as green. Some don't appear to be detected though like 'power subsystem' shows up as '?'. Nothing is failed or degraded.

Any recommendations or ideas of what this could be? It has to be software/driver related I completely swapped the hardware and I'm getting the same problem. I have the exact same set-up in 200 other locations with no issues. Same model server, same OS/application image.






Software Versions:

HP Insight Management Agents for Windows 2000/Server 2003 7.60.0.0
HP Version Control Agent 2.1.5.760
Network Teaming Intermediate Driver (NTID), 8.40.00.23, NDIS 5.1/5.2, x86, (free) 8.40.0.0
HP iLO Management Channel Interface Driver 1.9.2195.0
Compaq Integrated Management Log Viewer Utility 5.10.0.0
HP Insight Diagnostics Online Edition for Windows 7.6.0.1984
HP System Management Home Page for Windows 2.1.6.156
Broadcom NetXtreme II GigE VBD 2.8.15.0
HP ProLiant iLO 2 Management Controller Driver 1.1.0.0
ATI Radeon WindowsNT Miniport Driver 6.14.10.6583
HP Lights-Out online configuration Utility for Windows 1.5.1.0
HP Proliant Remote Monitor Service for Windows 5.10.0.0
HP Integrated Tape Application and Utilities for Windows 1.0.2.0
Smart Array 5x and 6x Controllers SCSI Miniport Driver 5.74.0.32
HP Array Configuration Utility 7.60.18.0
Smart Array 5x and 6x Controllers Storport Driver 5.11.0.32
Smart Array SAS/SATA Controller Storport Driver 5.10.0.32
HP Smart Array SAS/SATA Event Notification Service 5.8.0.32
HP Array Diagnostics Utility 7.60.18.0
HP Array Configuration Utility (CLI) 7.60.19.0

[Raghuarch]

Can you ping to the ILO IP?
Can you do a Telnet or ssh to iLO.

What is the behavior when you try to access it through web?

[Raghuarch]

What is the ILO version?
Try this if you have access to the server.

Power-off the server, unplugged power cable
Open the chassis and set the iLO security override switch (switch number 1 in the system maintenance switch in ON Position)

Try to login ..

Power off the server, then restore the iLO security override switch to position OFF

[KarloChacon]

hi

- update firmware 1.43
- if it is updated reflash/reapply ilo firmware again
- Reset the iLO 2 to factory defaults through the F8 iLO 2 POST interface on the host server.
let server reboots again and set the values again

check this POST my thread at the end
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1204801

regards

[Josh Catana]

Can not ping, telnet or ssh to the iLo.
The webpage is not accepting connections.

The server is in another country so doing anything physical is out of the question. I already had a new server shipped out and they did a drive swap because I initially thought it was bad hardware, but the problem persisted through the change in hardware.

All 200 servers are running java 1.4.2_03-b02 and all work pristinely except this one.

Unfortunately, I can not go into POST and change settings without ILO up and functioning.

[KarloChacon]

hi

well at this point

the only way is someone onsite and do some more Troubleshooting since ilo IP is not responding

regards

[DjOracles]

Interesting. I just applied Firmware CD 8.40 to my DL360 G5 and I've lost complete iLo connectivity. It has the same IP but I can no longer connect to it.

[ppetriska]

I've also lost the iLo configuration after application of Firmware CD 8.40. As the server is 2000 KM away, I am stuck on how to make it work. Anyone has found the solution?
Ilo not reposponding to the HPONCFG. Thanks
'Peter

[rustsh]

I've just updated one of my DL360 G5 servers from Firmware CD 8.40 and lost connection with it's iLO (no response to ping). Than I've powered off/powered on the server and now it does not boot or even show anything on the attached monitor.

Any ideas ?


Best regards,
Rustam.

[Radical]

I'm having the same problem on a DL 380 G5.

[Josh Catana]

I believe, unfortunately, the only fix for this was to replace the server hardware.
Luckily I worked in an environment that had plenty of spares. We sent out a new identical DL360 G5 and did a drive swap.
The problematic server was sent back to HP.

[rustsh]

HP staff replaced my motherboard.

[swingarm]

I've also had bad luck with the 8.40 version of the Firmware Maintenance CD, both BL460C G4's I've applied it to have lost ILO connectivity. As a matter of fact I can't even power them on anymore, the chassis seems to think there's no Blade in that bay.

[ppetriska]

In our case the HW MB had to be changed.
No response from HP on this firmware update?

[rustsh]

I've updated two other DL360 G5 servers since my first incidence. Both servers are fine.

[Peter Voyno-Yasenetskiy]

Same bad luck.

iLO just died after updating one of our BL460C from FW 8.40 CD.

[Peter Voyno-Yasenetskiy]

Sadly, not just iLO died, but whole server died.

After removing and inserting it back to the enclousure, blade server is in "Unknow" status and won't start.

[swingarm]

I got excited when I saw 8.50 was out so I tried it but sadly it does the same thing as 8.40, my system board is now a paperweight.

[Kyle Stokes]

Has there been any update on this? I have seven servers with this very same problem. Unfortunately all of them remote.

[Tyno]

I have this issue with about 25 servers, after a long discussion HP has recommended the follwoing config.

Online ROM Flash Component for Windows - HP Integrated Lights-Out 2 1.81 18 Jan 2010

HP ProLiant Integrated Lights-Out Management Interface Driver for Windows Server 2003/2008 1.15.0.0 18 Aug 2009

HP ProLiant iLO 2 Management Controller Driver for Windows Server 2003 1.12.0.0 14 Sep 2009

In order to get the ILO working to do the updated I will do a coldboot.

[Kyle Stokes]

I am still experiencing failures of my ILO2s. I have had 6 fail in the last 24 hours. The only fix is a cold boot.

Does anyone have any ideas/suggestions? I am running FW 2.00 on all of them yet they are still failing.

Even a method to reset them from inside the OS would help. Right now the HP Lights Out Online Config Utility doesnt load, gives Error encountered in execution of hponcfg errors since the ILO is not pingable.

[Oscar A. Perez]

Kyle,

Are you running a port scan tool like Nessus Scanner against the iLO2s?
There is a known issue where Nessus scanner could cause iLO2 to stop responding.

[Kyle Stokes]

Thank you for your reply!

It is possible. The servers are located in a customer environment.

I can ask the customers network admins for clarification on this.

Do you know any way to test or sense if this is happening, event logs either on the ILO or on the server? I would assume if its scanning the ILO it might be scanning the server, so are there any tools I can run on the server to detect this?

[Oscar A. Perez]

I guess, a way to know this is by looking at the scan results. Most of this tools will create a report. I would expect that the report would show what host stopped responding during the scan. At least Nessus Scanner does this.
Now, the issue I'm seeing in iLO with Nessus Scanner is not related to a particular vulnerability test script (or pluging), it is the fact that during the scan, iLO gets flooded with multiple TCP connections at the same time.

[Kyle Stokes]

Oscar, you are providing me with more help and leads than I have obtained in over 15 HP tickets!

Thank you!

You mention its a known issue. I believe you since I have this (or a similar) known issue as well. Do you know if HP has an official docs or statements on this that you have come accross?

I have found some references to Nessus, but recently the corporate network these servers exist on just migrated to a new alerting software and I believe that it may have some sort of vulnerability scanning component as well which may be causing these ILOs to fail en masse. I now have about 9 failed ILOs in little under 4-5 days. Failed in that they need power removed from the server to reset themselves to start working again, and then only until they fail again.