iLO4 unreachable

[rage866]

On a large number of my company's DL360p Gen8 servers the iLO appears to be completely unreachable.

We are running CentOS 6.6, the iLO firmware is at 2.30.

The servers are off warranty and the cost of swapping out the system board is too high.

The symptoms are:
Errors during post: Instead of getting the message to hit F8 to configure, I get an error message. See attached picture.

Intelligent provisioning won't load: I hit F10 during POST, it marks that I hit it, but it makes no visible effort to load it. hitting F11 and then choosing intelligent provisioning does nothing, also. 

Running hponcfg at the OS prompt gives errors. I ran it twice in the example shown the errors are slightly different.

[root@cnt]# hponcfg

HP Lights-Out Online Configuration utility

Version 4.4.0 Date 06/13/2014 (c) Hewlett-Packard Company, 2014

ERROR: CpqCiCreateFunc() 0 time failed.

Driver Error Code:(1,1h).

Driver Error Message: CPQCIDRV driver is not loaded.

ERROR: Error communicating with ILO

ERROR: Unable to communicate with the Management Processor.

@dq[root@cnt ~]# hponcfg

HP Lights-Out Online Configuration utility

Version 4.4.0 Date 06/13/2014 (c) Hewlett-Packard Company, 2014

ERROR: CpqCiCreateFunc() 0 time failed.

Driver Error Code:(1,1h).

Driver Error Message: CPQCIDRV driver is not loaded.

ERROR: CpqCiCreateFunc() 1 time failed.

Driver Error Code:(1,1h).

Driver Error Message: CPQCIDRV driver is not loaded.

ERROR: A general system error occurred while detecting Management Processor.

ACTION REQUIRED: Check if iLO and iLO driver are up and running.

@dq[root@cnt ~]#

When I run ipmitool nothing happens, I have hit cntl-c to get back to the prompt.

On some of the servers the iLO responds to pings, some don't.

The ones that respond to pings, seem to allow loading of the iLO logon page, but it doesn't allow me to put in any user information and that page loads extremely slowly.

I have tried the following:

Disconnecting all power for a full minute and powering back up.

Re-running the firmware install for version 2.30 from the OS prompt. This returned erorrs similar to hponcfg.

Toggling system maintenance switch 1 and 6 and booting in between.

[ITNW]

Try to load the iLO4 v2.40 (officialy comes out in the April SPP) :

https://www.hp.com/swpublishing/MTX-3cef797a6ca84451b23fa0af60

OR

ftp://ftp.hp.com/pub/softlib2/software1/sc-windows-fw-ilo/p1382755333/v111235

I have seen similar issues on a number of servers running different version of RHEL, SuSE and Windows. All after upgrading to iLO4 v2.30. I have a case open with HP and this was the first thing they asked me to try was to upgrade the firmware. In my case the firmware only addressed some of the servers but on some the system board had to be replaced. The replaced boards were shipped back to them and they are investigating the issue.

We saw alerts in the iLO Event log which looked like this:

Embedded Flash/SD-CARD: Failed

Embedded Flash/SD-CARD: Restarted

Embedded Flash/SD-CARD: Media controller exception 01.

My other recommendation is you disable IPMI, it is an in secure protocol and we disable it by default on all of our iLOs.

[munichman]

I have also tried installing the "hponcfg" tool on Ubuntu and Redhat 7, and in both cases I get the following error. And there is no solid solution to resolving this error with the "CPQCIDRV driver is not loaded".  Can someone please help? 

Note that, I am able to use python-hpilo module to at least communicate with the HP iLO software. However, both Python and Ansible require the "hponcfg" tool to work in order to configure the HP iLO. So I am stuck at the moment. 

[@localhost ~]$ sudo hponcfg
HP Lights-Out Online Configuration utility
Version 4.6.0 Date 09/28/2015 (c) Hewlett-Packard Company, 2015

ERROR: CpqCiCreateFunc() 0 time failed.
Driver Error Code:(1,1h).
Driver Error Message: CPQCIDRV driver is not loaded.

ERROR: CpqCiCreateFunc() 1 time failed.
Driver Error Code:(1,1h).
Driver Error Message: CPQCIDRV driver is not loaded.
ERROR: Could not find the Management Processor Device Node.
Please check if the iLO driver module is installed and running properly.
ACTION REQUIRED: Install/Re-install hp Lights-Out driver package('hp-ilo' RPM) for iLO.

[Erdogan Temur]

Hi,

NAND flash that can be accessed by pressing F10 at POST during Server boot. I think NAND flash faulty.

re-flash Intelligent provisioning for problem fix.

Step 1
Install the SPP 2017.04

Step 2
Install the Intelligent provisioning 1.64B

Step 3
-Unplug the power cord then wait 3 min.
-Start the server and test it.

*** Please provide feedback on the latest situation.