Remote Insight/ Integrated Lights-Out interface error

[orbtastic]

I've been having this problem now for a number of months on ProLiant DL360 G7s running Win2008 R2.

We've ~200 other HP servers running a variety of OSs but we are only having this issue on the 360 G7s. Suspect it because it's a problem with the iLO3.

I initially upgraded the iLO3 firmware to 1.25 from 1.20 and then to 1.26 and then to 1.28(b) but the problem kept occuring. I know that you are supposed to keep the driver updated inline with the firmware and this is where it got confusing for me.

Looking at the VCA via HP SIM I could see that there were two drivers listed:

HP ProLiant iLO 3/4 Management Controller Driver Package for Windows Server 2008 X64
HP ProLiant iLO 3 Management Controller Driver Package for Windows Server 2008 X64

both of which were listed in the repository as v3.5.0.0 but it kept failing on updating both.

I had to do this manually using cp016189.exe for the first one and cp013868.exe for the second. However you will note from the notes on the cp016189.exe that:

Prerequisite Notes

The HP ProLiant iLO 3/4 Channel Interface Driver for Windows X64 (version 3.4.0.0 or later) must be installed prior to this component. The Channel Interface Driver was previously included within this component, but is now installed separately.

Updating both the drivers mentioned above did NOT resolve the problem (we were running on v3.3.0.0 on the first and v3.2.0.0 on the second for most of the servers as they were built with the same PSP).

Obviously part of the updates means a reboot of the server, so the error on the iLO3 always got cleared. It was usually ok for a week or so but came back on every server. I then spent some more time investigating and found a few HP threads. One listed the Channel Interface Driver (hinted at in the release notes) but I couldn't find this listed in the PSP or the VCA so was left scratching my head. Eventually I managed to find it on HP's website and downloaded cp016736.exe which is v3.5.0.0 which I then upgraded manually. Some of the servers were running v3.1.0.0. and some were running v3.3.0.0.

Since updating it all servers have been fine for a couple of weeks now. The only servers which have come back with error have been the two which intentionally were left alone and didn't have the CHIF Driver updated. It's also worth noting that since this last update was done, it's now listed in the VCA, which it wasn't before.

So they now look like this:

HP ProLiant iLO 3/4 Channel Interface Driver for Windows X64 3.5.0.0 3.5.0.0 HP ProLiant iLO 3/4 Management Controller Driver Package for Windows Server 2008 X64 3.5.0.0 3.5.0.0 HP ProLiant iLO 3 Management Controller Driver Package for Windows Server 2008 X64 3.4.0.0 3.3.0.0
Online ROM Flash Component for Windows x64 - HP ProLiant DL360 G7 (P68) Servers 2011.05.05 2011.05.05

Online ROM Flash Component for Windows - HP Integrated Lights-Out 3 1.28 1.28

Hope this is of use to someone, as I've already wasted a couple of months trying to resolve this issue.

[Oscar A. Perez]

I cannot longer reproduce the issue after the fix we added to driver 3.4.0.0 but, since lots of people still report it, the case remains open and we are actively working on trying to reproduce the issue in our lab.

One thing we did was to re-write the whole iLO3 Channel Interface subsystem (aka CHIF) in version 1.29 in order to solve few known communication issues. The below FTP contains the new iLO3 version 1.29.

FTP Access  :

     ftp://il0t3st:G1vme1LO@ftp.usa.hp.com 

or:

     ftp://il0t3st:G1vme1LO@15.192.32.78  (Normal active IP)

or:

     ftp://il0t3st:G1vme1LO@15.193.0.64  (Standby IP)

HTTPS Access:

     https://ftp.usa.hp.com/hprc

[orbtastic]

Do you think the issue in some cases may be down to the fact that by default, the CHIF isn't installed by certain versions of the PSP or as part of the driver updates?

[Oscar A. Perez]

I don't think this would be the case because, when the CHIF driver isn't running, nothing on the OS side can talk to iLO. 

[thefez]

Since installing 1.28 firmware, 3.0.5 Channel Interface driver and 3.0.5(?) Management driver I have not seen this issue on servers we tested on.  One interesting note is the management driver displays version 3.0.5 during the install but 3.0.4 when checked through Device Manager or HPWMI query.  Has anyone else encountered this?  Can anyone confirm that these firmware/driver versions fix this issue?

[Oscar A. Perez]

There was a fix in Driver 3.4.0.0 but some customers came back saying the new driver just made the issue happen less often.  We recently released iLO3 1.29 (see FTP links few posts above this one) that contains a new CHIF subsystem in order to solve few known communication issues. 

Please give 1.29 a try and let me know the results.

[orbtastic]

When you say "talk" what do you actually mean?

I ask because I've had this issue for months now, it must go back at least 4-5 months and the iLO3 card works as normal for a couple of weeks then all of a sudden comes up in SMH/HP SIM with an error:

Event Details: (SNMP) Remote Insight/ Integrated LightsOut Interface Error (9006)

Resetting the iLO3 clears it, so does rebooting the server (as you would expect) but ONLY installing the CHIF resolves the problem (not updating the other driver or the firmware).

See my more detailed post further up for more info.

[orbtastic]

Yes, had the same issue. Once the CHIF is installed it shows up in the VCA but oddly the Management driver shows twice.

HP ProLiant iLO 3/4 Management Controller Driver Package for Windows Server 2008 X64 3.5.0.0 3.5.0.0

HP ProLiant iLO 3 Management Controller Driver Package for Windows Server 2008 X64 3.4.0.0 3.3.0.0

[tedpro]

what is the username and password for the FTP sites?

Ted

[Oscar A. Perez]

Please use iLO3 v1.50

Online ROM Flash Component for Windows 64-bit
ftp://ftp.hp.com/pub/softlib2/software1/sc-windows-fw-ilo/p1728391553/v73058

Online ROM Flash Component for Windows 32-bit
ftp://ftp.hp.com/pub/softlib2/software1/sc-windows-fw-ilo/p901462262/v73059

Online ROM Flash Component for Linux
ftp://ftp.hp.com/pub/softlib2/software1/sc-linux-fw-ilo/p1255562964/v73060

Note: clear your browser cache after upgrading iLO3 to this version.

[Tripoint HPUX Admin]

so we've got a bunch of BL460c G7's running Windows 2008 R2 and we upgraded to the following versions:

HP ProLiant iLO 3/4 Management Controller Driver Package for Windows 2008/2012 x64 - 3.7.0.0

Online ROM Flash Component for Windows - HP Integrated Lights-Out 3 - 1.50 

but we're still getting the  Remote Insight/ Integrated LightsOut Interface Error (9006) error from HP Sim.  Bear in mind that nothing actually happens to the iLO it's just an annoying alert we recieve a few times a week from 12 - 16 servers.

now someone mentioned installing the following driver will fix it:

HP Proliant iLO 3/4 Channel Interface Driver for Windows X64 version 3.7.0.0 (English (US), Japanese)

^ can someone confirm i need to install this as it doesn't appear in my list of installed drivers/firmware but if i expand Additional Items it is in the large list of items that can be installed

[Oscar A. Perez]

No, do not install those drivers. It won't make a difference.

I've been debugging this issue for several months on both the driver side and iLO firmware side but, progress has been very slow because, I've been able to reproduce this issue only on one of the 5 systems that I have in my lab where the issue is seen once every two or three weeks. Also, with the debug version of the channel interface driver, this issue has not been reproduced yet.

At this point, I do not believe this issue is iLO firmware related. I'm looking for somebody willing to send us a full HDD image from one of the systems that exhibits this issue. Please contact me via PM so, I can provide a private FTP site where the image can be uploaded.

Oscar

[zeroagemain]

We were plagued by this problem for a year or so.

Private correspondance with Oscar at HP suggested that it might be Windows OS' on G7/iLO3 that are running McAffee but correspondance stopped just as it started to look like that might be the case.

We have 100% solved this now (touch wood - no false alerts for 4 weeks now and prior to that we had the issue multiple times daily - our monitoring team are delirious) on all our affected W2k8 R2 x64 systems as follows;
- Manually Install cp017930 (3.7.7.0 CHIF)
- Manually Install cp017974 (3.7.7.0 Mgt Controller Driver)
- Reboot
...this worked whether iLO3 fware was 1.28 or 1.50, and NO other combination of drivers/fware worked....we tried many many combinations.

[Oscar A. Perez]

Update on this case: I believe we have found the root cause of this very intermittent issue and we are testing few solutions. Since the issue is very hard to reproduce for us, I'm thinking maybe some of you would like to test a Beta versions and report back the results.

For those of you running into the random Event ID 1111, SNMP Trap 9006, please go to the below FTP site and download iLO3 version 1.56 BETA. Please test this version and report back the results.

For those of you running into the random Event ID 78/79, I have a BETA version of the ProLiant Monitor Service version 3.9.0.0 on the same FTP site. Please test this version and report back the results.

ftp://ilo4me:G!v3t2me@ftp.usa.hp.com

[PNX Alan]

Tried the beta here on a BL460c G7 but still getting random 1111 events.

[Oscar A. Perez]

This is bad news.  Can you please send me your contact info via a PM? Thanks.

[Oscar A. Perez]

Yes, in the iLO3 Channel Interface driver (a.k.a CHIF driver) version 3.9.0.0 we have made changes to the CPQSMIF.DLL to improve the communication between iLO and the Agents/SMH.

Also, working with another customer last week, I noticed that they had a very old version of CPQSMIF.DLL in their systems, version that has known issues reestablishing communication with iLO. In case you didn't know, CPQSMIF.DLL is the API that allows user mode applications like the Agents and SMH to communicate with iLO3/iLO4 through the CHIF driver and it is shipped as a component of the iLO3/4 CHIF driver package.

Apparently, Version Control Agent (VCA) on this customer would only display the iLO3 Management Driver version (CORE) so, since they were relying on VCA to update their drivers and Agents, they were only updating CORE driver, leaving CHIF driver with a very old version on their systems.

Please check what version of CPQSMIF.DLL you have in your system. If you have anything older than version 3.5.0.0 you need to re-install the iLO3/4 Channel Interface Driver package on your systems.

The location of CPQSMIF.DLL is:
%SystemRoot%\System32 for a 32bit Windows
%SystemRoot%\System32 and %SystemRoot%\SysWOW64 for a 64bit Windows.


Also, if you want to test Beta drivers v3.9.0.0, you can find them in the below FTP.
ftp://ilo4me:G!v3t2me@ftp.usa.hp.com/driver/iLO3-v3.9.0.0-beta/

[CLEB]

I am seeing the 9006 snmp errors for two of my BL460c G7 servers. One of them definitely has old CPQSMIF.DLL and Agents.

I will upgrade these and report back with the result.

[PNX Alan]

Sorry for the long delay in getting back to you, I've been busy with other things.  My CHIF driver was out-of-date (3.1) and not showing in our VCA page.  I updated to the 3.9 drivers on the FTP site and flashed the iLO to 1.57.  So far so good so we'll see if it stays that way over the next few days.

Thanks

Alan

[PNX Alan]

One thing I have noticed now is that after applying the updates both servers are reporting a warning in System Center.  The Warning is

ServerOthers_SNMP.HealthState HealthWarning

and I also see

ServerOthers_SNMP.FailedComponent Other System Health (UID, POST)

I'm don't think these were showing before the update but may have been.  Everything in the SM Homepage is green and if I log onto the iLO everything looks green there are well.  I'm at a loss as to how to track this down.

[PNX Alan]

Ah solved it and it was unrelated :)

[Oscar A. Perez]

Please keep us updated with the results.

BTW, the iLO3 1.57 is no different than the version 1.55 on this regard. It is being released mainly to solve the DL980 false overtemp issue.

[CLEB]

I've received iLO 9006 interface error again after upgrading the CHF version.

HP Emulex 10GbE iSCSI Driver for Windows Server 2008 R2 4.0.317.0
HP Emulex 10GbE Driver for Windows Server 2008 R2 4.0.449.0
HP ProLiant Smart Array SAS/SATA Event Notification Service for Windows Server 2008 x64 Editions 6.26.0.64
HP ProLiant Smart Array P700m Clustering Support Software for Windows 1.3.0.0
HP ProLiant Array Configuration Utility for Windows 9.0.24.0
HP ProLiant Integrated Management Log Viewer for Windows Server x64 Editions 6.0.0.0
HP Network Configuration Utility for Windows Server 2008 R2 10.50.0.0
HP Insight Management Agents for Windows Server x64 Editions 9.30.0.0
Matrox G200eH Video Controller Driver for Windows Server 2008 X64 6.12.1.1020
Headless Server Registry Update for Windows 1.0.0.0
HP ProLiant Array Configuration Utility (CLI) for Windows 9.0.24.0
HP Insight Diagnostics Online Edition for Windows x64 Editions 9.0.0.4179
HP Lights-Out Online Configuration Utility for Windows 2003/2008/2012 x64 Editions 4.0.0.0
HP System Management Homepage for Windows x64 7.2.1.3
HP Storage Tape Drivers for Windows 3.5.0.0
PFA Server Registry Update for Windows 1.0.0.0
HP ProLiant iLO 3/4 Management Controller Driver Package for Windows Server 2008/2012 X64 3.8.0.0
HP Version Control Agent for Windows x64 7.2.0.0
HP Storage Fibre Channel Adapter Kit for the x64 QLogic Storport Driver 3.16.0.0
Online ROM Flash Component for Windows x64 - HP ProLiant BL460c G7 (I27) Servers 2011.05.05
Online ROM Flash Component for Windows x64 - HP Integrated Lights-Out 3 1.55

[Oscar A. Perez]

I don't see the  "HP ProLiant iLO 3/4 Channel Interface Driver for Windows X64" listed there. This is the one that comes with the CPQSMIF.DLL

Can you please double check it's version?  Are you testing the Beta CHIF driver v3.9.0.0 that I uploaded to the FTP?

[CLEB]

Version 3.1 was installed but not showing in the VC agent software list. I've installed v3.7

I've downloaded the beta version. Shall wait and see if I still get the errors.