Strange events from hpqilo2

[Jim Miller]

Just wanted to add that after trying ilo2 driver version 1.11 on 2003 and 2008 x64 servers (BL460c G5) ASR reboots still continued. Systems have been rolled back to ilo2 driver 1.80 and fingers are crossed. I have to put these systems in production next week.

[Mark_120]

Hi,
I have just completed migrating a few Netware (6.5 SP7) servers to new DL380 G5 hardware and PSP 8.10a
All are firmware patched to 8.40

Two of my servers keep rebooting for no aparent reason at all. Insight Manager indicates watchdog timer has triggered.

While the rest of my servers give me power management in iLo (power capping and power graphing) Both these two servers that keep rebooting do not give me any power management capabilities. I wonder if there is any correlation?
I am at my wits end as both these two servers are mission critical, so any info or advice will be gladly accepted.
I have disabled ASR but still had two reboots today.
Oh - and both times when I went to look at them the servers were sitting at the dos prompt saying "command.com not found and something about a memory error".
A ctrl-alt-del rebooted back up fine again.

Mark

[mjoyce4]

I applied PSP8.20A to DL385 g2's and DL385 g2 and most reboot from ASR after update. I backed 585 iLO 2 Management Controller Driver down from 1.11 to 1.8 manually. No prompt for reboot. Server rebooted again from ASR one day later, but keep in mind was not rebooted after driver change. I'll update if this is stable after reboot..... Note: PSP applied to DL585 G1 which has a different iLO (not iLO2) and appear to be stable. No issues with ASR on the older machines.

[mdelorie]

Just to add my name to the list of people having trouble - I'm having the same issue. DL380 G5, W2003 R2 x86, SP2

I just brought all my HP software up to date. See attached PDF for current inventory. I'll report back in a couple days on my results.

[Chris Ciapala]

I got this from HP support:
SOLUTION:
Work around -

1. It is recommended to force downgrade the psp to 8.15

2. Or downgrade the ILO Management Controller driver to 1.80 ,SMH version to 2.1.15.210 and HP Insight Management agents to 8.15, which are part of PSP 8.15 by force degrade.

Since I didn't liked it, they told me that problem will be escalated and real solution (driver upgrade) will be developed, however there is no way to tell when this will happen. Problem occurs only on 64 bit machines and ASR reboots might happen as the problems are caused by wrong reading from sensors.
Just FYI.

[Mihir Patel]

I even downgrade all the components to PSP 8.15 but the hpqilo2 error still presist. HP needs to come out with a fix ASAP!

[JerryS]

The problem we were experiencing was the server would not shutdown for 20 minutes after PSP 8.20 was applied to a DL360 G5 x32 2003 server. We also received event id 57. HP posted an advisory about this but the solution to upgrade to HP ProLiant iLO 2 Management Controller Driver Version 1.11.0.0 did not work for us: http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01704271&dimid=1045547850&dicid=alr_apr09&jumpid=em_alerts/us/apr09/all/xbu/emailsubid/mrm/mcc/loc/rbu_category/alerts.

I was able to solve the problem by downgrading the following components using psp 8.15(B):
HP Proliant ILO 2 Management Controller Driver to 1.8.0.0
HP Proliant Remote Monitor Service to 5.20.0.0
HP Proliant Interface Lights-Out Interface Management Driver to 1.13.0.0
HP Proliant Smart Array SAS/SATA Event Notification Service for Windows Server 2003/2008 to 6.10.0.32
HP System Management Homepage to 2.1.15.210

Hope this helps someone.

[mdelorie]

If HP thinks the problem only affects 64 bit systems, they need to know that 32 bit operating systems are having trouble as well.

I updated all my HP software as per the inventory attached to my post yesterday and am still having hpqilo2 errors in the system event log. No ASR reboots yet, but I think I'll disable ASR just to be safe.

So, to sort of recap what's happening, in general terms: It appears that the iLO Management Controller driver version 1.90 (and 1.10?) is causing erroneous errors to be reported, and possibly prompting ASR reboots as a result.

Someone correct me if I'm wrong. Hopefully HP releases an update soon.

[Vince Strausser]

I have confirmed that ilo 1.11 running in a 32 bit environment causes the errors as well. We've also had about 4 machines reboot because of this within 3 days.

[Guido Koetter]

Hello everybody,

same effects on Blade BL465c G5 running W2K3 x86 after upgrading to PSP 8.20. I also made a iLo firmwareupgrade 1.60 -> 1.70.

Is it possible to downgrade the firmware? May this problem being solved with an older firmware?

Driverversion 1.11 didn't solved the problem; using older versions also.

Any help is welcome.

Guido

PS: Solution from hp-Support was to update the driver on version 1.11

[olivier.brian]

I had the same problem with x64 and x32. HP could not help. but I found this nice forum.

So I downgraded the HP Proliant iLO2 Management Controller Driver from 1.9 back to 1.8 (version 1.11 doesnâ t work). After downgrade and reboot it works find. No ASR reboots and no event from hpqilo2.

Waiting for the next driver version.

[Chris Ciapala]

I have confirmation from HP that they are working on some fix, however they are unable to predict when it will be available. I would like to avoid messing up with different, forced driver versions on production machines, I had bad experiences.
For the moment I disabled ASR and I'm waiting for the fix.

[Roger Kihn]

I am also having this issue. I am back dating to V 1.8. I have noticed that when you do that, the "HP ProLiant iLO 2 Management Conctroller Driver" listed under "Multifunction adapters" goes away.

This makes me feel better, as it is obvious that the iLO2 controller/driver is causing the issue. Thanks for the help guys.

[olivier.brian]

It doesn't go away. You can find the driver under "System Device"

[Roger Kihn]

Got it.

Thanks.

[Mihir Patel]

For those who use MOM 2005 for their servers - Are you receiving "Extremely high processor activity detected" in MOM from the servers? I have a list of these alerts from different servers and wonder if it was produced because of the ILO errors.

[Mark_120]

well, as I said, my OS's are Novell Netware 6.5 so the drivers and versions are different to Windows & Linux, but the issues are still the same.
I have patched the iLo firmware from 1.70 to 1.77 but still had one server shutdown.
I then updated the PSP on my servers from 8.10a to 8.20 and so far they have not rebooted again.

Shutdowns occur whether ASR is turned on or off, the only difference is that with ASR on, the server will reboot back into the OS. With ASR off the server shuts down and has a flashing cursor in the top lefthand corner and requires ctl-alt-del to reboot back up again.

Mark

[olivier.brian]

We have also MOM and a abnormal high CPU activity. A lot of hardware interupts detected by processmon (from sysinternals). No idea where this came from.

[Juampa]

Same problem here, in a Windows Server 2003 SP1.

I wonder why if this is a global issue, the PSP 8.20 is still available for downloading.

It´s a joke that I had to upgrade to PSP 8.20 in order to upgrade the NCU and NC373i drivers to fix the TOE issues, ending up with all components updated except the NCU and the NC373i due to this bug:
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01684544〈=en&cc=us&taskId=101&prodSeriesId=428936&prodTypeId=15351

I had 1 problem before following HP instructions. Now I have 2.

Kind regards.

[JFMartin]

We revert to 1.80 of the ilo2 device driver and found that we are still experiencing the problem... we are now testing the 1.80 dev driver + 8.20 support paq + 1.77 of the ilo2 firmware firmware..... stay tuned.

[MattLavallee2]

For all having problems downgrading:

Be sure to go to Device Manager -> Multifunction Adapters -> iLO Management Drive and choose "Rollback Driver", even if you've downgraded the driver via the Version Control Agent. We had two servers that, for whatever reason, did not "take" the downgrade until forced.

The last stable version is definitely 1.8, and the problem has affected us in both 2003 and 2008, 32- and 64-bit systems.

-Matt

PS- Also seems to be affecting RedHat folks as well.

[lbecker]

For what it's worth, this is happening on the new DL380 G6 series as well. Been working on a case with HP that they've had me go back and forth updating this driver, downgrading that one, updating the ROM...Still getting the iLO duplicated thermal sensor alerts.

[Roger Kihn]

Don't you wish they still had their own testing dept?

[Mihir Patel]

I was supposedly told by HP that they have a test lab to test the updates before making them public to the customer. I guess they fell asleep when testing the ILO drivers.

[lbecker]

Another error I failed to mention is this one:

Source: Cisserv
Event ID 24588

Sensor number 1 has reported that the internal temperature has exceeded the preset limit. This sensor is located in box 1 which is connected to port 1I of array controller P410i [Embedded]. The array controller may attempt to shut down power to the attached box and/or spin down the installed disk drive(s).

Addressed in the following advisory, but the updated System ROM didn't fix it:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c01717386&jumpid=reg_R1002_USEN