Strange events from hpqilo2

[Pam Brosseau]

I had the issue with the hpqilo2 errors and teh server rebooting several times every day. It got worse and worse. I finally gathered enough info to get HP to come and fix. They had to replace the system board. Rebooting issue gone now for months.

[Donald J Wood]

I've just recently noticed that we have a few ProLiant BL685c G5 servers that run W3k3 enterprise. They won't reboot sucessfully and have all the hpqilo2 event ID 57 events. Last count was over 16929 instances of them since they were built about a month ago. This make reading the event logs real fun.

I've noticed a few other weird issues as well but I'm not sure that they're related to this issue with the PSP or firmware yet.

I'm troubleshooting a clutering issue where the cluster service just terminates due to the heartbeat loosing network connectivity. Having two of these builds clustered together and after doing a network trace, I've noticed that the heartbeat NIC (NC326i) on the active side of the cluster show four times the network latency as the failover node. There's no communication errors on the NIC cards or switches assoicated with the cluster.

After reading through this thread it seems to me that there's not one single combonation that resolves the event ID 57 or the inability to shutdown a server gracefull.

[Henry KK Chau]

Hi all,

Found that the hpqilo2 issue has been resolved by using PSP 3.0 and updating to the latest firmware on Proliant DL380 G5 servers.

Hope it is helpful to solve yours problem too.

Thank you!
Henry KK Chau

[Pavel_Simsalek]

Hello all,

We have some BL465c blades,
iLo 1.80
PSP 8.30
latest firwares for bios, on-board admins, iLo2 Management Controller Driver 1.12.0.0.,
still there are events 57 hpqilo2 failed get sensor reading, sensor 5.
Of course BMC IPMI Watchdog Timer Timeout: Action=System Power Reset and ASRs

so i don't think PSP 8.30 is solving this issue.

[SRowe]

Pavel,
The issue you are seeing is caused by the power module firmware. It should be 3.4, if you show a version of .7 or .8 I would update it ASAP. In the mean time you can disable the ASR to stop the server from re-booting. If you have a version of .5 there is no upgrade path, but on the good side .5 does not have this issue.

[Pavel_Simsalek]

Hello SRowe,

thanks for reply. But we have Power Management Firmware 3.4
There's no newest one, i think.
We'll disable ASR, but this will cause the server to freeze on blue screen if that event occur - am i right?
Thanks

[MattLavallee2]

Hi Pavel,

Please verify in Device Manager that your iLO Management Controller is using the correct driver version... there have been occasions where Windows is not using the version specified in VCA.

You are correct that disabling ASR will leave the server frozen, so it's not very useful.

-Matt

[Pavel_Simsalek]

Hi Matt,
yes, the driver shown in device manager is the same like in VCA.
Pavel

[w. wigman]

Same problem here, now after some months HP is willing to replace the motherboards.

Al the drivers and firmware are up to date.

[Zviadi]

I don`t understand after all these struggling, why nobody has written that the problem is solved in PSP 8.30 ?

i have spent 2 days looking for the solution on this board and i could not find it

at least in my case it worked perfectly on all 5 servers.

thanx anyway.

[w. wigman]

PSP 8.30 did not silve the issue for me.

On one server the systemboard has been replaced and the hpqilo2 messages don't appear anymore.
ASR stays, but less frequent.

Now testing with an spare indentically system to see if the ASR occurs also.

[w. wigman]

PSP 8.30 did not solve the issue for me.

On one server the systemboard has been replaced and the hpqilo2 messages don't appear anymore.
ASR stays, but less frequent.

Now testing with an spare indentically system to see if the ASR occurs also.

[maarten.vanachte]

Hi all,

we also have problems with some servers of us.
We have 4 BL680c G5 server that run in 2 clusters. These are verry important to us because they are our SAP SQL production servers.

At the moment we didn't notice any ASR and I hope that stays so, but every few seconds an error is reported in the event log.

---------------------------------------------
Event Type: Warning
Event Source: hpqilo2
Event Category: None
Event ID: 57
Description: GetBMCGlobals failed: 0xC1
---------------------------------------------

PSP 8.30 is installed on this servers for some time, but that isn't an solution.

---------------------------------------------
HP Insight Management Agents 8.30.0.0
ATI Radeon WindowsNT Miniport Driver 6.14.10.6746
Broadcom NetXtreme II GigE VBD 5.0.16.0
HP ProLiant Smart Array SAS/SATA Event Notification Service 6.16.0.64
HP ProLiant Array Configuration Utility 8.30.5.0
HP iLO Management Channel Interface Driver 1.15.0.0
HP ProLiant Remote Monitor Service 5.21.0.0
Network Teaming Intermediate Driver (NTID), 9.70.00.03, NDIS 5.1/5.2, x64, (free) on Win2K3 9.70.0.0
Headless Server Registry Update for Windows 1.0.0.0
HP ProLiant Array Configuration Utility (CLI) 8.30.5.0
HP ProLiant Array Diagnostics Utility 8.10.3.0
Smart Array SAS/SATA Controller Storport Driver 6.18.0.64
HP Insight Diagnostics Online Edition 8.3.0.3320
HP Lights-Out Online Configuration Utility 2.2.0.5
HP ProLiant iLO 2 Management Controller Driver 1.12.0.0
HP System Management Homepage 3.0.2.77
HP ProLiant Integrated ATA RAID Management Utility 2.2.0.0
HP NULL IPMI Controller Driver 1.0.0.0
HP Version Control Agent 2.2.1.830
Power Management Controller 3.4
Compaq Remote Insight ROM 1.79
HP Smart Array P400i Controller 5.26
---------------------------------------------

So does anybody have an idea or an solution?

Thanks

[Erdogan Temur]

Hi Maarten,

Here is the advisory for this issue:
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=3239482&prodTypeId=12169&objectID=c01802766

Online ROM Flash Component for Windows - HP Integrated Lights-Out 2
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3454580&prodNameId=3562397&swEnvOID=4022&swLang=8&mode=2&taskId=135&swItem=MTX-6f07f70d1ae04c0d843cc6a752

HP ProLiant Integrated Lights-Out Management Interface Driver for Windows Server 2003/2008
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3454580&prodNameId=3562397&swEnvOID=4022&swLang=8&mode=2&taskId=135&swItem=MTX-fa4293286417437a97aa0c4eb1

This is how we thank each other in the forum

http://forums11.itrc.hp.com/service/forums/helptips.do?#33

Enjoy:)

[fmash]

I still do not think the most recent advisory is completely correct or resolves the issue.

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01802766&dimid=1077331190&dicid=alr_mar10&jumpid=em_alerts/us/mar10/all/xbu/emailsubid/mrm/mcc/loc/rbu_category/alerts

I have a DL380 G5 running Windows 2003 X32.
I've been running with 1.81 iLo firmware and 1.14.0.0 for ilo driver for a while now with ASR disabled and I am still getting the same Error 57 duplicates over and over. I'm not going to enable ASR to see if it reboots or not.

[Megan Bigelow]

I am having the exact same problems except with DL380 G6 running Windows 2008 R2. I have updated the iLO Firmware to 1.81. Although the iLO Driver update utility says the driver is updated, the hpqilo2.sys version is 1.9. Am I looking at the wrong driver?

[maarten.vanachte]

Our problem was solved using the ILO 1.81 and updating the bios version.

@Megan Bigelow: try to manualy replace the drivers in the device panel.

[Megan Bigelow]

Thanks for the advice. It turns out there is a difference between the Lights Out Management Interface Driver and the iLO2 Managment Controller Driver. It is the latter which needs to be updated for this problem.

[hpqilo2]

BL460c G1 server, it reboots every 7-10 days, ASR reboot, there's 2 warnings. in Event Viewer Properties
Event ID 57,: Soure hpqilo2
Decription: NetFN 0X6, command 0x25 timed out

then the ASR reboots the server
running windows 2003 R2 Server
ILO Firwamre 1.81, and HP Proliant IL2 Management Controller Driver version is 1.8.0.0

these are the errors in ILO
iLO 2 03/31/2010 21:58 03/31/2010 21:58 1 BMC IPMI Watchdog Timer Timeout: Action=System Power Reset.
iLO 2 03/31/2010 21:58 03/31/2010 21:58 1 Server power removed.
iLO 2 03/31/2010 21:59 03/31/2010 21:59 1 Server power restored


iLO 2 03/22/2010 10:43 03/22/2010 10:43 1 BMC IPMI Watchdog Timer Timeout: Action=System Power Reset.
iLO 2 03/22/2010 10:43 03/22/2010 10:43 1 Server power removed.
iLO 2 03/22/2010 10:43 03/22/2010 10:43 1 Server power restored.

Please advise

[Tom33]

I am having the exact same problem with 5 BL460c blade servers. Updating to the current firmware, iLO 1.82, and driver 1.15 did NOT fix the problem.

I have opened 3 tickets with HP and gotten the same answer every time, "install 1.13+ and iLO 1.82".

THIS DOES NOT FIX THE PROBLEM.

HP please fix this.

[James Kennedy_5]

Give iLO2 Firmware 2.00 a try, it's now available.

[Donald J Wood]

This is an ILO driver issue. hpqilo2.sys v1.11.0 is the root cause. Upgrade it. I've used v1.12.0. Once you upgrade it it will be necessary to remove power for the ILO. On Blade Servers you can use the CLI on the OA after you shutdown the OS to reset the ILO (reset server #). On rack and stack servers you will have to shutdown the OS and have someone remove the power cord for about 20 seconds.

[Lau Poh Heng]

Hi all,

I have the firmware updated to 1.82 running Win 2008 R2 64bits. 8GB of RAM. hpqilo still giving me this reboot issue once every 2 weeks. Anyone has the solution?

[Norman Ogatis]

Any update on the solution to this problem?