Server frozen, hang an reboot ASR EventID 57 hpqilo2

[c.wolf]

Hi,
since Update to newest Versions of Driver and Firmware some Server DL380 G5 and DL360 G5 were frozen, hang or reboot (ASR)

Eventlog shows at this time:

Event Type: Warning
Event Source: hpqilo2
Event Category: None
Event ID: 57
Date: 28.04.2009
Time: 05:47:58
User: N/A
Computer: NAMExx
Description:
NetFN 0x36, command 0x2 timed out
Data:
0000: 00 00 00 00 02 00 58 00 ......X.
0008: 00 00 00 00 39 00 00 80 ....9..
0010: 00 00 00 00 00 00 00 00 ........
0018: 00 00 00 00 00 00 00 00 ........
0020: 00 00 00 00 00 00 00 00 ........

or

NetFN 0x4, command 0x2D timed out

we have installed the following Versions:
HP Insight Management Agents for Windows Server 2003/2008 8.22.0.0
HP NC-Series Broadcom 1Gb Multifunction Driver for Windows Server 2003 4.6.112.0
HP ProLiant Smart Array SAS/SATA Event Notification Service for Windows Server 2003 and Windows Server 2008 6.12.0.32
HP ProLiant Array Configuration Utility for Windows 8.25.5.0
HP ProLiant Integrated Lights-Out Management Interface Driver for Windows Server 2003/2008 1.14.0.0
HP ProLiant Integrated Management Log Viewer for Windows 5.22.0.0
HP ProLiant Remote Monitor Service for Windows Server 2003/2008 5.21.0.0
HP Network Configuration Utility for Windows Server 2003 9.60.0.0
HP ProLiant Array Configuration Utility (CLI) for Windows 8.25.5.0
HP ProLiant Array Diagnostics Utility for Windows 8.26.1.0
HP ProLiant Smart Array SAS/SATA Controller Driver for Windows Server 2003 6.14.0.32
HP Insight Diagnostics Online Edition for Windows Server 2003/2008 8.2.0.3058
High Precision Event Timer Identifier for Windows Server 2003 1.0.1.0
HP Lights-Out Online Configuration Utility for Windows Server 2003/2008 2.1.0.1
HP ProLiant iLO 2 Management Controller Driver for Windows Server 2003 1.11.0.0
HP System Management Homepage for Windows 3.0.0.64
HP NULL IPMI Controller Driver for Windows Server 2003 1.0.0.0
HP NC-Series Intel N1E Driver for Windows Server 2003 9.12.30.0
HP StorageWorks Fibre Channel Adapter Kit for the x86 QLogic Storport Driver 1.7.1.2
HP Version Control Agent for Windows 2.2.0.820
Online ROM Flash Component for Windows - HP Integrated Lights-Out 2 (Japanese) 1.70
Online ROM Flash Component for Windows - HP ProLiant DL380 G5 (P56) Servers 2008.11.01
Online ROM Flash Component for Windows - Smart Array P400 and P400i 5.26

The Revision History of:

HP ProLiant iLO 2 Management Controller Driver for Windows Server 2003

Version: 1.11.0.0 (7 Apr 2009)
Fixes


* Corrected an issue that could cause Windows to hang when shutting down, restarting, or entering hibernation.
* Resolved a problem where system could spontaneously reboot (ASR) if all CPU's were under continuous 100% load, and iLO 2 was reset (e.g. due to firmware update, changes to network settings, etc.).


Version 1.11.0.0 doesn't solve the problem.

Disable ASR can not solve the problem, it's better the server reboot and the application is after 5 min alive.

greetings

Chris

[James Kennedy_4]

Make sure you force downgrade the HP Proliant iLO2 Management Controller Driver for Windows Server 2003 to version 1.8.

Should resolve your issues with both the hpqilo2 errors and the ASR/Blue Screen reboots.

[c.wolf]

Thanks for the fast response.
We try it with Version 1.8.0.0 on one maschine.

[MattLavallee2]

As James said, you need to rollback to 1.8 until the issue is resolved. I posted an "informational" thread about this last month:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1331632

-Matt

PS - out of curiosity, are you running 64-bit?

[c.wolf]

Hi,
we have received responce from HP-Support.

3 Modules must downgrade until next release of new Patches from HP

HP ProLiant iLO 2 Management Controller Driver for Windows Server 2003
to Version 1.8.0.0 from Version 1.11.0.0

HP System Management Homepage for Windows
to Version 2.1.15.210 from Version 3.0.0.64

HP Insight Management Agents for Windows Server 2003/2008
to Version 8.15.0.0 from Version 8.22.0.0

we use 32bit

[CHunter]

The server that is having the ASR condition is running with:

Ilo 2 management controller 1.8.0.0
SMH⠦⠦⠦⠦⠦⠦⠦⠦⠦⠦⠦⠦⠦⠦. 2.1.15.210
Hp Insight Agents ⠦⠦⠦⠦⠦⠦..8.15.0.0

These are the ones recommended in the article but he server continues to ASR. My server is a BL- 460c G1

[CHunter]

The server that is having the ASR condition is running with:

Ilo 2 management controller 1.8.0.0
SMH 2.1.15.210
Hp Insight Agents 8.15.0.0

These are the ones recommended in the article but he server continues to ASR. My server is a BL- 460c G1

[c.wolf]

Hi CHunter,
the ASR ist caused by
Ilo 2 management controller 1.8.0.0

See revision historie:
Version 1.12.0.0 (Contains Bug Fixes)
Corrected a potential Windows STOP 0x9F bugcheck that could occur if the system is shut down while the driver is resetting the iLO 2 interface.
Version 1.11.2.0 (Contains Bug Fixes)
Fixed a iLO 2 communications error that caused command timeout messages in the Windows Event Log.
Version 1.11.1.0 (Contains Bug Fixes)
Fixed improper logging of warnings to the Windows Event Log after a reset of the iLO 2 interface. These messages all used Event ID 57.
Version 1.11.0.0 (Contains Bug Fixes)
Fixes
Corrected an issue that could cause Windows to hang when shutting down, restarting, or entering hibernation.
Resolved a problem where system could spontaneously reboot (ASR) if all CPU's were under continuous 100% load, and iLO 2 was reset (e.g. due to firmware update, changes to network settings, etc.).

please use:
HP ProLiant iLO 2 Management Controller Driver for Windows Server 2003 1.12.0.0
HP Insight Management Agents for Windows Server 2003/2008 8.30.0.0
HP System Management Homepage for Windows 3.0.2.77

[HCH_1]

Normally upgrading to the latest versions fix the issue, if the iLO driver is not compatible with the Firmware or if the Driver is newer than the driver then the Server causes ASR, most of the times it gives event id 57 and sometimes issues with the watchdog timer, the source is always Hpqilo thats is the iLO driver upgrading to 1.15 of the driver and 1.81 of the FW should fix the issue:

This is the one for the driver:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=3884315&prodNameId=3884316&swEnvOID=4024&swLang=13&mode=2&taskId=135&swItem=MTX-12afece9bb00404792308ee0dc

and that one for the FW:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=3884315&prodNameId=3884316&swEnvOID=4024&swLang=13&mode=2&taskId=135&swItem=MTX-6f07f70d1ae04c0d843cc6a752

[HCH_1]

also make sure that you update the management controller driver, not just the interface because this one is the one that coordinates communication with windows specially for ASRs

This is the link for that one:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=3884315&prodNameId=3884316&swEnvOID=4024&swLang=13&mode=2&taskId=135&swItem=MTX-c3d2ff4892e04e9b87124aefca

I hope that helps to solve your issue