BL460c ILO2 problem after firmware (ROM flash) upgrade to 1.70

[Torbjorn Mohn]

After upgrading to the latest support pack (8.20A) with Online ROM Flash for windows x64 ver 1.70 ILO2 and management controller 1.9.0.0 Im getting hpqilo2 warning messages in my Windows System Event log like these:
-(Event ID 57)--
Failed GET SENSOR READING, sensor 9
NetFN 0x4, command 0x2D timed out
Maximum indicator count (1) exceeded
Duplicate thermal sensor instances: 1
Duplicate thermal sensor indices: 1
Duplicate thermal sensor instances: 2
----
and on and on....
They appear twice every day, on ALL three of my BL 460C blades. This did not happen at all before this upgrade.
Is there something wrong with the firmware? Is there something wrong with all my blades? Or is there anything I can do to prevent this from happening?
Systems are running Windows Server ver 2003 R2 x64 with all security patches installed!
(Same firmware and controller driver works fine on my "older" ML370 systems!)

[James Kennedy_4]

I am having these messages as well on my BL460c blades.

I've had the ILo 1.70 firmware on for a long time and hasn't been a problem until I installed the PSP 8.20 yesterday. So definately something with the drivers int he 8.20 pack that is doing it.

Also getting some blue screen crashes now too.

[Torbjorn Mohn]

I'va also had a few BSOD crashes lately. Never happened before! So 8.20 is definately to blame. What to do? Go back to 8.15? Is that possible?

[James Kennedy_4]

So far, it seems just specific to x64 machines. I'm hoping to not have to uninstall the entire PSP. I'd like to try to isolate which driver is causing it.

Right now I have ASR disabled on some of the machines, hoping to catch the crash memory dump this time.

[Torbjorn Mohn]

It's only my Blade servers doing this. I have ML370's with 64 bits installation, and they are working fine with support pack 8.20. So I guess it's a BL460c/x64 issue? (I have no BL460c's with 32bit OS to compare with).

[James Kennedy_4]

Yeah, just BL460c with x64.

I reverted the ilo2 management driver back to 1.8 and the hpqilo2 messages have not returned yet.

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3808910&prodNameId=3808911&swEnvOID=1113&swLang=13&mode=2&taskId=135&swItem=MTX-03c99c6a52a047d0bf38df0331

Still waiting to see if there's any more crashes though.

[Torbjorn Mohn]

I've now done the same here; downgraded to 1.8 on ONE of my blades. I'll keep an eye on it and see what happens. This hpqilo2 message only appeared twice a day here, so I may be in for an overnight wait. :-)

[Torbjorn Mohn]

Seems to me now that going back to the ilo2 management driver back 1.8 may have solved the problem. Since I did that on a few of the servers here, the problem seems to be gone. I will now downgrade ALL my servers here to the previous version (i DID get the same error messages on the ML370's as well eventually!) tonight, and keep an eye on them over the weekend.
Now what? Will HP do something about this, or what?

[James Kennedy_4]

Yeah, so far so good here. It definately stopped the error message from appearing, and have not yet had any crashes. Time will tell.

PSP 8.20 isn't "officially" available until the 31st of the month. It was listed as available for servers for a few days, and then it was pulled. Perhaps HP discovered the issue and is working on a newer version of that driver.

I'll update again if I hear anything or have any further issues.

[Torbjorn Mohn]

Well.... bad news... the error message reappeared here on one of the servers, so there is definitely more to it than just this driver. The error message is obviously not so frequent as it was before, but still there. The fact that HP has pulled the 8.20 update package I guess means they have realized something is wrong. I hope I can live with the problem until they re-release it?

[Cliff McQuigg]

We're getting these too, as well as a LOT of blue screen crashes. Kind of a problem since we're rebuilding a lot of stuff to 64bit right now and our firmware/drivers are the most current.

Anyone find which driver or firmware is at fault?

[James Kennedy_4]

For me, me upgrading to PSP 8.20 and keeping the downgraded version of the ilo2 management driver back at 1.8, solves all my issues so far.

No more errors, and no reboots... so far.

[Torbjorn Mohn]

All my 64 bit systems seem to be stable now using ilo2 management driver 1.8 ! However two of my older 32 bit systems (with only ilo1) has developed BSOD's with no event error messages after I upgraded them with support pack 8.20. Non of my servers (neither ilo2 nor ilo1, 32 or 64 bits) has BSOD'ed before.
They were all stable with support pack 8.15, so my advice is to stay with 8.15B until HP has re-released 8.20.

[Torbjorn Mohn]

HP today re-released Support Pack 8.20(A): http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=no&prodTypeId=3709945&prodSeriesId=1842750&prodNameId=3288156&swEnvOID=1113&swLang=8&mode=2&taskId=135&swItem=MTX-7e8dfb45569e4425bb197254d7
Have anyone here had the guts to try it out? The 1.9 ILO2 controller seems to me to be the same in the "new" Support Pack?

[James Kennedy_4]

Yeah, i'm afraid of that driver. I'm not going to be putting on any blades for a while until it gets tested. I'm sure there will be a lot more people rolling out the 8.20 support pack, now that it's available again. Keep an eye on the forums.

[Rastislav Isunin]

The same issue on new DL 580 G5 with Windows 2003 x64.
After facing BSOD issues according recomendation implemented newest PSP - unfortunately 8.20(A). Right after installation server bluescreened on crappy Network driver bundled here. I have upgraded this particular driver to one released March 27 and server seems to be stable, but this hpqilo2 event Warnings ID 57 start comming to system log daily. So no improvment in (A) version done regarding this issue.

[Kostas Kakogiannis]

Try installing,

HP ProLiant iLO 2 Management Controller Driver for Windows Server 2003
Version :1.11.0.0
7 Apr 2009

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1842750&swItem=MTX-295127d6d21d4885a860038d5b&prodNameId=3288156&swEnvOID=1005&swLang=8&taskId=135&mode=4&idx=2

It seems to correct this issue...

[Kostas Kakogiannis]

Sorry, the above link is for x86 win2k3, here is the link for Win2k3 x64!!!

HP ProLiant iLO 2 Management Controller Driver for Windows Server 2003 x64 Editions


Type: Driver - System Management
Version: 1.11.0.0 (7 Apr 2009)
Operating System(s): Microsoft Windows Server 2003 for 64-bit Extended Systems

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1842750&swItem=MTX-b016a4092d95486b88c4ebe86d&prodNameId=3288156&swEnvOID=1113&swLang=8&taskId=135&mode=4&idx=2

[Rcash_1]

I installed Version: 1.11.0.0 on my blade bl460 server running Win2008 x64 and it did not help. It continues to report hpqilo2 errors in the event log then reboot (ASR). Went back to version 1.8 and so far the errors have not returned.

Installed version 1.11 on a ML350 G5 running win2008 x64 and it also started reporting hpqilo2 errors and reboots (ASR) went back to 1.90 on it and errors went away.

Installed version 1.11 on a DL360 G4 runnning Win2008 x86 and so far it has no errors and no reboots (ASR)

[Kostas Kakogiannis]

My suggestion as a workaround, would be, in the event, you see errors in system event log like :

57 hpqilo2 Duplicate thermal sensor instances: 5
57 hpqilo2 Duplicate thermal sensor indices: 6

To try installing iLO2 Driver 1.11.0.0

If after installation of driver and possible OS restart errors persist, try versions 1.0.8.0 and 1.0.9.0 respectively.

[Torbjorn Mohn]

It seems to me that HP has NOT addressed this problem at all? What does it take for them to do so? I'm sticking to ver 1.8 for now. I'm not even going to try the new version 1.11.0

[Steven Jordan]

I have the same problem. I am going to disable the ASR so it will not bring the server down.

[Jim Dunne]

I also see this, but on 32 bit Windows 2003. Server is a DL585 G2. I went back to Sys Mgmt Driver 1.8 (from PSP 8.15) and we will see if it stops the Event Log message. I have had two ASR reboots but it is not yet fully proven they were related to this driver.

[Dmitry Dolgov]

I have two blade(bl460c) with same problem, one 32bit Windows 2003, another 64bit. 1.8 and 1.11 management driver does not solve this problem. I had to disable the ASR.

[Torbjorn Mohn]

1.8 did not solve my problem until after a cold reboot of the blade! It's been working now without any problems since I went back to 1.8!