DL380 G5 / Linux RHEL 5 / Reboot by ASR

[Dean Barnes]

Just to let everyone know, I am running 3 DL380 G5 and have been having the same error message but I have NOT experienced a reboot, and so far no issues. The reason I guess is that I never installed HP OpenIPMI but instead installed the OpenIMPI from RHEL5 cd. As for my SMH it is working fine and providing what so far appears to be accurate stats. Just thought I would throw my 1.5 cents into it. For the record I have installed the following:
OpenIPMI-libs-2.0.6-5.el5.4
OpenIPMI-2.0.6-5.el5.4
OpenIPMI-tools-2.0.6-5.el5.4
No problems so far.

[Russell Cowan]

I am having the same ASR reboot issue with our 3 dl380 g5 servers running rhel 4.4. We are also having random lockups as well when the servers becomes complete unresponsive. We cannot even ping the servers when these lockups happen.
I was wonder why does this only seem to happen on Intel equiped servers? This has never happened on any of our AMD equiped 585 servers and it seems all the servers referenced in this thread are Intel part numbers.

[Russell Cowan]

We are also having this issue on 3 of our 4 dl320 g4 servers. 2 running RHEL 4.0, 1 running rhel 4.4, and 1 running Windows Server 2003. The only one that does not ASR reboot or lockup is the one running Windows server 2003.

[RobH_Seattle]

Pretty sure the ASR reboots are still caused by HP's version of OpenIPMI. Simply remove it.
Stop the hp agents
remove the hp-openipmi driver
restart agents.

problem fixed. I use the std redhat openipmi driver on over 250 machines and have not had a asr reboot since.

[Yossi Lagstein]

Hi Russell

Did you identify the root cause of your servers lockup?
I have couple of DL380 G4 running RHEL 3U8 with PSP 7.70 and RHEL 4U3 with PSP 7.60 that â freezesâ but do not ASR and I have yet to find the root cause of the freeze.

Yossi

[Megan Kielman]

I just had a DL 380 G5 Windows Server 2003 x32 R2 SP2 server experience the following bugcheck:

STOP: 0x0000000A (0xF7006C40, 0xD000001F, 0x00000000, 0x80837E42)

I realized the BIOS version installed is 20070406 and I plan to upgrade it to the latest. Anyone else seen any issues simliar to this? Does a BIOS upgrade resolve the
problem?

[Don Vanco - Linux Ninja]

While this thread deals with Linux, I think the consensus is that the HP driver is not optimal. Can't say for sure what the best recourse is with Windows, but I'd start by getting on all the latest code, and if required open a ticket with HP support.

[Radu Alexa]

hi, has anyone encountered this issue with hp-openipmi 8.2.0 from the psp 8.20?
thanks.

[Kris Knigga]

Yes, Radu, I just saw it on a RHEL machine running the packages from PSP 8.20.

[studerje]

I am also having this problem with a DL380 G5 and a DL360 G5 running x86 Windows Server 2003 R3.
I just found a firmware update cp010540.exe on the HP site released 29 Apr 2009.
The 'Fixes' page for the updated states:
"Fixed an issue where iLO could cause the ASR timer to trigger during a broadcast storm".
I've applied the update this morning hoping it will fix the problem on these 2 servers. If so, I will be pushing this update to all my G5 servers.
-J.S.

[studerje]

As of this morning, no reboots on servers in question. still monitoring.

[eis1953]

Jonathan Studer
How often would your re-boots occur, is one day with the 1.77 (29 Apr 2009) patch enought?

Ed

[studerje]

The reboots were happening at least twice a day. Yesterday and last night, no reboots, but I'm still monitoring to see if I have any reboots before pushing out to other servers. If I go more than 48 hours without reboot, I'm going to start pushing it out.

[makysk]

Do someone definetly solution of this problem?
Because there are a several methods,and I'm not sure which is the correct.

[makysk]

Do someone have definetly solution of this problem?
Because there are a several methods,and I'm not sure which is the correct.

[studerje]

I'm still getting NetFN 0x4, command 0x2D timed out errors in Event viewer.
I've had 2 crashes since I applied the firmware to the ILO.
One of which created a windows dump file that I debugged. it reported problem with a file serialcom.sys for a product specific driver we have installed.

but the second I can't tie to that problem driver.

I am also receiving other error in EV:
Failed GET SENSOR READING, sensor 22
Maximum indicator count (1) exceeded
Duplicate supply instances: 1
Duplicate supply instances: 2
Duplicate container instances: 3
Insert logical container 2 failed
Duplicate fan instances: 1
Duplicate fan instances: 2
Insert logical container 2 failed
Duplicate thermal sensor instances: 5


I'm not sure the firmware update fixed it.
Anyone get open a ticket with HP in this issue?

[Yossi Lagstein]

Hi,

Regarding RHEL 5.s ASRs, to avoid getting ASRs out of the blue we are installing all PSP components but HP-IPMI. (bare in mind that by doing this we are loosing the value that ASR gives in the case the server hangs, but until we are sure HP has fixed this issue we don't have a choice)

Regarding Windows 2003, we have an open case. We got an instructions to downgrade the following components:
HP Insight Management Agents for Windows Server 2003/2008 - http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1842750&prodNameId=3288156&swEnvOID=1005&swLang=8&taskId=135&swItem=MTX-27fa96e466cc46b1b6e3af1d5d&mode=3

HP System Management Homepage for Windows - http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1842750&prodNameId=3288156&swEnvOID=1005&swLang=8&taskId=135&swItem=MTX-84b4161b7cd3455fb34ac57586&mode=3

HP ProLiant iLO 2 Management Controller Driver for Windows Server 2003 - http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1842750&prodNameId=3288156&swEnvOID=1005&swLang=8&taskId=135&swItem=MTX-93850678b0aa4460a0527451a6&mode=3

As HP is scheduled to release PSP 8.25 in next few weeks I have asked to verify whether a solution will be included in this release. I'm waiting for a response, and as I much prefer to upgrade than to downgrade I didn't downgrade my servers yet.

Yossi

[eis1953]

We have downgraded to Proliant Support Pack 8.15 for Windows x64 and our DL585's have been up for 14 days.

And Yes we have a HP case #

Ed

[eis1953]

We have downgraded to Proliant Support Pack 8.15 for Windows x64 and our DL585's have been up for 14 days.

And Yes we have a HP case #:3605731827

Ed

[MarkVHDA]

Did HP ever get you all fixed?

I have a DL585 G2 x64 running SP 8.11 and am having similar issues with mysterious reboots and nothing in the event log but the standard 'previous shutdown unknown' message.

I noticed SP 8.25 came out on 6/2 and am getting ready to try it. Release notes state the following related to iLO
:
Fix Note:
Corrected an issue that could cause Windows to hang when shutting down, restarting, or entering hibernation.
Resolved a problem where system could spontaneously reboot (ASR) if all CPU's were under continuous 100% load, and iLO 2 was reset (e.g. due to firmware update, changes to network settings, etc.).

I had issues rebooting also and am hoping this is the fix!

Mark

[MarkVHDA]

Well, I just opened a case with HP Support as well. My HP engineer is sending links for downgrading, probably the same links referenced by the earlier posts. She said that a co-worker was working another case where the 8.25 support pack still did not resolve these issues. She said that after the downgrade you still receive the ilo warnings in the logs but the reboots stop :|. I am confident that this will work as I have an identical server on SP 8.11 not having this problem.

[Don Vanco - Linux Ninja]

Downgrade to what?? There hasn't been a functional version of this software yet, has there? Or was there a stable release on Windoze?

[MarkVHDA]

This was the response from my HP engineer on wht I should downgrade and the versions I should downgrade to. I am going to make these changes in about an hour or so...

HP Engineer:

Below you will find the links in order to perform the downgrade on some HP system tools and the Ilo Management driver.

HP ProLiant iLO 2 Management Controller Driver for Windows Server 2003 x64 Editions
File name: cp008854.exe (611 KB)
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=3646081&prodNameId=3646082&swEnvOID=1113&swLang=8&taskId=135&swItem=MTX-03c99c6a52a047d0bf38df0331&mode=3
Installation:
The downloaded file is a self-extracting executable with a filename based on the Component Number.
From that drive and directory, execute the downloaded file.
Click the "Install" button to proceed with the installation. The utility will automatically be installed.

HP System Management Homepage for Windows:
File name: cp009426.exe (10 MB)
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=3646081&prodNameId=3646082&swEnvOID=1113&swLang=8&taskId=135&swItem=MTX-84b4161b7cd3455fb34ac57586&mode=3
Installation:
Download the Smart Component to a directory on your hard drive and change to that directory. The downloaded file is a self-extracting executable with a filename based on the Smart Component Number.
From that drive and directory, execute the downloaded file.
HP Insight Management Agents for Windows Server 2003/2008
File name: cp009725.exe (6.6 MB)
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?swItem=MTX-04defeca28964da4ba6ef6ac4b〈=en&cc=us&mode=3&
Installation:
Download the Smart Component to a directory on your hard drive and change to that directory. The downloaded file is a self-extracting executable with a filename based on the Smart Component Number. From that drive and directory, execute the downloaded file. Click the "Install" button to proceed with the installation. The agents will automatically be installed.

[Don Vanco - Linux Ninja]

At the risk of going all Net-nanny, I do wish the Windows folks would open a thread to cover their OS of choice. I'm following this waiting for a fix for my issue on Linux. I know it's merely another email notification, but I track many on numerous forums.
That said I can appreciate when you find _someone_ having at least a similar issue.....

But it is my guess that everyone on Linux is simply ignoring the HP components altogether - as I am.

Good luck Mark, et al on Windoze.

[Yossi Lagstein]

Hi Don

Regarding RHEL 5.s ASRs, to avoid getting ASRs out of the blue we are installing all PSP components but HP-IPMI. (bare in mind that by doing this we are loosing the value that ASR gives in the case the server hangs, but until we are sure HP has fixed this issue we don't have a choice)

Regarding windows there is dedicated thread http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1242917005955+28353475&threadId=1332008

I understand your frustration and delivered the same message to HP in a meeting last week, PSP quality is recent times is below par. However PSP is very important for our business and I'm sure for many others, so I can only hope someone in HP will take responsibility, issue fixes and overhaul the Quality Assurance of the PSP.

Yossi