DL360P Reboots Itself: Predictably Every 7 Days?

[agkyle03]

I have two HP DL360p Gen8 servers. Storage for C:\ is onboard. Storage for D:\ is iSCSI to a Dell EqualLogic; these are SQL nodes in a cluster. We know the SAN is a performance bottleneck for us but can't do anything about that right now.

Node 1 Time Between Restarts: 07d:00h:07m:03s, 07d:00h:08m:05s, 07d:00h:08m:02s, 07d:00h:09m:58s, 07d:07h:36m:56s

Node 2 Time Between Restarts: 07d:22h:22m:18s, 22d:07h:21m:14s, 07d:00h:06m:15s, 07d:00h:05m:48s, 07d:00h:10m:45s

1.) Servers reboot themselves about every 7 days for no reason. I think it's a blue screen but we're not getting any .dmp files. I've disabled auto-restart on blue screen but I've got to wait until the 7 day mark hits to see if that's it.

2.) The event logs in Windows just show "Windows recovered from a non-graceful shutdown". Nothing happens right before the reboot.

3.) There are no scheduled tasks in SQL or Task Scheduler set during these time frames; no auto-reboot tasks period.

4.) SAN shows the iSCSI drive drop off during reboot, but that's to be expected while the server reboots.

6.) I also saw POST failures in the iLO indicating Post Error 1719: A controller failure occured prior to this power-up.

7.) The machines were never installed to best practice. So yesterday and today I've installed all windows updates, firmware upgrades, driver upgrades, etc. Even the onboard storage controller wasn't installed in Windows. I think this may have been a problem because I've read some stuff where people running older drivers on the P420i were seeing random reboots. We didn't have ANY HP drivers installed outside of the standard Windows generic stuff.

8.) I have not flashed the BIOS yet as I haven't made it to the datacenter and/or I'm waiting for iLO advanced licenses to come in to flash it remotely.

9.) We're not losing power; the servers are in our datacenter. I have verified uptime on the power supplies as well.

10.) I've also failed over the SQL instance to the node that fails second to see if the primary node still fails without SQL on it.  Which I'm sure it will.

11.) No Antivirus installed to ensure that isn't the problem.

So I'm leaning towards a controller related blue screen which would explain the .dmp files missing because if the controller failed, then you can't really write to C:\. This is occurring on both machines though. But since neither machine was updated or anything, they could be suffering similar issues because, of course, best practice is to fully patch prior to installing applications which the vendor didn't do (I wasn't employed here yet, don't blame me haha)

Now that being said, with the updates/firmware I've done, it may not even fail now.  However, I want to enable as many patches/potential fixes as possible as we're about to go-live with this cluster and if I can't get it to work I'm going to have to create some SQL VM's and spin the cluster up there.  However, the preferred is to stick with these physical nodes because they are very beefy.

Below is the log of everything I've installed so far.  If anyone can think of anything else it would be much appreciated.  I'm open to any suggestion no matter how crazy it may seem.

////////////////////////////////////////////////////////////////////////////////

//  HP Smart Update Manager for Target localhost Started: Sun Jul 15 2012 00:52:12

////////////////////////////////////////////////////////////////////////////////

Installed Components:

  Component File Name: cp010892.exe

  Component Name: Headless Server Registry Update for Windows

  Original Version: None

  New Version: 1.0.0.0

  Installation Result: Success

  Component File Name: cp013225.exe

  Component Name: HP ProLiant Smart Array SAS/SATA Controller Driver for Windows Server 2008 x64 Edition

  Original Version: None

  New Version: 6.22.0.64

  Installation Result: Success, reboot required

  Component File Name: cp014123.exe

  Component Name: HP Broadcom 1Gb Driver for Windows Server 2008 x64 Editions

  Original Version: 10.100.4.0

  New Version: 15.0.0.21

  Installation Result: Success

  Component File Name: cp014722.exe

  Component Name: HP ProLiant iLO 3/4 Channel Interface Driver for Windows X64

  Original Version: None

  New Version: 3.4.0.0

  Installation Result: Success

  Component File Name: cp014360.exe

  Component Name: HP ProLiant Agentless Management Service for Windows X64

  Original Version: None

  New Version: 9.0.0.0

  Installation Result: Success

  Component File Name: cp014415.exe

  Component Name: HP ProLiant Integrated Management Log Viewer for Windows Server x64 Editions

  Original Version: None

  New Version: 6.0.0.0

  Installation Result: Success

  Component File Name: cp014432.exe

  Component Name: HP ProLiant iLO 3/4 Management Controller Driver Package for Windows Server 2008 X64

  Original Version: None

  New Version: 3.4.0.0

  Installation Result: Success, reboot required

  Component File Name: cp014436.exe

  Component Name: HP System Management Homepage for Windows x64

  Original Version: None

  New Version: 7.0.0.24

  Installation Result: Success

  Component File Name: cp014641.exe

  Component Name: HP ProLiant Smart Array SAS/SATA Event Notification Service for Windows Server 2008 x64 Editions

  Original Version: None

  New Version: 6.26.0.64

  Installation Result: Success

  Component File Name: cp014726.exe

  Component Name: HP Network Configuration Utility for Windows Server 2008 R2

  Original Version: None

  New Version: 10.50.0.0

  Installation Result: Success

  Component File Name: cp014784.exe

  Component Name: HP Version Control Agent for Windows x64

  Original Version: None

  New Version: 7.0.0.900

  Installation Result: Success

  Component File Name: cp014868.exe

  Component Name: HP ProLiant Array Configuration Utility for Windows

 Original Version: None

  New Version: 9.0.24.0

  Installation Result: Success

  Component File Name: cp014870.exe

  Component Name: HP ProLiant Array Configuration Utility (CLI) for Windows

  Original Version: None

  New Version: 9.0.24.0

  Installation Result: Success

  Component File Name: cp015005.exe

  Component Name: Combined Chipset Identifier for Windows Server 2008 R2

  Original Version: None

  New Version: 8.1.0.0

  Installation Result: Success

  Component File Name: cp015133.exe

  Component Name: HP Insight Diagnostics Online Edition for Windows x64 Editions

  Original Version: None

  New Version: 9.0.0.4179

  Installation Result: Success

  Component File Name: cp015159.exe

  Component Name: HP Lights-Out Online Configuration Utility for Windows 2003/2008 x64 Editions

  Original Version: None

  New Version: 4.0.0.0

  Installation Result: Success

  Component File Name: cp015518.exe

  Component Name: HP Storage Tape Drivers for Windows

  Original Version: None

  New Version: 3.5.0.0

  Installation Result: Success

  Component File Name: cp015976.exe

  Component Name: Matrox G200eH Video Controller Driver for Windows Server 2008 X64

  Original Version: None

  New Version: 6.12.1.1020

  Installation Result: Success, reboot required

Installed Components:

  Component File Name: cp010920.exe

  Component Name: HP ProLiant PCI-express Power Management Update for Windows

  Original Version: None

  New Version: 1.3.0.0

  Installation Result: Success, reboot required

  Component File Name: cp013459.exe

  Component Name: HP ProLiant 100-Series Management Controller Driver for Windows Server 2003/2008 x64 edition

  Original Version: None

  New Version: 1.4.0.0

  Installation Result: Not updated - already current

  Component File Name: cp015908.exe

  Component Name: HP Storage Tape Drivers for Windows

  Original Version: 3.5.0.0

  New Version: 3.6.0.0

  Installation Result: Success

  Component File Name: cp018067.exe

  Component Name: HP Broadcom 1Gb Driver for Windows Server 2008 x64 Editions

  Original Version: 15.0.0.21

  New Version: 15.4.0.17

  Installation Result: Success

  Component File Name: cp017181.exe

  Component Name: HP Network Configuration Utility for Windows Server 2008 R2

  Original Version: 10.50.0.0

  New Version: 10.65.0.0

  Installation Result: Success

  Component File Name: cp017430.exe

  Component Name: HP ProLiant Array Configuration Utility for Windows

  Original Version: None

  New Version: 9.30.15.0

  Installation Result: Success

  Component File Name: cp017432.exe

  Component Name: HP ProLiant Array Configuration Utility (CLI) for Windows

  Original Version: None

  New Version: 9.30.15.0

  Installation Result: Success

  Component File Name: cp017597.exe

  Component Name: Matrox G200eH Video Controller Driver for Windows Server 2008 X64

  Original Version: 6.12.1.1020

  New Version: 6.12.1.1030

  Installation Result: Success, reboot required

  Component File Name: cp017871.exe

  Component Name: PFA Server Registry Update for Windows

  Original Version: None

  New Version: 1.0.0.0

  Installation Result: Success, reboot required

  Component File Name: cp017930.exe

  Component Name: HP ProLiant iLO 3/4 Channel Interface Driver for Windows X64

  Original Version: 3.4.0.0

  New Version: 3.7.0.0

  Installation Result: Success, reboot required

  Component File Name: cp017974.exe

  Component Name: HP ProLiant iLO 3/4 Management Controller Driver Package for Windows Server 2008/2012 X64

  Original Version: 3.4.0.0

  New Version: 3.7.0.0

  Installation Result: Success, reboot required

  Component File Name: cp018001.exe

  Component Name: Online ROM Flash Component for Windows - HP ProLiant DL360p Gen8 (P71) Servers

  Original Version: 2012.07.15

  New Version: 2012.08.20

  Installation Result: Success, reboot required

  Component File Name: cp018024.exe

  Component Name: HP ProLiant Agentless Management Service for Windows X64

  Original Version: 9.0.0.0

  New Version: 9.25.0.0

  Installation Result: Success

  Component File Name: cp018176.exe

  Component Name: HP Lights-Out Online Configuration Utility for Windows 2003/2008 x64 Editions

  Original Version: 4.0.0.0

  New Version: 4.0.1.0

  Installation Result: Success

  Component File Name: cp018352.exe

  Component Name: HP Version Control Agent for Windows x64

  Original Version: 7.0.0.900

  New Version: 7.1.2.0

  Installation Result: Success

  Component File Name: cp018398.exe

  Component Name: HP ProLiant Integrated Management Log Viewer for Windows Server x64 Editions

  Original Version: 6.0.0.0

  New Version: 6.3.0.0

  Installation Result: Success

  Component File Name: cp018453.exe

  Component Name: HP Insight Diagnostics Online Edition for Windows x64 Editions

  Original Version: 9.0.0.4179

  New Version: 9.3.0.4614

  Installation Result: Success

  Component File Name: cp018517.exe

  Component Name: HP ProLiant Smart Array SAS/SATA Controller Driver for Windows Server 2008 x64 Edition

  Original Version: 6.22.0.64

  New Version: 6.24.0.64

  Installation Result: Success, reboot required

  Component File Name: cp018574.exe

  Component Name: HP ProLiant Smart Array SAS/SATA Event Notification Service for Windows Server 2008 x64 Editions

  Original Version: 6.26.0.64

  New Version: 6.30.0.64

  Installation Result: Success

  Component File Name: cp018579.exe

  Component Name: Online ROM Flash Component for Windows - MM0500GBKAK and MM1000GBKAL Drives

  Original Version: HPGB

  New Version: HPGC

  Installation Result: Not updated - already current

  Component File Name: cp018601.exe

  Component Name: HP System Management Homepage for Windows x64

  Original Version: 7.0.0.24

  New Version: 7.1.2.3

  Installation Result: Success

  Component File Name: cp018726.exe

  Component Name: Online ROM Flash Component for Windows - Smart Array P220i, P222, P420i, P420, P421, P721m, and P822

  Original Version: 3.04

  New Version: 3.22

  Installation Result: Success, reboot required

Installed Components:

  Component File Name: cp013459.exe

  Component Name: HP ProLiant 100-Series Management Controller Driver for Windows Server 2003/2008 x64 edition

  Original Version: None

  New Version: 1.4.0.0

  Installation Result: Success, reboot required

  Component File Name: cp017430.exe

  Component Name: HP ProLiant Array Configuration Utility for Windows

  Original Version: None

  New Version: 9.30.15.0

  Installation Result: Not updated - already current

  Component File Name: cp017432.exe

  Component Name: HP ProLiant Array Configuration Utility (CLI) for Windows

  Original Version: None

  New Version: 9.30.15.0

  Installation Result: Success

  Component File Name: cp018001.exe

  Component Name: Online ROM Flash Component for Windows - HP ProLiant DL360p Gen8 (P71) Servers

  Original Version: 2012.07.15

  New Version: 2012.08.20

  Installation Result: Success

  Component File Name: cp018579.exe

  Component Name: Online ROM Flash Component for Windows - MM0500GBKAK and MM1000GBKAL Drives

  Original Version: HPGB

  New Version: HPGC

  Installation Result: Success

 Component File Name: cp018726.exe

  Component Name: Online ROM Flash Component for Windows - Smart Array P220i, P222, P420i, P420, P421, P721m, and P822

  Original Version: 3.04

  New Version: 3.22

  Installation Result: Success

////////////////////////////////////////////////////////////////////////////////

//  Exit Code for Target localhost: 1

//  HP Smart Update Manager for Target localhost Finished: Wed Dec 12 2012 13:21:29

////////////////////////////////////////////////////////////////////////////////

////////////////////////////////////////////////////////////////////////////////

//  HP Smart Update Manager for Target localhost Started: Wed Dec 12 2012 13:36:25

////////////////////////////////////////////////////////////////////////////////

 

[Johan Guldmyr]

Is there perhaps a weekly job on the SAN? (clone, backup, something)

[agkyle03]

That's a negative.  After we upgraded we still got the failure.  Blue screen indicates an F4 BSOD.  Error Post 1719: controller failure occurred prior to this power up, and now the event viewer correctly shows HP errors and one stating: Array controller P420i has reported that it previously locked up with code 0.

Have HP on the phone now.  According to this: http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL-SL/Absolute-nightmare-of-a-DL380-G7/td-p/4709685 it appears others are having issues with this controller as well.

[agkyle03]

For anyone reading this, it was a problem with the cache.  We replaced the battery backed cache and the flash backed cache and have gone for 15 days without an unexpected reboot.

[Uepapx]

Hello colleagues on the issue! Sorry for my English, Google translator. Solution! But with a few caveats) In my case it was purchased two ProLiant DL380 G7 server with 8 disks. One of them was constantly reboots and error "POST Error: 1719 - A controller failure event occurred prior to this power-up". After replacing the motherboard with a new one, he worked for a year and started all over again. Helped people from the HP support in Russian by the name of: Andrey Kurakin Technytsal Solution Tsonsultant GSD Customer SolutionsCenter Russia. For which separate him thanks. The reason was this: her mother cache memory unit BBU has failed. After viewing the log ACU errors are found in the module itself. Tomorrow he will be with me. We make the change and accomplish your goal after a while!