HP ML350G6 Abnormal Program Termination

[JoeFR]

We have two identical servers HP ML350G6 ( 2x X5560, 32GB Ram, Smart Aray 410i 512MB BBWC,8x 1TB Hard Drives )
Servers are runing MS Windows 2008 R2 OS, and both servers are runing HyperV and 4 virtual servers.

We have constant problem with unexpected rebooting on one server.

Here is the Integrated Management Log:
Critical OS 04/26/2010 02:27 04/26/2010 02:27 1 Abnormal Program Termination (BugCheck, STOP: 0x00000101 (0x000000000000000D, 0x0000000000000000, 0xFFFFF88002200180, 0x000000000000000A))
Informational System Revision 04/24/2010 13:34 04/24/2010 13:34 1 ROM flashed (New version: 03/30/2010)
Critical OS 04/15/2010 01:13 04/15/2010 01:13 1 Abnormal Program Termination (BugCheck, STOP: 0x00000101 (0x000000000000000D, 0x0000000000000000, 0xFFFFF88002200180, 0x000000000000000A))
Critical OS 04/13/2010 05:30 04/13/2010 05:30 1 Abnormal Program Termination (BugCheck, STOP: 0x0000000A (0x0000000000000004, 0x0000000000000002, 0x0000000000000001, 0xFFFFF800028E4AC1))
Critical OS 02/19/2010 05:17 02/19/2010 05:17 1 Abnormal Program Termination (BugCheck, STOP: 0x00000101 (0x000000000000000D, 0x0000000000000000, 0xFFFFF880022E2180, 0x000000000000000C))
Critical OS 02/08/2010 05:44 02/08/2010 05:44 1 Abnormal Program Termination (BugCheck, STOP: 0x00000101 (0x000000000000000D, 0x0000000000000000, 0xFFFFF88002200180, 0x000000000000000A))
Critical OS 01/25/2010 13:44 01/25/2010 13:44 1 Abnormal Program Termination (BugCheck, STOP: 0x00000101 (0x000000000000000D, 0x0000000000000000, 0xFFFFF880023C4180, 0x000000000000000E))
Critical OS 01/18/2010 23:53 01/18/2010 23:53 1 Abnormal Program Termination (BugCheck, STOP: 0x00000101 (0x000000000000000D, 0x0000000000000000, 0xFFFFF88002200180, 0x000000000000000A))
Informational System Revision 12/02/2009 20:54 12/02/2009 20:54 1 ROM flashed (New version: 10/02/2009)
Critical OS 11/18/2009 13:00 11/18/2009 13:00 1 Abnormal Program Termination (BugCheck, STOP: 0x0000007A (0xFFFFF6FC40049190, 0xFFFFFFFFC000000E, 0x000000046165CBE0, 0xFFFFF88009232000))

And this is the Windows log when server boot up:
Problem signature:
Problem Event Name: BlueScreen
OS Version: 6.1.7600.2.0.0.274.10
Locale ID: 1060

Additional information about the problem:
BCCode: 101
BCP1: 000000000000000D
BCP2: 0000000000000000
BCP3: FFFFF88002200180
BCP4: 000000000000000A
OS Version: 6_1_7600
Service Pack: 0_0
Product: 274_3
------------------------------
On 24.04.2010 we made fresh install of MS Windows 2008 R2 on problematic server with the latest HP SmartStart 8.40 and updated the firmware and drivers with the latest HP Smart Update DVD V 9.0.

Two days later the server rebooted again.

Do you have any suggestion how to solve this problem?

Best Regards Joze

[marcus1234]

hmm run offline diagnostics with latest smart start cd what happens now.

[cnb · Accepted solution]

Joze,

Do they both have the same BIOS and CPU versions, & are they set exactly the same way in BIOS?

Sounds like this issue?

See this thread -

http://blogs.msdn.com/virtual_pc_guy/archive/2009/10/16/hyper-v-hotfix-for-0x00000101-clock-watchdog-timeout-on-nehalem-systems.aspx

Rgds,

[JoeFR]

The CPU on both servers are the same, the bios settings are the same. Both servers are identical HP ML350G6 CTO models purchased on the same day.

Working server has bios version D22 06/20/2009 and iLO version 1.78 06/10/2009.

On the problematic server I made 3 bios updates and the server is still rebooting.

I will run offline diagnostic on saturday because it is a production server and I cant take it offline during working days.

Then I will try to aply the fix you specify.

Best Regards Joze

[marcus1234]

hmm any event id 57 in windows log..
which is related to ilo driver and ilo system management driver.

post offline diagnostics here , run it in advanced survey mode

also run array diagnostics utility and post log here

ill cast an eye on it ..

[JoeFR]

This is the offline diagnostic test log

Best Regards Joze

[JoeFR]

This is the Smart Array P410i diag Report

Best Regards Joze

[JoeFR]

This is the complete configuration log

Best Regards Joze

[marcus1234]

hmm Joe all looks reasonable except for hard bus faults , check cables are secure on disk cage backplane ,

appears disc cage backplane , has issues

could be cable or the cage backplane in this instance from viewing logs

i would ensure cables are reseated if issue persists , i would sugest new disc cage backplane

goodluck :)

[JoeFR]

Thanks mark1234

I will reseated all cables on the backplane and cotroller and we will see if problem is gone.

Best Regards Joze

[Aarboard]

We too have a ML350G6 with 2008R2 hyper-v installed, and it also reboots at random (About every 1-2 days)

We see such errors in iml
BugCheck, STOP: 0x00000101

We also noticed that in the VM the clock is frequently put forward by about seconds (Sometimes killing all terminalserver sessions)

We did install the MS hotfix yesterday evening, today arround 12:50 the server was once more rebooting.
I did now install the latest bios from 30/04/2010 (Had the one from january 2010) and see if it helps.

Otherwise we will turn off some of the power management settings in the bios (But so far we don't know whih ones might help)

André

[Naures]

Hi, At several of our cutomers we also get this same issue! is there a solution found for this problem?

Grtz...

Naures

[JoeFR]

The solution for our case was, that our service provider change the motherboard and disc cage backplanes.

From that time our HP ML350G6 server works without random reboots.

Best Regards Joze