1748150 Members
3452 Online
108758 Solutions
New Discussion юеВ

Re: BSOD on BL460c

 
Terai
Occasional Advisor

BSOD on BL460c

Hello,

I've encountered BSOD's for many times since March. With support from Microsoft and hp, I've been tackling with this problem for more than 6 months but BSOD still happens.

The situation is, BSOD happens in various timings, like 3 times in a week or nothing in a month. I've turned Driver Verifier on and have kept monitoring for a while but BSOD had never happened during that period for some reason.

Stop Codes vary from 0x0a, 0x8e or 0xD0. 0xD0 happened most (5 times for 3 servers for a month!). Microsoft and hp have checked the momory dumps for all BSOD and almost all the BSODs were caused by memory corruption (overwritten by nil) but so far they've not been able to pin down which process currupted the memory.

The servers details are as follows.

Hardware
- ProLiant BL460c's x 3 in a C7000 enclosure
- connected to EVA 4100 by Qlogic's QMH2462 4Gb FC HBAs
- NC325m 4-ports Gb network adapter

Software
- Windows 2003 Server Enterprise R2 SP1(these servers are used as just Windows file servers.)
- SystemRomя╝ИBIOS)уААуАА2008/2/29
- SmartArrayE200i 1.72
- SAS HDD HPD9
- QHM2462 1.26
- NC373i 1.9.6
- NC325m 3.28
- HP PSP 8.00
- Qlogic 9.1.3.18 for HPуАА
- StorPort.sys 5.2.3790.3030уАА
- Clusdisk.sys 5.2:3790.2938

Configuration
- MSCS is working with 3 nodes failover ring cluster configuration.

Usage
- 300 client connections per server
- 1.5% average CPU usage

If someone have had the similar problems and successfully fixed them, please give me advise.
Thanks in advance.
64 REPLIES 64
Steven Clementi
Honored Contributor

Re: BSOD on BL460c

Have you checked the integrated Management Log? The iLo logs?

What version of MPIO are you using?

What happened in March, when this first started? Any upgrades? Drivers? firmware?

What version of XCS are you running on the EVA? What about the firmware on your SAn Switches?


Steven
Steven Clementi
HP Master ASE, Storage, Servers, and Clustering
MCSE (NT 4.0, W2K, W2K3)
VCP (ESX2, Vi3, vSphere4, vSphere5, vSphere 6.x)
RHCE
NPP3 (Nutanix Platform Professional)
Terai
Occasional Advisor

Re: BSOD on BL460c

Hello, Steven,

Thanks for your response. Pls find my answers below.

>>Have you checked the integrated Management Log? The iLo logs?

When BSOD heppned, only Blue Screen Trap was recorded like this;
"Blue Screen Trap (BugCheck,STOP: 0x000000D0 (0x00000008,0x00000002,0x00000000,0xE089AF77))"

There was no iLo log.

>>What version of MPIO are you using?
MPIO : 3.00.00

>>What version of XCS are you running on the EVA? What about the firmware on your SAn Switches?

XCS : 6.110
SANswitch : 5.3.0.d

Thanks.
Terai
Occasional Advisor

Re: BSOD on BL460c

Steven,

I forgot to answer one thing.

>>What happened in March, when this first started? Any upgrades? Drivers? firmware?

March is when these servers were initially installed. Since then BSOD happened as follows.

March: Once (0xD0)
Apr & May: none
June: 5 (all 0xD0)
July: twice (0x0a & 0x8e)
Aug: none
Sep: upgraded to Win2K3 SP2 but BSOD happened (0x0a)

thanks.
Angelina
New Member

Re: BSOD on BL460c

Hi,

0xD0 is the typical stop code of pool
corruption caused mostly by bad driver.

Please let me know couple of things.

Have you ever done memory dump analysis?
Post the analysis report if so.

Did you install any 3rd party software especially which contains filter drivers.
Symntec products or trend micro's are the exmaples.

What was the system's load when you got that problem?

Did you see the msdn article and try ProtectNonPagedPool registry?
http://msdn.microsoft.com/en-us/library/ms796128.aspx
Please be cautious to set the registry, it might increase the possibility of bsod but might help you to identify the root cause.

Terai
Occasional Advisor

Re: BSOD on BL460c

Hello, Angelina,

Thank you for your comment.
As to your questions, please find the below.

Q1. Have you ever done memory dump analysis?
Post the analysis report if so.
-> Find the attached file. Pls note the stop error code of this case was 0x8E, not 0xD0.

Q2. Did you install any 3rd party software especially which contains filter drivers.
Symntec products or trend micro's are the exmaples.
-> Symantec's Backup Exec agent is installed on each server.

Q3. What was the system's load when you got that problem?
-> The system's load was not particularly high.

Q4. Did you see the msdn article and try ProtectNonPagedPool registry?
-> Discussing this point with our Microsoft and hp consultants.

Regards,
O. Terai
Blade user
Occasional Advisor

Re: BSOD on BL460c

Hi

We've also just moved across to the C7000 enclosure with Bl460c's and are experiencing the same issue.
We are using a Hitachi SAN Array instead of an EVA, but apart from that everything else is pretty much the same.
Looking at applying the ProtectNonPagedPool key , but sounds like it may be something common to the BL460c Blade?

Any help would be greatly appreciated.

Blazhev_1
Honored Contributor

Re: BSOD on BL460c

check the memory dump, most probably it is the multifunction NIc driver...
Eric Gazrighian
New Member

Re: BSOD on BL460c

I also encouter BSOD and ASR on numerous ProLiant BL460c G1 since octobre 2008 like Blue Screen Trap (BugCheck, STOP: 0x000000D0 (0x00000008, 0xD0000002, 0x00000000, 0xE0899F77)) in different Rack using the last Rack firmware, lame firmware and 8.10 Proliant support Pack. Can not find any valid reason for the moment.

Management Processor Firmware (Active) 1.60
Server Blade Enclosure Firmware 2.25
System ROM Firmware-I15 (Active) 2008.06.25
System ROM Firmware-I15 (Redundant) 2008.01.24
HP NC373i Multifunction Gigabit Server Adapter 1.9.6
HP NC373i Multifunction Gigabit Server Adapter #2 1.9.6
HP NC373i Multifunction Gigabit Server Adapter 1.1.3
HP NC373i Multifunction Gigabit Server Adapter #2 1.1.3
Disk Drive Firmware HPDA
Disk Drive Firmware HPDA
Storage Enclosure Processor Firmware N/A
Storage Enclosure Processor Firmware N/A
Array Controller Firmware 1.72


karim h
Valued Contributor

Re: BSOD on BL460c

Check this bulletin on smartarray drivers and storport.sys


http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01068337&dimid=1489071792&dicid=alr_may07&jumpid=em_alerts/us/may07/all/xbu/emailsubid/mrm/mcc/loc/rbu_category/alerts

***PROBLEM***
Affected Software Configuration:

- Any Edition of Microsoft Windows Server 2003 (x86 or x64).
AND

- HP ProLiant Smart Array 5x/6x Controller Driver (HPCISSS.SYS) Version 5.18.0.64 (or earlier) OR HP ProLiant Smart Array SAS/SATA Controller Driver (HPCISSS2.SYS) Version 5.10.0.32 or 5.10.0.64 (or earlier).
AND

- Microsoft Storport Driver for Windows Server 2003 Version 5.2.3790.2880 (for SP1) OR 5.2.3790.4021 (for SP2) from Microsoft KB932755.
AND

- HP Insight Management Storage Agents (any version).

***RESOLUTION***
The blue screen event has been corrected in the following updates:

For ProLiant Servers Running 64-bit Versions of Windows Server 2003:

(HPCISSS.SYS) HP ProLiant Smart Array 5x and 6x Controller Driver for Windows Server 2003 x64 Editions Version 6.4.0.64 (or later)

(HPCISSS2.SYS) HP ProLiant Smart Array SAS/SATA Controller Driver for Windows Server 2003 x64 Editions Version 6.2.0.64 (or later)

For ProLiant Servers Running 32-bit Versions of Windows Server 2003:

(HPCISSS2.SYS) HP ProLiant Smart Array SAS/SATA Controller Driver for Windows Server 2003 Version 6.2.0.32 (or later)