BSOD on BL460c

[karim h]

Terai,
Thanks for the headsup. Have you tried running driver verifier against just storport.sys and ql2300.sys?

I think that HP MPIO driver (hpeaadms.sys) can be ruled out as both you and I are running EMC back end arrays and are having issues, but then again, who knows?

I don't believe that hpciss2.sys is causing the issue as some of my servers do not even have this driver and they still crash with the same symptoms.

I'm currently running 2 baseline configs (each baseline is applied to 17 blades - BL460c's and BL680c's):

BASELINE 1.) (W2003 X64 R2 SP2) PSP8.10 - FW Maintenance 8.20 with storport.sys 5.2.3790.4173 and ql2300.sys v9.1.7.16
-- This is unstable

BASELINE 2.) (W2003 X64 R2 SP2) PSP8.15 - FW Maintenance 8.30 with storport.sys 5.2.3790.4303 and ql2300.sys v9.1.7.17
-- Rolled out 6 days ago, no BSODs as yet but too early to tell.

As i mentioned a few emails ago, my dumps indicate that cqmgstor.exe (Insight Storage Agent) may be causing the issue...

[ACHCHGUY]

I have applied the below registry to some of our bl 460c as I was having BSOD that could not be pinned down.
After applying the issue has not happened again on any of these Servers.

These servers and sans were all at latest revisions (as you do) but would still BSOD.
Cheers
Steve

Did you see the msdn article and try ProtectNonPagedPool registry?
http://msdn.microsoft.com/en-us/library/ms796128.aspx
Please be cautious to set the registry, it might increase the possibility of bsod but might help you to identify the root cause.

[karim h]

Update -

I have not had any problems since applying PSP 8.15, FW 8.30 and the updated storport driver I mentioned above. If you haven't already, I suggest that you try this as a course of action.

- 17 blades stable for 3 weeks (previously seeing incidents every few days)


Karim

[Melissa O'Brien]

Hi Karim,
Just a warning - our blue screens went away after updating all firmware and drivers to version 8.1 in August. We then upgraded our storport.sys as a prereq for the newer version of Veritas, and the blue screens came back. I've tried newer storports.

I deployed 8.15 and some are still blue screening. It's almost as if it's the order of installs that matter.

Very frustrating.

[Robert Walker_8]

Well I'd like to report that I have been having a frustrating time since March'08 when we converted from DL360's to BL460c's.

My citrix farm has 16 of these which all but one fails - the only one is a 2003 x64 server! The 2003 32bit servers die like flies with veritable smorgasbord of codes! I count 108 BSODs, 29 ASRs since March 08. The BSODs are nasty - with ASR switched off they just stick and require manual intervention to restart the server.

23/01/2009 5:23 Blue Screen Trap (BugCheck, STOP: 0x0000007E (0xC0000005, 0x808402FE, 0xF707AB28, 0xF707A824))
22/01/2009 9:18 Blue Screen Trap (BugCheck, STOP: 0x00000019 (0x00000020, 0x86546000, 0x86546818, 0x1B030000))
1/01/2009 1:01 Blue Screen Trap (BugCheck, STOP: 0x000000C5 (0x00000000, 0xD0000002, 0x00000001, 0x8089C4BB))
30/11/2008 1:13 Blue Screen Trap (BugCheck, STOP: 0x00000024 (0x0019033D, 0x8DD44514, 0x8DD44210, 0x808402FE))
25/11/2008 4:42 Blue Screen Trap (BugCheck, STOP: 0x000000E1 (0x8082BF6F, 0x0000001F, 0x8A36BD50, 0x8A36BD50))
25/11/2008 1:51 Blue Screen Trap (BugCheck, STOP: 0x000000D0 (0x00000000, 0xD0000002, 0x00000001, 0x8089968D))
22/11/2008 17:43 Blue Screen Trap (BugCheck, STOP: 0x000000D1 (0x00000004, 0xD0000002, 0x00000001, 0xBA2022DF))
17/09/2008 8:01 Blue Screen Trap (BugCheck, STOP: 0x0000008E (0xC0000005, 0x8092AE38, 0x9A140C94, 0x00000000))
7/09/2008 2:01 Blue Screen Trap (BugCheck, STOP: 0x000000C2 (0x00000007, 0x0000121A, 0x00000000, 0x83000168))
19/08/2008 6:05 Blue Screen Trap (BugCheck, STOP: 0x0000001A (0x00000401, 0xC01D33AC, 0x8002B867, 0x00000000))
29/07/2008 6:34 Blue Screen Trap (BugCheck, STOP: 0x000000BE (0xD22C4000, 0x8002B121, 0xF792E5B8, 0x0000000B))
1/07/2008 9:33 Blue Screen Trap (BugCheck, STOP: 0x0000004E (0x00000007, 0x00080010, 0xFFFFFFFF, 0x00000000))
22/05/2008 15:43 Blue Screen Trap (BugCheck, STOP: 0x0000000A (0xFFFFFFB8, 0xD000001B, 0x00000000, 0x808541D3))

I have updated to PSP 8.15a, FW 8.3 and STORPORT.SYS 4368. Have also got driver verifier running:

C:\> verifier /querysettings
Special pool: Enabled
Force IRQL checking: Disabled
Low resources simulation: Disabled
Pool tracking: Enabled
I/O verification: Disabled
Deadlock detection: Disabled
Enhanced I/O verification: Disabled
DMA checking: Disabled
Disk integrity checking: Disabled

Verified drivers:

hpcisss2.sys
storport.sys
videoprt.sys
cpqcidrv.sys
hpqilo2.sys
cpqteam.sys
ptilink.sys
icacdd.sys
tmtdi.sys
cdfdrv.sys
dump_diskdump.sys
dump_hpcisss2.sys
ctxpidmn.sys
ctxsbx.sys
tmpreflt.sys
vsapint.sys
tmxpflt.sys
ctxaltstr.sys
cdm.sys
ctxsmcdrv.sys
tmcomm.sys
wdica.sys
icareduc.sys
pdrframe.sys
pdcrypt1.sys
vdtw30.dll

HP, Microsoft and Citrix have all been looking at it with much finger pointing and shrugged shoulders.

Robert.

[CCaldwell_1]

I just got done with a call to HP. Apparently this is now a known issue. I was told this is a driver conflict between the storport.sys and HpCISSs.sys drivers. Microsoft has a hotfix for this issue. Of course it's a hotfix that you have to request. Here is a link to the Microsoft page:

http://support.microsoft.com/kb/912944/en-us

Just a little background info, I was experiencing the exact same issue. However, I was having this problem with a DL180 G5 server. I had 11 BSODs in the past two and a half months. All instances occurred while the server was being backed up.

Anyway, I hope this info helps someone because it didn't really help me much since the hotfix is specifically for Server 2003 and I was having this problem on a 2008 server.

[Melissa O'Brien]

Thanks CCaldwell. This is an old version of Storport.sys and I've tried about 4 versions since and it didn't solve my problem! I do believe the issue is related to storport, seeing as the problem started when we first upgraded our storport.sys.

[Robert Walker_8]

Gday,

Microsoft got back to me a said its a definite HP issue with their HPCISSS2.SYS driver so went back to HP and they got back to me with this URL:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1842750&swItem=MTX-7c14f90c69b2478fbf362337e2&prodNameId=3288156&swEnvOID=1005&swLang=8&taskId=135&mode=5

HP ProLiant Smart Array SAS/SATA Controller Driver for Windows Server 2003
Type: Driver - Storage Controller
Version: 6.14.0.32 (22 Jan 2009)
Operating System(s): Microsoft Windows Server 2003
File name: cp008740.exe (552 KB)

I havent tried it yet but will be doing so in the next few days.

If anyone has already tried this - it would be good to know.

Robert.

[Melissa O'Brien]

Thanks Robert. I will try this latest version to see if it makes a difference. I doubt it though.

[MR.Bakken]

Hi,

We are currently having the same issue.
The configuration are:
- BL 460c
- C7000
- SAN passthrough
- Hitachi SAN
- LAN passthrough
- Emulex
- Broadcom
- Veritas Storagefoundation

- Windows 2003 R2 SP2
- Storport.sys - 5.1.3790.4163
- hpciss2.sys - 6.12.4.32
- cludisk.sys - 5.2.3790.3959
- HP NC10xx/NC67xx/NC77xx Gigabit Driver NDIS5.1 Driver. 10.85.0.0
- Network Teaming Intermediate Driver (NTID), 9.36.00.00, NDIS 5.1/5.2, x86, (free) on Win2K3 9.36.0.0
- HP Insight Management Agents 8.15.0.0
- HP ProLiant Array Diagnostics Utility 8.10.3.0
- HP ProLiant Array Configuration Utility (CLI) 8.10.2.0
- HP ProLiant Array Configuration Utility 8.10.1.0
- HP Insight Diagnostics Online Edition 8.1.5.2890
- ATI Radeon WindowsNT Miniport Driver 6.14.10.6744
- Smart Array SAS/SATA Controller Storport Driver 6.13.0.32
- HP ProLiant Smart Array SAS/SATA Event Notification Service 6.10.0.32
- HP ProLiant Integrated Management Log Viewer 5.21.0.0
- HP ProLiant Remote Monitor Service 5.20.0.0
- Broadcom NetXtreme II GigE VBD 4.6.17.0
- HP System Management Homepage 2.1.15.210
- HP Version Control Agent 2.1.10.801
- HP Lights-Out Online Configuration Utility 2.0.4.1
- HP iLO Management Channel Interface Driver 1.13.0.0
- HP StorageWorks Fibre Channel Adapter Kit for Emulex Storport Driver 1.10.1.1
- HP ProLiant iLO 2 Management Controller Driver 1.8.0.0
- HP Protect Your Data Information and Application Downloads 1.0.3.0
- HP High Precision Event Timer NULL INF 1.0.1.0
- HP NULL IPMI Controller Driver 1.0.0.0
- Compaq System ROM 2008.09.29
- Compaq Remote Insight ROM 1.61
- HP Smart Array E200i Controller 1.80

BSOD happends now and then, and in the last week 3 of four has been BSOD.

Somebody fixed this issue?