STOP error after PSP 7.8

[Joshua Small_2]

Be aware if you simply uninstall the Management Agents and don't install any other version, you won't be able to manage and monitor your hardware. For most servers, this is also not a good position to be in.

[Jesse Zellmer]

Joshua makes a good point.

I installed (after uninstalling 7.8A), PSP 7.7A and have not had a BSOD in over 17 days. I have closed my case with Microsoft, since they keep wanting more memory dumps, and if my servers are no longer BSOD, I cannot provide them with one.

I am signed up for HP ProLiant Advisories, so hopefully soon they will come out with a fix for whatever the 7.8A and Windows 2003 SP2 combo is causing.

[Arian van der Pijl]

Should this be the BSOD fix from hp?
HP Smart Array Driver Update -> v6.6.2.32;

'Updated the driver to properly handle shutdown events. On certain HP ProLiant servers configured with an HP Smart Array SAS/SATA controller (Driver version 6.6.0.x or earlier) as the boot controller, the system may blue screen or become unresponsive during shutdown after upgrading to Windows 2003 SP2 or applying Storport QFE (KB932755).'

http://h18007.www1.hp.com/support/files/storage/us/download/27348.html

[Arian van der Pijl]

Unfortunately this update does not resolve the BSOD when shutting down or rebooting the server. At least not on my server (DL380G5).

To my knowledge there are 2 seperate issues with PSP 7.80?
- BSOD with the latest nic drivers on reboot/shutdown.
('Resolved' by degrading nic drivers)
- 'Random' BSOD G5 machines while in production.

Maybe problem nr2 could be solved by this update?

[Joshua Small_2]

It does stand to reason that the Insight Agents cause the system to crash whilst interacting with a bad driver.
To anyone for whom the agents were the cause of the issue, I would be interested in seeing how this goes for them.

Although I would expect any of the people with a case open at the moment would have been alerted to this new driver, so who knows?

Unfortunately, my issue still seems to be the NIC drivers, so I'm still awaiting an update. HP _have_ acknowledged that these drivers cause an issue on reboot on shutdown, so I can't understand why we're still waiting on a fix.

[Joshua Small_2]

Being involved in this thread, with the amount of posts made, must have gained you guys a hell of a lot of points :)

[Bruce Holzrichter]

This is the fix for the random BSOD issue:
http://h18023.www1.hp.com/support/files/server/us/download/27349.html

I tested this in Beta form first with HP and confirmed it fixed our dl585 g2's from blue screening with the 7.8 Insight Management Agents.
The version of the driver is 6.6.2.64

I had not been back to this thread in a while, but this works on our systems, hopefully will work for some of you too. I do not know about the shutdown issue though, as we never saw it..

Good luck!

[Heather Christenson]

I am having this issue as well with 2 servers running that are DL380 G5. OS is Windows 2003 R2 with Service Pack 2 x64.

The servers are running PSP 7.8. Every time I run secure copy jobs of data which is NIC intensive the servers keep blue screening. These are file share servers with FSRM installed as well.

I cancelled the implementation of these servers into production since the BSODs were happening without an explanation when copying data over.

I had downgraded to the PSP 7.7 with force, but in looking at the blog I may need to do this differently. I was still seeing the issue. I am working on the issue once again and wondering what the best solution is for the problem. I have more servers of similar configuation that are in the queue to be implemented and I need a solution ASAP to move forward. Any advise since you have been tackling the multiple troubleshooting efforts. Thanks for any advise you can give.

[Heather Christenson]

Our BSODs are random as well while data is copying. The servers are not in production so traffic is not consistently hitting it yet. The last patch published in the blog was to fix shutdown events, but our problem is during processing data.

[DennisCThornton]

You may have to go into add/remove rrograms to fully remove 7.8 (we did). Bruce's post above may be the fix for the random reboots (appears to be NIC driver related).

[Joshua Small_2]

Hi Heather,

You have it the wrong way around. The latest update, the post PSP 7.8 SATA driver, is there to fix random BSODs. There is presently no fix for the shutdown event issue.

I am currently running PSP 7.8 with the updated SATA driver and the downgraded NIC, and appear to be fully operational.

I have yet to feel game enough to try putting my network team back together however. I don't have a noncritical server to test it on.

[Joshua Small_2]

Hi Bruce,

I'm interested in how you ever got a hold of a beta driver.
In our multiple discussions with HP Carepaq support all we ever got told to do was to search the hp.com website ourselves for the latest.

When I was told there was a more recent version to the one I had found I asked to be emailed a link, and was told this was not possible.

[Arian van der Pijl]

About the BSOD on reboot / Network issue:

Broadcom has released new public drivers;
http://www.broadcom.com/support/ethernet_nic/netxtremeii.php

I extracted and manually updated the drivers on a DL380G5 and had no BSOD while rebooting / shutting down.

Nic Driver - 3.4.10.0
Virtual Bus - 3.4.10.0

Unfortunately the HP Nic / Teaming tool (7.80) does not show the network cards when using these non HP drivers.

Guess it is waiting on HP branded drivers.

[Chris Young_6]

All,

I am running PSP 7.7, and using HP Teaming successfully after diabling RSS in the registry.

HKLM\System\CCS\Services\TCPIP\Parameters\EnableRSS dword value = 0.

DL380 G5, W2K3 x64 /SP2, now in production with hardware load balanced SQL Server 2005 x64 /SP2.

I have not experienced any BSODs since removing PSP 7.8.

Prior to disabling RSS, I did have issues with netlogon service failures.

[Markus Kolbeck]

Hi everyone,

here's our situation:
- 2x ProLiant DL 380 G5
- both connected to SAN (EMC DMX3)
- MS File Cluster Server (A/P)
- Windows Server 2003 R2 SP1 (x86)

- PSP 7.80
- HP Management Agents, HP OVO ...
- TrendMicro ServerProtect 5.58
- Symantec StorageExec 5.5

First we encountered numerous BSOD (like the ones decribed above) over a couple of weeks.

We applied cp007813.exe (MS QFE 932755 was already installed), removed HP OVO, removed StorageExec, and everything was fine.
Seemed to be related to the storport.sys driver.
Both cluster nodes didn't have any BSOD since then.

Yesterday, we installed both StorageExec and HP OVO Agents on one cluster node.

Since then, every time heavy load occurs either from the backup software (Legato) or from network access (user profiles and home shares are on the cluster server) one of the cluster nodes reports the following eventlog warning:

Event ID: 2021
Source: SRV
The server was unable to allocate a work item times in the last seconds.

Some time later network access is impossible. Cluster resources become unavailable, users complain, ... the full monty.

No BSOD anymore, though.

The other node does not report any srv warnings after switching the resources. On this server everything works fine.

Then, we removed StorageExec. To no avail.
When switching to the faulty node back again the same warnings occur and some time later no one can access the server anymore (accompanied by numerous cluster errors, of course).

The only difference among the cluster nodes I found: HP OVO.

Today, we removed it, rebooted and the cluster resources will be switched tonight back again to the faulty node.
I'll let you know the results.

The problem is: we need HP OVO.

Has anyone similar problems (2021 warnings) or "just" BSOD?
Seems to be related to HW drivers as well ...

ATB
Markus

[DennisCThornton]

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=115&prodSeriesId=1121486&prodTypeId=15351&objectID=c01102958

[Markus Kolbeck]

Hi all,

we solved our Problem (the 2021 Warning) by changing the IRPStackSize Registry Key.

On both cluster nodes they were configured differently and the faulty one had a value of 33. Together with the DL380-CPUs that caused the problem.

http://support.microsoft.com/kb/924749/en-us

ATB
Markus

[Jesse Zellmer]

Well, what a depressing Monday morning...

I went almost a month without a BSOD issue after downgrading to 7.7A; but this morning I got a BSOD (probable cause was CPQTeam.sys). I am going to reinstall 7.7A, and try updating to the new HP ProLiant Smart Array SAS/SATA Controller Driver for Windows Server 2003 v6.6.2.32. I will post results.

< http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01102961&dimid=1480679232&dicid=alr_jul07&jumpid=em_alerts/us/jul07/all/xbu/emailsubid/mrm/mcc/loc/rbu_category/alerts >

[Bruce Holzrichter]

I had not been back to this thread in a while, looks like it's still active though...

Joshua,
I got access to the test driver by being 'extremely persistent' :o)
I had an Microsoft Ticket Open
I had an HP ticket Open
I called on each frequently, and eventually Microsoft was able to point out to HP that the issue was there's, and then HP escalated to some engineering dept that I began working with. I was nice enough working with them, I guess, that when a test driver came out, they wanted to pick a couple customers that were experiencing the issue to try it out, and sent it to me directly.

For those of you that are seeing the random BSOD's, and want to run 7.8 (not the NIC shutdown issues..) I would install this driver: http://h18023.www1.hp.com/support/files/server/us/download/27349.html

[Joshua Small_2]

Hi Bruce,

Thanks for your reply. I'm guessing you got lucky with who you talked to. I couldn't have been much more persistant, and every day when I called I got the same canned response about how to search the HP website for the latest PSP. As for having an open ticket, there seemed to be nothing I could do to keep it open- HP just told me to reinstall and that the ticket was considered closed based upon that.

Anyway, the link you posted does resolve the daily BSOD issues for most people by the looks of it.

Also looks like HP finally bought out a new NIC driver,
http://h18023.www1.hp.com/support/files/server/us/download/27389.html
Although disturbingly, the issue we've all seen isn't listed as a "fix", i'll be trying it out and reporting back.

[Bhavnish]

Hi Kindly go through from below Customer Advisory it will help you.

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01102958

Thnx
Bhavnish

[Arian van der Pijl]

FYI; New KB Article with updated storport.sys -> (SP2) v5.2.3790.4121.

'You receive a "Stop 0x0000009C" error message when you shut down a computer that is running the Microsoft Storport storage driver in Windows Server 2003'

http://support.microsoft.com/kb/939315/

[dms_1]

Good news for everyone - PSP 7.9:

http://h18007.www1.hp.com/support/files/server/us/revision/3376.html


Regards

[Jesse Zellmer]

Well, the release notes look promising. I am going to push this to our test servers and see what happens.

I dont want to see a new thread entitled "STOP error after PSP 7.9"!

[Arian van der Pijl]

MS KB Article;
'You receive a Stop error message after you install update 932755 on an HP ProLiant server that is running Storport in Windows Server 2003'
http://support.microsoft.com/kb/940015/

CAUSE
This problem occurs because the Storport storage driver that is included in Windows Server 2003 SP2 and in update 932755 is incompatible with HP driver Hpcisss2.sys. This problem occurs because this update to the Storport driver exposes an issue with I/O Control (IOCTL) calls that are made by the HP Insight Management Storage Agents.