ProLiant Servers (ML,DL,SL)
cancel
Showing results for 
Search instead for 
Did you mean: 

STOP error after PSP 7.8

SOLVED
Go to solution
Joshua Small_2
Valued Contributor

STOP error after PSP 7.8

Hi,

We were running PSP 7.6.
We applied 7.8, rebooted and things seemed fine yesterday.

This morning we got a BSOD. Analysis on the dump finishes with:

Probably caused by : fltmgr.sys ( fltmgr!FltpDispatch+6f )

This could well be an MS issue, but the standard MS response is that this is likely a disk driver issue.

Version Control Agent assures me I am running the latest, same with E200 firmware.

Any advise?
190 REPLIES
Daniel Leblanc
Honored Contributor

Re: STOP error after PSP 7.8

What server model?
What OS?

TY

Dan
Daniel Leblanc
Honored Contributor

Re: STOP error after PSP 7.8

Suggestion
1. chkdsk /r
2. Re-allocate paging space
3. Reseat RAM
4. Check overheat
5. Check the PSU has adequate power supply to drive all devices including USB device
Daniel Leblanc
Honored Contributor

Re: STOP error after PSP 7.8

Is it possible to have the dump file texte here in the forum?
Joshua Small_2
Valued Contributor

Re: STOP error after PSP 7.8

Sorry, mentioned it then cut + pasted it somewhere else.

The server is a DL380 G5. OS is Windows 2003 R2 with Service Pack 2.

There are no USB devices, nothing added from standard HP config except more RAM (HP badged).

System Management Homepage reports no overheating, redundant fans operating, no issues other than a silly version control agent error (referenced in my other post).

The dump file came to 256MB, I'm assuming I can't post it here with that size.

I have tried your suggestions, but the server went 24 hours without issue following my post, before I touched it, so it's hard to say if it helped.

Things I am also suspicious of at this stage are Symantec Backup Exec and McAfee Antivirus. As this is a production server, I'd rather now try removing those until I see the issue recur (they involve a reboot).
Arian van der Pijl
Regular Advisor

Re: STOP error after PSP 7.8

Hi,

I see you use McAfee AV. What version?
Do you use the latest hotfix as I can recall that McAfee did have some serious issues with their products.
fltmgr.sys does look like a driver that filters I/O requests.
R2 does also have extended quota, DFS-Replication and file usage reporting tools, do you use these options?
Joshua Small_2
Valued Contributor

Re: STOP error after PSP 7.8

Yes, we have the latest version of Mcafee(8.5i) and we have the only released patch (patch1) applied.

We are not using any quota options at all, and have never touched hte DFS service.
Joshua Small_2
Valued Contributor

Re: STOP error after PSP 7.8

I've just had another one, got this output this time:

Probably caused by : memory_corruption

Except my memory looks OK in the system management homepage.
Joshua Small_2
Valued Contributor

Re: STOP error after PSP 7.8

I finally got a useful looking dump. Output this time:
Probably caused by : cpqteam.sys ( cpqteam+11fb9 )

Any possible issues with the teaming driver?
Jded
Occasional Advisor

Re: STOP error after PSP 7.8

We have the same server with the same OS.
And we have random BSOD.
I tried to disolve the teaming and the results was better but not OK. He have BSOD on every restart or shutdown.
I upgraded to PSP 7.8. The same results. I made the team again and it seems that it is working now (but this is the first day).
I'll upgrade every firmware to the latest and we'll keep an eye on the server for some time...
Mikearm
Occasional Advisor

Re: STOP error after PSP 7.8

Have similar issue, narrowed it down to Nic driver cp007290.exe.

Scenario:

4 x DL360 G5 Virgin Builds no imaging involved.
Applied PSP 7.8
On shutdown, automatically rebooted.
Removed auto reboot on error from windows.
Shutdown using command line or gui gave BSOD, with stop error 0x0000007e (......etc) but with no driver info listed.
Powered off server, then powered on and ok, until reboot.
Uninstalled all agents from add/remove programs, didn't fix it.
Forced install of PSP 7.7, which fixed the problem.
Started adding PSP 7.8 packs one by one and rebooted in between. Found cp007290 to be the one affecting my BSOD.

Conclusion
I'm going to stick with 7.7, for time being and update to 7.9 when it arrives.
Joshua Small_2
Valued Contributor

Re: STOP error after PSP 7.8

Based on your suggestion, it should be safe for you to apply PSP 7.8, then just downgrade the NIC driver.

I don't think I'm having the same issue though- my BSODs have been very sporadic. I haven't been able to get onsite to shutdown and run any hardware tests, but I've continued to see one reboot a day, and the error is often different.

That really should point to memory, but all my RAM is fitted with "Advanced ECC" which has is still reporting 0 single bit errors. I could well be motherboard or similar but I'm unsure how I would ever diagnose that.
Mikearm
Occasional Advisor

Re: STOP error after PSP 7.8

Yes your right I could do the downgrade of just the nic driver but I prefer to authorise a certain PSP within the company so we have a standard across all servers.

Also on various other forums people are recognising other issues with 7.8, and BSOD.

I was thinking you might want to force a downgrade yourself and see if it helps clear things up.

Generally if a server fails here after applying HP packs, in the past we have found a downgrade or uninstall and reinstall often fixes the problem.
Joshua Small_2
Valued Contributor

Re: STOP error after PSP 7.8

Just an update on this.
All diagnostics appear to check out OK.

The server still bluescreens once a day at least. Doesn't apear related to any particular activity.

I get different dump analysis each time.

Recently I've seen a number of event logs shortly before the bluescreens, relating to checksum errors in our SQL database.

I've also seen Active Directory log errors relating to database writes so it really looks like there's some sort of storage related error, BUT all diagnostics check out.

All HP phone support (yes we have a carepack) have told us up until this point is to go and reapply the PSP, which doesn't affect the situation. We have also reflashed the E200 RAID card.

Any possible advice would be greatly appreciated here.
Jonathan Rees
Occasional Advisor

Re: STOP error after PSP 7.8

We have same errors with our ML370 G5 servers, but running x64 R2 SP2. Removing SP2 and reverting to PSP 7.7 seems to have fixed it??
Joshua Small_2
Valued Contributor

Re: STOP error after PSP 7.8

After an investigation by HP, they have claimed the issue was known and would be resolved by applying this firmware update:

http://h18023.www1.hp.com/support/files/storage/us/download/25762.html?jumpid=reg_R1002_USEN

Prior to even logging the call I had check with the Version Control Agent that we had applied all updates, then double checked the "Drivers and Software" for a DL380 G5 from www.hp.com.au to ensure we were up to date on everything.

This firmware however:
- Does not show up on VCA
- Doesn't seem to exist anywhere on the australian (.au) part of the HP website
- Isn't listed on any page related to the server in question

Which made it very hard to find. I'm really concerned that an update, which was required to prevent regular Bluescreens, was so hard to get at.

The update was only applied a few hours ago, so it's too early to tell if it was really the fault. I'm still concerned someone's just following the "update everything" part of a checklist because I haven't found any technote or mention anywhere of this actually being a critical update.

But I will advise shortly.
Jonathan Rees
Occasional Advisor

Re: STOP error after PSP 7.8

We are running this version of disk firmware, long before the fault. This does NOT fix the fault, at least on x64.
Joshua Small_2
Valued Contributor

Re: STOP error after PSP 7.8

I have since been asked to update my E200 RAID firmware - again.
During our first conversation with HP, that was what was suggested. I had advised we were already running the latest, and was asked to reapply it.
I did so, and then sent them the ADU report to prove it.
They went through a few driver updates, then suggested we update the E200 firmware.

The server has been stable thus far- but it hasn't been 24 hours yet, and the server has held its own longer than that.

I'm really hoping this is it, I can't sit here reflashing the E200 every day in the hope of fixing it.
Jonathan Rees
Occasional Advisor

Re: STOP error after PSP 7.8

We are running P400 raid controllers. New fimware became available via HP SIM yesterday. Cannot find it on the HP site, and is dated as a June 2007 release. I have applied it to 3 servers, which have not crashed since, but only 24 hours so far
Joshua Small_2
Valued Contributor

Re: STOP error after PSP 7.8

Well no suprises here, the HDD update didn't fix anything.

I've got a business critical server going offline daily, as it has been for some time, and when calling HP back today, well, let's talk about updating my firmware some more.

Any assistance that doesn't involve reflashing again and again would be very greatly appreciated, I'm at my wits end here.
Jded
Occasional Advisor

Re: STOP error after PSP 7.8

The first thing that made result here was to desolve the teaming and then disable one NIC.
That was with PSP 7.7.
The only problem that we had was the BSOD on every restart or shutdown (it is better than a random BSOD).

Now with PSP 7.8 and all the firmware updates I reenabled the NIC and made the team again. The server is running fine (I think).
Joshua Small_2
Valued Contributor

Re: STOP error after PSP 7.8

I have new light to shed. On clean (start->shutdown->restart) reboot, I got this today in my IML:

POST Error: 1792-Drive Array Reports Valid Data Found in Array Accelerator

I know exactly what this error is supposed to mean. That the card kept data in its cache after some kind of unclean shutdown. Only the shutdown was perfectly clean. Event log also shows there was nothing in the way of a STOP error.

Am I correct in assuming there could be something wrong with my E200 RAID card?
I'm seeing continuing event logs relating to checksum errors in my SQL service, Im' really suspicious *something* is wrong with my storage.

I've had a case open with HP for several days, and I'm really hoping the next person to call me back suggests something other than reflashing the system board again.
Jonathan Rees
Occasional Advisor

Re: STOP error after PSP 7.8

I have had this error as well, but it has been after a server crash so is expected. Testing so far seems to indicate that using PSP 7.7 has fixed problem, but have all firmware/bios at latest version. We have been able to induce the crash fault by backing up 200 + Gbytes of data. We are using DataProtector 5.5, fully patched. Seems to be related to server load, i.e. backup results in enough load to trigger fault. If you are running Storport drivers there is a new release from 28 March that fixes a number of stop errors. Check http://support.microsoft.com/kb/912944/en-us
Joshua Small_2
Valued Contributor

Re: STOP error after PSP 7.8

Hi Jonathon,

That's an old storport patch :)
We already applied this one to try resolving the issue ourselves:

http://support.microsoft.com/kb/932755

It didn't :(

Jonathan Rees
Occasional Advisor

Re: STOP error after PSP 7.8

Oops, posted the wrong link, mean't the 1 you posted, did not fix our problem either, but MS blurb does mention stop errors with other HP raid cards, i.e. 5x/6x, which require driver updates.