topic Re: STOP error after PSP 7.8 in ProLiant Servers (ML,DL,SL)

STOP error after PSP 7.8

Joshua Small_2 — Thu, 17 May 2007 19:24:54 GMT

Hi,

We were running PSP 7.6.
We applied 7.8, rebooted and things seemed fine yesterday.

This morning we got a BSOD. Analysis on the dump finishes with:

Probably caused by : fltmgr.sys ( fltmgr!FltpDispatch+6f )

This could well be an MS issue, but the standard MS response is that this is likely a disk driver issue.

Version Control Agent assures me I am running the latest, same with E200 firmware.

Any advise?

Re: STOP error after PSP 7.8

Daniel Leblanc — Fri, 18 May 2007 14:04:45 GMT

What server model?
What OS?

TY

Dan

Re: STOP error after PSP 7.8

Daniel Leblanc — Fri, 18 May 2007 14:08:16 GMT

Suggestion
1. chkdsk /r
2. Re-allocate paging space
3. Reseat RAM
4. Check overheat
5. Check the PSU has adequate power supply to drive all devices including USB device

Re: STOP error after PSP 7.8

Daniel Leblanc — Fri, 18 May 2007 14:12:18 GMT

Is it possible to have the dump file texte here in the forum?

Re: STOP error after PSP 7.8

Joshua Small_2 — Fri, 18 May 2007 22:34:46 GMT

Sorry, mentioned it then cut + pasted it somewhere else.

The server is a DL380 G5. OS is Windows 2003 R2 with Service Pack 2.

There are no USB devices, nothing added from standard HP config except more RAM (HP badged).

System Management Homepage reports no overheating, redundant fans operating, no issues other than a silly version control agent error (referenced in my other post).

The dump file came to 256MB, I'm assuming I can't post it here with that size.

I have tried your suggestions, but the server went 24 hours without issue following my post, before I touched it, so it's hard to say if it helped.

Things I am also suspicious of at this stage are Symantec Backup Exec and McAfee Antivirus. As this is a production server, I'd rather now try removing those until I see the issue recur (they involve a reboot).

Re: STOP error after PSP 7.8

Arian van der Pijl — Sat, 19 May 2007 03:36:56 GMT

Hi,

I see you use McAfee AV. What version?
Do you use the latest hotfix as I can recall that McAfee did have some serious issues with their products.
fltmgr.sys does look like a driver that filters I/O requests.
R2 does also have extended quota, DFS-Replication and file usage reporting tools, do you use these options?

Re: STOP error after PSP 7.8

Joshua Small_2 — Sat, 19 May 2007 18:04:34 GMT

Yes, we have the latest version of Mcafee(8.5i) and we have the only released patch (patch1) applied.

We are not using any quota options at all, and have never touched hte DFS service.

Re: STOP error after PSP 7.8

Joshua Small_2 — Sat, 19 May 2007 18:13:09 GMT

I've just had another one, got this output this time:

Probably caused by : memory_corruption

Except my memory looks OK in the system management homepage.

Re: STOP error after PSP 7.8

Joshua Small_2 — Sun, 20 May 2007 19:27:26 GMT

I finally got a useful looking dump. Output this time:
Probably caused by : cpqteam.sys ( cpqteam+11fb9 )

Any possible issues with the teaming driver?

Re: STOP error after PSP 7.8

Jded — Wed, 23 May 2007 12:19:01 GMT

We have the same server with the same OS.
And we have random BSOD.
I tried to disolve the teaming and the results was better but not OK. He have BSOD on every restart or shutdown.
I upgraded to PSP 7.8. The same results. I made the team again and it seems that it is working now (but this is the first day).
I'll upgrade every firmware to the latest and we'll keep an eye on the server for some time...

Re: STOP error after PSP 7.8

Mikearm — Thu, 24 May 2007 03:44:59 GMT

Have similar issue, narrowed it down to Nic driver cp007290.exe.

Scenario:

4 x DL360 G5 Virgin Builds no imaging involved.
Applied PSP 7.8
On shutdown, automatically rebooted.
Removed auto reboot on error from windows.
Shutdown using command line or gui gave BSOD, with stop error 0x0000007e (......etc) but with no driver info listed.
Powered off server, then powered on and ok, until reboot.
Uninstalled all agents from add/remove programs, didn't fix it.
Forced install of PSP 7.7, which fixed the problem.
Started adding PSP 7.8 packs one by one and rebooted in between. Found cp007290 to be the one affecting my BSOD.

Conclusion
I'm going to stick with 7.7, for time being and update to 7.9 when it arrives.

Re: STOP error after PSP 7.8

Joshua Small_2 — Thu, 24 May 2007 04:50:42 GMT

Based on your suggestion, it should be safe for you to apply PSP 7.8, then just downgrade the NIC driver.

I don't think I'm having the same issue though- my BSODs have been very sporadic. I haven't been able to get onsite to shutdown and run any hardware tests, but I've continued to see one reboot a day, and the error is often different.

That really should point to memory, but all my RAM is fitted with "Advanced ECC" which has is still reporting 0 single bit errors. I could well be motherboard or similar but I'm unsure how I would ever diagnose that.

Re: STOP error after PSP 7.8

Mikearm — Thu, 24 May 2007 05:04:48 GMT

Yes your right I could do the downgrade of just the nic driver but I prefer to authorise a certain PSP within the company so we have a standard across all servers.

Also on various other forums people are recognising other issues with 7.8, and BSOD.

I was thinking you might want to force a downgrade yourself and see if it helps clear things up.

Generally if a server fails here after applying HP packs, in the past we have found a downgrade or uninstall and reinstall often fixes the problem.

Re: STOP error after PSP 7.8

Joshua Small_2 — Sat, 26 May 2007 20:50:37 GMT

Just an update on this.
All diagnostics appear to check out OK.

The server still bluescreens once a day at least. Doesn't apear related to any particular activity.

I get different dump analysis each time.

Recently I've seen a number of event logs shortly before the bluescreens, relating to checksum errors in our SQL database.

I've also seen Active Directory log errors relating to database writes so it really looks like there's some sort of storage related error, BUT all diagnostics check out.

All HP phone support (yes we have a carepack) have told us up until this point is to go and reapply the PSP, which doesn't affect the situation. We have also reflashed the E200 RAID card.

Any possible advice would be greatly appreciated here.

Re: STOP error after PSP 7.8

Jonathan Rees — Mon, 28 May 2007 02:12:48 GMT

We have same errors with our ML370 G5 servers, but running x64 R2 SP2. Removing SP2 and reverting to PSP 7.7 seems to have fixed it??

Re: STOP error after PSP 7.8

Joshua Small_2 — Mon, 28 May 2007 04:38:59 GMT

After an investigation by HP, they have claimed the issue was known and would be resolved by applying this firmware update:

http://h18023.www1.hp.com/support/files/storage/us/download/25762.html?jumpid=reg_R1002_USEN

Prior to even logging the call I had check with the Version Control Agent that we had applied all updates, then double checked the "Drivers and Software" for a DL380 G5 from www.hp.com.au to ensure we were up to date on everything.

This firmware however:
- Does not show up on VCA
- Doesn't seem to exist anywhere on the australian (.au) part of the HP website
- Isn't listed on any page related to the server in question

Which made it very hard to find. I'm really concerned that an update, which was required to prevent regular Bluescreens, was so hard to get at.

The update was only applied a few hours ago, so it's too early to tell if it was really the fault. I'm still concerned someone's just following the "update everything" part of a checklist because I haven't found any technote or mention anywhere of this actually being a critical update.

But I will advise shortly.

Re: STOP error after PSP 7.8

Jonathan Rees — Mon, 28 May 2007 18:15:30 GMT

We are running this version of disk firmware, long before the fault. This does NOT fix the fault, at least on x64.

Re: STOP error after PSP 7.8

Joshua Small_2 — Mon, 28 May 2007 18:25:08 GMT

I have since been asked to update my E200 RAID firmware - again.
During our first conversation with HP, that was what was suggested. I had advised we were already running the latest, and was asked to reapply it.
I did so, and then sent them the ADU report to prove it.
They went through a few driver updates, then suggested we update the E200 firmware.

The server has been stable thus far- but it hasn't been 24 hours yet, and the server has held its own longer than that.

I'm really hoping this is it, I can't sit here reflashing the E200 every day in the hope of fixing it.

Re: STOP error after PSP 7.8

Jonathan Rees — Mon, 28 May 2007 18:40:48 GMT

We are running P400 raid controllers. New fimware became available via HP SIM yesterday. Cannot find it on the HP site, and is dated as a June 2007 release. I have applied it to 3 servers, which have not crashed since, but only 24 hours so far

Re: STOP error after PSP 7.8

Joshua Small_2 — Tue, 29 May 2007 19:36:52 GMT

Well no suprises here, the HDD update didn't fix anything.

I've got a business critical server going offline daily, as it has been for some time, and when calling HP back today, well, let's talk about updating my firmware some more.

Any assistance that doesn't involve reflashing again and again would be very greatly appreciated, I'm at my wits end here.