Proliant DL380 G7 Fatal PCI Express Device Error PCI ? B00/D00/F00

[MisoVranes]

Hi friends.

I am experiencing serious problems with HP Proliant Servers DL380 G7.

These servers reboot or crash much more then any proliant I have ever seen before. After reboot I often see this error:

"Fatal PCI Express Device Error PCI ? B00/D00/F00 " Also, there are false power supply #1 errors. On one of these servers couple of times I also experienced uncorrectable ecc memory errors which were not solved by replacing memory and happen again and again .

The Integrated Management log :
Uncorrectable PCI Express Error (Embedded device, Bus 0, Device 0, Function 0, Error status 0x00000000)

I have upgraded all firmwares to current version (today is 26.08.2011), except smart array controller. Fixes and release notes for BIOS and ILO3 firmwares are pointing out that  behaviour regarding these problems is improved, but for my affected servers it is still NO GO.

I am aware of advisory from last year:

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: c02178619

Version: 1

Advisory: HP Integrated Lights-Out 2 (iLO 2) - "Fatal PCI Express Device Error" Message May be Displayed After an AC Power Cycle When Memory Size Is Reduced to Below 4 GB

https://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?javax.portlet.tpst=ba847bafb2a2d782fcbb0710b053ce01&javax.portlet.prp_ba847bafb2a2d782fcbb0710b053ce01=wsrp-navigationalState%3DdocId%25253Demr_na-c02178619%25257CdocLocale%25253Den&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

and I am aware of the post

http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL-SL/DL380G7-win2k8r2-w-sp1-fatal-pci-express-device-error/m-p/4769378#M110670

My servers have two aditional NC364T network cards in PCIe slots 2 and 3.  Servers have two Xeon 5640 processors.

Servers are running VMWare ESXi 4.1 U1.

Does anyone experience similar problems or have a solution?

Best regards to all.

[Sirajul haque]

I rememember this error with P410 controllers and I suppose you use the same..

Uncorrectable PCI Express Error (Embedded device, Bus 0, Device 7, Function 0, Error status 0x00004000)

But in your case, the error message is not exactly same.. However, you may try below said action plan:

1. Update the firmware of Storage controller, i think you have P410 controller, download Version:     5.06 (C) (29 Jul 2011) from HP.com

I wrote something about it here:

http://www.tricksguide.com/blue-screen-error-hardware-malfunction-pci-express-error-hp-proliant-server.html

Also try:

To disable “No-Execute Memory Protection” on your server, enter in to Server BIOS (RBSU) by pressing the F9 key at Server Boot -> under System options -> Select Processor options –> Now select No Execute Memory Protection –> Set to Disabled


Note: The erros message has some difference "Device 7" and "Device 0", so this is just a recommendation from my side.. :),  But I think its worth a try :)

[AkiraX]

I see the same issues with my bl460c G7's even with the latest 5/5/2011 BIOS......

Sometimes, like 5 minutes ago, when you reboot the server it just comes up with a black screen without the PCI Express Error. A coldboot via Ilo resolves it until next time.

[AkiraX]

also, because its a hyper-v server I cant disable the No-Execute memory option.

At the advice of HP, I toggle the dip switch 6(I think), power-on, re-toggle and still no luck as it eventually comes back.

[MisoVranes]

I have tried to analyze situation using bootable linux but I can not find out whait PCIe device is under incestigation. It must be ILO3. I am not sure, I will try to upgrade the smart array, and Iwill conduct offline Insight diagnostics, but stability is ridicilous.

DL380 series were my favourites before. Hey, HP are we becoming beta testers?

[MisoVranes]

I will try.

Thanks.

[MisoVranes]

All updates installed and still no go. Servers crash very often. I set at the end "NO c-states" in BIOS and set power management to maximum performance. I am trying to avoid that stupid mbedded PIe device error by setting PCIe compliance to PCIe Gen #1. I will inform what happened.

[Sirajul haque]

Hows it going now? Did that work?

[µ¿½ÉÇÏÀÌÅ×Å© ½Å ¸íÈÆ]

i solved this problem..
dame error messages. :)
my case is... sata cable problem.. :(
sata cable replace..

[Daniel Lepak]

Based on in the field experience always check the FW (Firmware) first. Unless you are using a special BIOS setup. Otherwise refer to the latest & greatest FW to resolve most issues w/ ProLiant Servers G7 and above using SPP2013020B...

http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

and utilize the HP Support Center on the HP Support Site; as noted below...

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/mostViewedDisplay/?javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken&javax.portlet.prp_efb5c0793523e51970c8fa22b053ce01=wsrp-navigationalState%3DdocId%253Demr_na-c03261617-2%257CdocLocale%253Den_US&javax.portlet.tpst=efb5c0793523e51970c8fa22b053ce01&sp4ts.oid=4091412&ac.admitted=13...

Hope this resolves your issue or anyone else experiencing similar issues w/ there G7 or above.

Please note: ProLiant G6 & below Servers may or may not work using SPP2013020B (Service Pack for ProLiants 2013) they may have to utilize the old school PSP's & SmartStart FW DVD/CD's.

[Panmudg]

Hello Friends

I am facing a problem with my HP DL360 G7 "Fatal PCI Express Device Error PCI ? B00/D00/F00 "

I found in this post and applied the solution No-Execute Memory Protection.

To disable “No-Execute Memory Protection” on your server, enter in to Server BIOS (RBSU) by pressing the F9 key at Server Boot -> under System options -> Select Processor options –> Now select No Execute Memory Protection –> Set to Disabled

When I do this my server works fine but gives same trouble upon reboot; can you please help me applying apermanent solution.

Regards

[Jimmy Vance]

Could be a backplane or cable issue

http://h20564.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c03261617