DL380p Gen8 with uncorrectabl PCI express error

[Hank9999]

HI everyone,

there is a problem with my DL380p gen8. The server keeps crashing with always the same error messages

Critical PCI Bus 03/13/2013 17:12 03/13/2013 17:12 1 Uncorrectable PCI Express Error (Embedded device, Bus 0, Device 2, Function 2, Error status 0x00000000)
Critical System Error 03/13/2013 17:12 03/13/2013 17:12 1 Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible
Caution POST Message 03/13/2013 16:43 03/13/2013 16:43 1 POST Error: 1792-Slot X Drive Array - Valid Data Found in Cache Module. Data will automatically be written to drive array.
Caution POST Message 03/13/2013 16:43 03/13/2013 16:43 1 POST Error: 1719 - A controller failure event occurred prior to this power-up
Critical PCI Bus 03/13/2013 16:41 03/13/2013 16:41 1 Uncorrectable PCI Express Error (Embedded device, Bus 0, Device 2, Function 2, Error status 0x00000000)
Critical System Error 03/13/2013 16:41 03/13/2013 16:41 1 Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible

There is no PCI card installed, external USB drives are already detached, the latest Service Pack and Intelligent Provisioning was installed and updated.

Could this be caused by the PCI riser cage, although nothing is installed?

[Bjoern13]

Hello Hank,

if you have updated the complete server and everything is to the latest version, you can try to run the server without the riser cage, if nothing is installed and it is not needed anyways right now.

Concerning the external USB drives: This is good that you tried it without it/them, since this is often the cause for unexpected behaviour.
If you can test it without the PCI riser. In either case (solved or not afterwards) I would advise you to contact the support so that the part(s) can be replaced afterwards.

If you do so, make sure you provide an AHS log (downloadable from iLO). This will help the support colleagues and figure out what went wrong. They can check the PCI Bus and which device/ system component is using the Bus 0, Device 2

[Med-H]

Hi,

what is the frequency of the issue ?

Please try to eliminate the NIC card installed in the server "HP Ethernet Adapter"

Services Media Library:

https://sml-csr.ext.hpe.com/results.htm?SID=5177957&MEID=FB1D0B1C-7DCD-42FF-B69E-C7069640D148

 [Moderator edit: Removed the broken link. Please refer to https://support.hpe.com/]

[krishblr]

Update System ROM dated 12/21/2011: has the fixes for Unexpected Uncorrectable PCI Express Error.

Ref the advisory.
https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_ce934189ac014577a4fbbdc7e1

If you still see the following error, Please raise a support case with HP along AHS logs for further technical assistance.

Caution POST Message 03/13/2013 16:43 03/13/2013 16:43 1 POST Error: 1792-Slot X Drive Array - Valid Data Found in Cache Module. Data will automatically be written to drive array.
Caution POST Message 03/13/2013 16:43 03/13/2013 16:43 1 POST Error: 1719 - A controller failure event occurred prior to this power-up

Cache module could have an issue.

---------------
I am an HP employee.

If you wish to thank, Just click the KUDOS Star.

[Hank9999]

Hello again,
thanks for the fast reply.

The frequency of the issue was first every few days, mostly during the weekend. This is why I suspected the USB drives.
We already had a systemboard replacement and after that the freqency went up. It was almost every 15 minutes then that the server crashed.

This is why I asked for the PCI riser cage. I have taken it out now, and up to now, there was no further issue.
I have to monitor it for a while to see if that actually stopped it. There is no additional network card installed. Just the build in one

[madhuiss]

The Offline Advanced Survey will have more details about the device & the bus.AHS will also be ok to know the device.

Embedded device, Bus 0, Device 2, Function 2, Error status 0x00000000

There are 2 things could be workedaround:

- Disable Intel QPI in the RBSU

- Disable all C-states( No c-states) for verification.

Thank You

I am an HP Employee

[Hank9999]

now there was again an unexpected restart although the PCI riser is out.

so the frequency slowed down but it was not solved.
since the Systemboard was already replaced and the Smart Array Controller is embedded, could it be the cache module that causes the issue?

[Med-H]

could you please provide us the ahs logs.

How to Generate Active Health System log via iLO GUI :

1. Logon to HP iLO 4 GUI (IP address at POST visible or check server iLO Default Network Settings tag for DNS name).

2. Go to the menu on the left called: Active Health System Log .

3. Verify the date interval (default is 7 days) and make sure that all possible failures are covered within this date range as selected.

4. Enter the possible Contact information so that the HP Support agent can easily get back to user with follow up questions.

   The Contact information text is the only readable text in this binary AHS log file.

5. Press the Download button to start downloading the AHS log file.

   thanks

[Allgäu]

Hello,

we have the same issue with 14 Gen8 servers. Is there a solution for this problem.

Thanks

Chris

[Sbrown]

1. firmware is outdated.

2. P420 is overheating.

3. Bad motherboard. We found it would only show up with ESXi - that would show tons of ECC correctable errors. HP memtestd nothing.

We got a new MOBO and all was good! I think the first batch were buggy lol!

(install ESXI using hp oem installer) , warm it up with some benchmarks, and ssh into the host and

dmesg

or look at the logs. It was a mess!

[aperson]

We have the same issue with ESXi 5.0 U3. We still cannot resolve the issue and it occurs with DL380p Gen 8 8-core and 12-core models. 

The HBA on the riser card fails. Only this HBA 81Q: 

QLogic PCI to Fibre Channel Host Adapter for HPAK344A:

Host Device Name vmhba3

BIOS version 3.13

FCODE version N/A

EFI version 6.23

Flash FW version 5.09.00

Is there any resolution? We have updated drivers and FW of system board, replaced system board and riser board, yet we still get the same failures after some days. 

[PMI_WINCHAM]

Same issue here on a DL360p Gen 8 running Xenserver 6.1.

Happens on average once a month.

System ROM - P71 02/25/2012

It is very frustrating to not to see much effort on HP's behalf to sort this issue.

[DaneTruscott]

We have 4 DL380p and we are getting the same issue on only one of them,  as the last post was asbout a year ago has anyone had a fix for this? 

[ferkop1]

Hi today same server crash, Uncorrectable PCI Express Error (Slot 2, Bus 0, Device 28, Function 0, Error status 0x00010000)

latest firmware

at server is running mikrotik router OS, added 1x 10Gbit sfp+ card

system installed at SSD

I need help ASAP.

somebody fixed this issue?

[proflan]

same error here, does anyone have a solution?

Unrecoverable System Error (NMI) has occurred.  System Firmware will log additional details in a separate IML entry if possible

Uncorrectable PCI Express Error (Embedded device, Bus 0, Device 28, Function 7, Error status 0x00100000)

Operating System failure (Windows bug check, STOP: 0x00000080 (0x00000000004F4454, 0x0000000000000000, 0x0000000000000000, 0x0000000000000000))

System ROM: P70 05/21/2018

iLO Firmware Version:  2.60 May 23 2018

[proflan]

found the solution - the graphic driver was the problem:

https://community.hpe.com/t5/ProLiant-Servers-ML-DL-SL/DL360p-gen8-STOP-0x00000080-Uncorrectable-PCI-Express-Error-NMI/td-p/6996054

[f22raptorroland]

I had this exact same issue, exact same message.
Uncorrectable PCI Express Error (Embedded device, Bus 0, Device 2, Function 2, Error status 0x00000000)

Also, I noticed my HD controller was running at 65°C (I'm not sure if it was the maximum sensor reading, but I mounted a huge heating on it from a motor drive, and barely got any cooler)
I tried to move things in PCIe slots around without the rizer cage, etc. But to no avail
It got so bad that the server rebooted every few minutes by the end.

In the end, I threw out the Cache module (1Gb) and it solved all the problems.
There are no more PCI issues from this, and the HD controller is running at 40c