DL380p Gen8 with uncorrectabl PCI express error

[Hank9999]

HI everyone,

there is a problem with my DL380p gen8. The server keeps crashing with always the same error messages

Critical PCI Bus 03/13/2013 17:12 03/13/2013 17:12 1 Uncorrectable PCI Express Error (Embedded device, Bus 0, Device 2, Function 2, Error status 0x00000000)
Critical System Error 03/13/2013 17:12 03/13/2013 17:12 1 Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible
Caution POST Message 03/13/2013 16:43 03/13/2013 16:43 1 POST Error: 1792-Slot X Drive Array - Valid Data Found in Cache Module. Data will automatically be written to drive array.
Caution POST Message 03/13/2013 16:43 03/13/2013 16:43 1 POST Error: 1719 - A controller failure event occurred prior to this power-up
Critical PCI Bus 03/13/2013 16:41 03/13/2013 16:41 1 Uncorrectable PCI Express Error (Embedded device, Bus 0, Device 2, Function 2, Error status 0x00000000)
Critical System Error 03/13/2013 16:41 03/13/2013 16:41 1 Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible

There is no PCI card installed, external USB drives are already detached, the latest Service Pack and Intelligent Provisioning was installed and updated.

Could this be caused by the PCI riser cage, although nothing is installed?

[Bjoern13]

Hello Hank,

if you have updated the complete server and everything is to the latest version, you can try to run the server without the riser cage, if nothing is installed and it is not needed anyways right now.

Concerning the external USB drives: This is good that you tried it without it/them, since this is often the cause for unexpected behaviour.
If you can test it without the PCI riser. In either case (solved or not afterwards) I would advise you to contact the support so that the part(s) can be replaced afterwards.

If you do so, make sure you provide an AHS log (downloadable from iLO). This will help the support colleagues and figure out what went wrong. They can check the PCI Bus and which device/ system component is using the Bus 0, Device 2

[Med-H]

Hi,

what is the frequency of the issue ?

Please try to eliminate the NIC card installed in the server "HP Ethernet Adapter"

Services Media Library:

https://sml-csr.ext.hpe.com/results.htm?SID=5177957&MEID=FB1D0B1C-7DCD-42FF-B69E-C7069640D148

[krishblr]

Update System ROM dated 12/21/2011: has the fixes for Unexpected Uncorrectable PCI Express Error.

Ref the advisory.
https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_ce934189ac014577a4fbbdc7e1

If you still see the following error, Please raise a support case with HP along AHS logs for further technical assistance.

Caution POST Message 03/13/2013 16:43 03/13/2013 16:43 1 POST Error: 1792-Slot X Drive Array - Valid Data Found in Cache Module. Data will automatically be written to drive array.
Caution POST Message 03/13/2013 16:43 03/13/2013 16:43 1 POST Error: 1719 - A controller failure event occurred prior to this power-up

Cache module could have an issue.

---------------
I am an HP employee.

If you wish to thank, Just click the KUDOS Star.

[Hank9999]

Hello again,
thanks for the fast reply.

The frequency of the issue was first every few days, mostly during the weekend. This is why I suspected the USB drives.
We already had a systemboard replacement and after that the freqency went up. It was almost every 15 minutes then that the server crashed.

This is why I asked for the PCI riser cage. I have taken it out now, and up to now, there was no further issue.
I have to monitor it for a while to see if that actually stopped it. There is no additional network card installed. Just the build in one

[madhuiss]

The Offline Advanced Survey will have more details about the device & the bus.AHS will also be ok to know the device.

Embedded device, Bus 0, Device 2, Function 2, Error status 0x00000000

There are 2 things could be workedaround:

- Disable Intel QPI in the RBSU

- Disable all C-states( No c-states) for verification.

Thank You

I am an HP Employee

[Hank9999]

now there was again an unexpected restart although the PCI riser is out.

so the frequency slowed down but it was not solved.
since the Systemboard was already replaced and the Smart Array Controller is embedded, could it be the cache module that causes the issue?

[Med-H]

could you please provide us the ahs logs.

How to Generate Active Health System log via iLO GUI :

1. Logon to HP iLO 4 GUI (IP address at POST visible or check server iLO Default Network Settings tag for DNS name).

2. Go to the menu on the left called: Active Health System Log .

3. Verify the date interval (default is 7 days) and make sure that all possible failures are covered within this date range as selected.

4. Enter the possible Contact information so that the HP Support agent can easily get back to user with follow up questions.

   The Contact information text is the only readable text in this binary AHS log file.

5. Press the Download button to start downloading the AHS log file.

   thanks

[Allgäu]

Hello,

we have the same issue with 14 Gen8 servers. Is there a solution for this problem.

Thanks

Chris

[Sbrown]

1. firmware is outdated.

2. P420 is overheating.

3. Bad motherboard. We found it would only show up with ESXi - that would show tons of ECC correctable errors. HP memtestd nothing.

We got a new MOBO and all was good! I think the first batch were buggy lol!

(install ESXI using hp oem installer) , warm it up with some benchmarks, and ssh into the host and

dmesg

or look at the logs. It was a mess!