topic Re: ProLiant DL385 G10 Plus OCP PCIe bus width issue in ProLiant Servers (ML,DL,SL)

ProLiant DL385 G10 Plus OCP PCIe bus width issue

kondor — Wed, 26 Oct 2022 00:37:34 GMT

Hi Everybody,

I recently installed OCP NIC Mellanox 566A which is basically 2x100Gb/s network controller in my DL385 G10 Plus server. This NIC is PCIe 4.0 x16 lanes OCP card. It was recognized in bios with bus width of x8. In this server by default there are only 8 PCIe lanes connected directly to the CPU1. If you want additional 8 lanes you can achieve that with something like P14318-001 which is OCP Pass Through NVMe Cables (for processor 1). I connected the proper cable today but my NIC still shows in BIOS with bus width x8. I'm guessing it should be now detected with bus width x16. Am I missing something? Is there a setting somewhere in the BIOS to enable the full widht?

Query: ProLiant DL385 G10 Plus OCP width issue

support_s — Thu, 13 Oct 2022 01:31:43 GMT

System recommended content:

1. Advisory: HPE ProLiant DL325 or ProLiant DL385 Gen10 Plus Servers - "High" Is The Only Fan Speed Available in the OCP Aux Mode Cooling on Servers Running a System ROM Prior to Version 2.40

2. HPE ProLiant DL385 Gen10 Plus v2 Server User Guide

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

Re: ProLiant DL385 G10 Plus OCP PCIe bus width issue

TVVJ — Tue, 13 Aug 2024 07:56:18 GMT

Hello,

You may enable or disbale "~~Enabling or disabling I/O Non-posted Prefecting~~" in the server's RBSU and check if you are able to achieve desired results.

Regards,

[Moderator edit: Removed the broken link. Please refer to https://support.hpe.com/]

Re: ProLiant DL385 G10 Plus OCP PCIe bus width issue

kondor — Fri, 14 Oct 2022 19:51:39 GMT

Hi TVVJ,

Thank you for your reply. I tried but I couldn't find such option in my BIOS. I have the newest 2.60 revision.

Are you sure it's something for AMD CPU servers or is it available only for Intel Proliant version?

Within "Power and Performance Options" I have a submenu there for "I/O Options" but no item there is about prefecting either.

Thanks,

Re: ProLiant DL385 G10 Plus OCP PCIe bus width issue

pchops — Sat, 15 Oct 2022 13:19:27 GMT

What is the current bifurcation setting for the OCP slot (I think it gets enumerated as "PCIe Slot 10")?:

System Configuration > BIOS/Platform Configuration (RBSU) > PCIe Device Configuration > Advanced PCIe Configuration > PCIe Bifurcation Options

Also, I presume that you only installed the P14318-001 cable and connected it from the OCP extension port on the system board to port 8A, also on the system board -- is that correct?

As an aside, I can't find any information about a "Mellanox 566A" -- is it perhaps this adapter?:

P31348-B21 HPE InfiniBand HDR/Ethernet 200Gb 2-port QSFP56 PCIe4 x16 OCP3 MCX653436A-HDAI Adapter

Re: ProLiant DL385 G10 Plus OCP PCIe bus width issue

kondor — Mon, 17 Oct 2022 00:57:43 GMT

Hey Pchops,

Thank you for your reply. You are right, it's PCIe slot 10. Current bifurcation option is set to "Auto". Other two are "Bifurcation" and "Dual bifurcation". I checked the former too and they didn't give any other results than the Auto mode. It's all the same: Data Width: x8, and Negotiated Link Width: x8.

I only installed P14318-001 cable like you stated and exactly that way. The adapter I'm using is more like P10112-B21 part version but 100Gb instead of 25Gb.

Is there a chance it's only showing x8 because it shows only the link width per one port in BIOS? There are places in the BIOS where you can see each port of that NIC as a separate instance. For example in "System Configuration" there are two entries for that NIC: "OCP Slot 10 Port 1..." and "OCP Slot 10 Port 2...". The "System Information"->"PCI Device Information" shows only one port name with only one MAC address as the whole "Device Name". Maybe the whole thing is just a "display issue"?

Re: ProLiant DL385 G10 Plus OCP PCIe bus width issue

pchops — Tue, 18 Oct 2022 07:30:07 GMT

Is there a chance it's only showing x8 because it shows only the link width per one port in BIOS? There are places in the BIOS where you can see each port of that NIC as a separate instance. For example in "System Configuration" there are two entries for that NIC: "OCP Slot 10 Port 1..." and "OCP Slot 10 Port 2...". The "System Information"->"PCI Device Information" shows only one port name with only one MAC address as the whole "Device Name". Maybe the whole thing is just a "display issue"?

I don't know for sure (I don't have a comparable system to try it on), but I wouldn't think the BIOS would display the number of PCIe lanes by port (I would think this would happen behind the scenes on the card itself). My guess is that you have two entries for that NIC so that you can display/change per-port characteristics in the BIOS. In order to see if it is just a "display issue", I can only think of two methods off the top of my pointy head that you could try:

temporarily remove the P14318-001 cable and see if the BIOS still shows x8 on each NIC port (I suspect so)
boot into an OS with a utility that can display the PCIe topology (ie: a Linux live distro with "lspci")

BTW, are there any configuration options on the "System Configuration" page for that NIC that are related to PCIe settings?

Re: ProLiant DL385 G10 Plus OCP PCIe bus width issue

Sham82 — Thu, 20 Oct 2022 00:55:23 GMT

Hello,

please share the details for - Mellanox 566A
as we are not able to find "Mellanox 566A"

please check and share the details of the adapter you are using

For reference : DL385 Gen10 Plus Server Quickspecs
https://www.hpe.com/psnow/doc/a00073549enw.html?jumpid=in_pdp-psnow-qs

Regards
HPE

Re: ProLiant DL385 G10 Plus OCP PCIe bus width issue

kondor — Fri, 21 Oct 2022 18:55:38 GMT

Hi Pchops,

I tried your approach. Problem solved.

I installed barebone linux on that machine to use lspci. It was showing downgraded mode (x8) for the actual link speed both with cable and without cable. The mistake I made is I forgot this card had been powered all the time by one BIOS setting even during reboots. Rookie mistake. I disconnected the whole power and the card was recognized right away with x16 link width. Thank you for your help and your suggestions!

Re: ProLiant DL385 G10 Plus OCP PCIe bus width issue

Sunitha_Mod — Tue, 25 Oct 2022 08:38:24 GMT

Hello @kondor,

Excellent!

We are glad to hear the problem has been resolved.