HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter throughput only 40 Gbps.

[RolledSteel]

Can anyone help me figure out why my HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter is bottlenecked at 40Gbit/s?
---
Environment
Server Specifications
Platform: HPE ProLiant DL380 Gen9 BIOS v2.90
Processors: Dual Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Operating System: 20.04.2 LTS (GNU/Linux 5.4.0-84-generic x86_64)
HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter
Mellanox MT27800 Firmware 16.31.1200

---
When I run a lshw -class network I get the output below.

*-network:1
description: Ethernet interface
product: MT27800 Family [ConnectX-5]
vendor: Mellanox Technologies
physical id: 0.1
bus info: pci@0000:84:00.1

logical name: ens5f0
version: 00
serial: 94:40:XX:XX:XX:XX|
capacity: 40Gbit/s
width: 64 bits
clock: 33MHz

capabilities: pciexpress vpd msix pm bus_master cap_list ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation

configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.4-1.0.3 duplex=full firmware=16.31.1200 (HPE0000000009) ip=10.X.X.X latency=0 link=yes multicast=yes

resources: irq:17 memory:96000000-97ffffff memory:39ffae00000-39ffcdfffff
---

You can see that the Capacity is 40Gbit/s, however when I run ethtool I do see the Speed as 100000Mb/s. The Cisco Nexus Switch also shows the link speed at 100Gbps. When we performed an iperf test we were getting only around 40 gbps which indicates to we are being bottlenecked.

---
Environment
Server Specifications
Platform: HPE ProLiant DL380 Gen9 BIOS v2.90
Processors: Dual Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Operating System: 20.04.2 LTS (GNU/Linux 5.4.0-84-generic x86_64)
HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter
Mellanox MT27800 Firmware 16.31.1200
---
Is there an issue with the PCI configuration or the auto negotiation? Is there a hidden setting/configuration somewhere that I need to change to get the full 100 GBPS??

Things I've tried:
- Install MLNX OFED Drivers per documentation guidance: https://www.hpe.com/psnow/doc/a00094362enw
- Updated Host Adapter firmware
- Ensuring the Host Adapter is plugged into a PCIE Gen 3 x16 slot on the secondary riser card

[Suman_1978]

Hi,

I am not a Network expert, but most of the times the Network cables could be an issue.
In your post, I did not see mentioing the type of cables used, CAT5, CAT5e or CAT6.

See if these settings helps you to get the speed:
Enforcing auto-negotiation activation
Manually setting the link speed and duplex

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/ch-configuring_network_connection_settings

Thanks

[RolledSteel]

No CAT cables for these throughputs. We are using Cisco QSFP-100G-CU3M 100GBase CR4 Passive cables. We are using Ubuntu 20.04 headless and it appears the app you are suggesting requires a GUI. Thanks for the response but I think using Network-Manager (ubuntu package of NetworkManager) is not the solution...

[ksram]

Hi,

Doesnt looks like any Hardware Issues.

May be we can still try Firmware and Drivers - just a general suggestion.

Thank you

RamKS