- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - Linux
- >
- HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28...
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-21-2021 04:03 PM - last edited on 09-30-2021 02:14 AM by support_s
09-21-2021 04:03 PM - last edited on 09-30-2021 02:14 AM by support_s
HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter throughput only 40 Gbps.
Can anyone help me figure out why my HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter is bottlenecked at 40Gbit/s?
---
Environment
Server Specifications
Platform: HPE ProLiant DL380 Gen9 BIOS v2.90
Processors: Dual Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Operating System: 20.04.2 LTS (GNU/Linux 5.4.0-84-generic x86_64)
HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter
Mellanox MT27800 Firmware 16.31.1200
---
When I run a lshw -class network I get the output below.
*-network:1
description: Ethernet interface
product: MT27800 Family [ConnectX-5]
vendor: Mellanox Technologies
physical id: 0.1
bus info: pci@0000:84:00.1
logical name: ens5f0
version: 00
serial: 94:40:XX:XX:XX:XX|
capacity: 40Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pciexpress vpd msix pm bus_master cap_list ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.4-1.0.3 duplex=full firmware=16.31.1200 (HPE0000000009) ip=10.X.X.X latency=0 link=yes multicast=yes
resources: irq:17 memory:96000000-97ffffff memory:39ffae00000-39ffcdfffff
---
You can see that the Capacity is 40Gbit/s, however when I run ethtool I do see the Speed as 100000Mb/s. The Cisco Nexus Switch also shows the link speed at 100Gbps. When we performed an iperf test we were getting only around 40 gbps which indicates to we are being bottlenecked.
---
Environment
Server Specifications
Platform: HPE ProLiant DL380 Gen9 BIOS v2.90
Processors: Dual Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Operating System: 20.04.2 LTS (GNU/Linux 5.4.0-84-generic x86_64)
HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter
Mellanox MT27800 Firmware 16.31.1200
---
Is there an issue with the PCI configuration or the auto negotiation? Is there a hidden setting/configuration somewhere that I need to change to get the full 100 GBPS??
Things I've tried:
- Install MLNX OFED Drivers per documentation guidance: https://www.hpe.com/psnow/doc/a00094362enw
- Updated Host Adapter firmware
- Ensuring the Host Adapter is plugged into a PCIE Gen 3 x16 slot on the secondary riser card
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-23-2021 07:50 AM
09-23-2021 07:50 AM
Re: HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter throughput only 40 Gbps.
Hi,
I am not a Network expert, but most of the times the Network cables could be an issue.
In your post, I did not see mentioing the type of cables used, CAT5, CAT5e or CAT6.
See if these settings helps you to get the speed:
Enforcing auto-negotiation activation
Manually setting the link speed and duplex
Thanks
I work for HPE.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-23-2021 10:53 AM
09-23-2021 10:53 AM
Re: HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter throughput only 40 Gbps.
No CAT cables for these throughputs. We are using Cisco QSFP-100G-CU3M 100GBase CR4 Passive cables. We are using Ubuntu 20.04 headless and it appears the app you are suggesting requires a GUI. Thanks for the response but I think using Network-Manager (ubuntu package of NetworkManager) is not the solution...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-30-2021 02:12 AM
09-30-2021 02:12 AM
Re: HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter throughput only 40 Gbps.
Hi,
Doesnt looks like any Hardware Issues.
May be we can still try Firmware and Drivers - just a general suggestion.
Thank you
RamKS
I work for HPE.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]