https://community.hpe.com/t5/proliant-servers-ml-dl-sl/hpe-tuning-for-high-bandwidth-with-mellanox-connectx-6/m-p/7204205#M184850 <P dir="auto" style="margin: 0;">No such user guide from HPE..</P> Wed, 10 Jan 2024 08:56:05 GMT Suvamay 2024-01-10T08:56:05Z https://community.hpe.com/t5/proliant-servers-ml-dl-sl/hpe-tuning-for-high-bandwidth-with-mellanox-connectx-6/m-p/7203769#M184749 <P>Hello all. I am using a HPE ProLiant server with mellanox ConnectX-6 NICs. After following NVIDIA's performance tuning guide (<A href="https://enterprise-support.nvidia.com/s/article/performance-tuning-for-mellanox-adapters" target="_blank">https://enterprise-support.nvidia.com/s/article/performance-tuning-for-mellanox-adapters</A>), I used the perftest benchmark and concluded that I can extract a maximum of aprox. 55 Gbps from each optical fiber cable, where 100 Gbps is expected.<BR />I have a high-bandwidth application which needs a theoretical maximum of 49 Gbps and it fails on most critical use cases. I also verified with ethtool -S that before failing it sends lots of pause frames for the connection peer.<BR />After digging for some time i found this HPE tuning guide for low-latency applications (<A href="https://h50146.www5.hpe.com/products/software/oe/linux/mainstream/support/whitepaper/pdfs/emr_na-c01804533.pdf" target="_blank">https://h50146.www5.hpe.com/products/software/oe/linux/mainstream/support/whitepaper/pdfs/emr_na-c01804533.pdf</A>) but found nothing about tuning for high-bandwidth. Is there any similar tuning guide for high-bandwidth?<BR /><BR />Thanks for any help.</P> Fri, 12 Jan 2024 06:46:11 GMT https://community.hpe.com/t5/proliant-servers-ml-dl-sl/hpe-tuning-for-high-bandwidth-with-mellanox-connectx-6/m-p/7203769#M184749 MarcoMon 2024-01-12T06:46:11Z https://community.hpe.com/t5/proliant-servers-ml-dl-sl/hpe-tuning-for-high-bandwidth-with-mellanox-connectx-6/m-p/7204205#M184850 <P dir="auto" style="margin: 0;">No such user guide from HPE..</P> Wed, 10 Jan 2024 08:56:05 GMT https://community.hpe.com/t5/proliant-servers-ml-dl-sl/hpe-tuning-for-high-bandwidth-with-mellanox-connectx-6/m-p/7204205#M184850 Suvamay 2024-01-10T08:56:05Z https://community.hpe.com/t5/proliant-servers-ml-dl-sl/hpe-tuning-for-high-bandwidth-with-mellanox-connectx-6/m-p/7206919#M185502 <P>In my experience, there are many possibilities for it not to work.<BR />Are you using PCIe x16? PCIe x8 will limit maximum bandwidth.<BR />Have you configured MTU 9000 across all network equipment?<BR />Do you have a CPU bottleneck?<BR />Did the other end of the connection also have performance tuning?<BR />This article is a little old, but you will find valuable tips to reach near the 100Gbps.<BR /><A href="https://fasterdata.es.net/network-tuning/" target="_blank">https://fasterdata.es.net/network-tuning/</A></P> Mon, 19 Feb 2024 22:06:22 GMT https://community.hpe.com/t5/proliant-servers-ml-dl-sl/hpe-tuning-for-high-bandwidth-with-mellanox-connectx-6/m-p/7206919#M185502 rogermoti 2024-02-19T22:06:22Z