HPE tuning for high-bandwidth with Mellanox ConnectX-6

[MarcoMon]

Hello all. I am using a HPE ProLiant server with mellanox ConnectX-6 NICs. After following NVIDIA's performance tuning guide (https://enterprise-support.nvidia.com/s/article/performance-tuning-for-mellanox-adapters), I used the perftest benchmark and concluded that I can extract a maximum of aprox. 55 Gbps from each optical fiber cable, where 100 Gbps is expected.
I have a high-bandwidth application which needs a theoretical maximum of 49 Gbps and it fails on most critical use cases. I also verified with ethtool -S that before failing it sends lots of pause frames for the connection peer.
After digging for some time i found this HPE tuning guide for low-latency applications (https://h50146.www5.hpe.com/products/software/oe/linux/mainstream/support/whitepaper/pdfs/emr_na-c01804533.pdf) but found nothing about tuning for high-bandwidth. Is there any similar tuning guide for high-bandwidth?

Thanks for any help.

[Suvamay]

No such user guide from HPE..

[rogermoti]

In my experience, there are many possibilities for it not to work.
Are you using PCIe x16? PCIe x8 will limit maximum bandwidth.
Have you configured MTU 9000 across all network equipment?
Do you have a CPU bottleneck?
Did the other end of the connection also have performance tuning?
This article is a little old, but you will find valuable tips to reach near the 100Gbps.
https://fasterdata.es.net/network-tuning/