NVMe SSD performance is low

[Straeder]

Hi,

I am trying to get NVMe SSD's working in a HPE DL380 gen9 via the riser card. I have tried multiple controllers from multiple third parties. All with the same result: bad performance

Additional information:

- With a single controller card with 2 NVMe SSD's, only 1 SSD is detected. Even with bifurcation set to 8x. 

- With two controller cards with 1 NVMe SSD each, 2 SSD's are detected. Bifurcation 16x or 8x does not influence this, as is expected. 

- The NVMe's are Samsung 970 EVO Plus 2TB. We used multiple brands of controller cards. For instance the Asus Hyper M.2 v2.

- The OS is Ubuntu 22.04 LTS running on 2 x Intel SSD's on a SmartArray in RAID1.

- The DL380 gen9 is updated to the most recent firmware, iLO and other available updates. The server has 256GB RAM and 2 x E5 2687w v4 CPU's.

The issue:

The read/write throughput is limited to around 750MB/s.  I tried all kinds of BIOS settings and OS configurations.

This is the test we ran:
dd if=/mnt/storage/testfile of=/dev/null bs=1G iflag=direct

Result:

1+0 records in, 1+0 records out. 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.796324 s, 1.3 GB/s

Expected result:

A Samsung 970 EVO Plus on a PCIe gen3 (16x or 8X) should reach 3GB/s read speed. 

We tried many scenarios with NVMe SSD's and controller cards, all with similar results. We can rule out the NVMe's and the controller cards. So we expect the bottleneck to be in the server. 

Question:

Does anyone have similar experiences and does anyone have a solution or at least an explanation?

Kind regards,
Maarten

[rabindra11sharm]

Dear  Straeder

As my opinion, Please use only HPE recomended/ supported hardware. Thou, Please share your both NVME M.2 disk Part no and PCI controller cards part to check the configurations and supports. Other vendor spare parts are not qualified for HPE servers. If samsung or any other vendors parts are used in any server, noth only HPE qualified parts are used. non qualified spares might be creating multiple types/ uncommon issues.

Please go through the document as Quick Spec for HPE DL 380 gen 9 to understand supported hardwares and part details.

Thanks & Regards.

Rabindra

[Straeder]

@rabindra11sharm 

Thank you Rabindra for your reply. The Quick Spec for HPE DL 380 gen 9 only mentions NVMe's in the front slots. I understand that you urge us to use only HPE recommended or supported hardware.

But there must be alternative hardware not know to HPE that should work perfectly. The Asus Hyper M.2 v2 for instance is more recent than the HPE DL380 gen9. If you take a look at the specifications of the controller card, they fit well between specification boundaries of the server. 

My question is about the performance of the PCIe slots. They seem to be limited to 1GB/s. At least, that is the maximum throughput we achieved. 

To the community, is there a limit of 1GB/s on the PCIe slots of the DL380 gen9? Has anyone had similar experiences or even a solution?

[rabindra11sharm]

Dear Straeder

You are right, that many hardwares which are not certified by HPE is working at HPE servers perfectly. but I am telling you to use recommended part cause qualified/certified hardwares are tested, firmware and drivers also available for it.  Thou, I have found some documents, might be it will help you. Please check this.

Also Please share PCIe cards details and installed hardware details with part no.

HP PCIe Workload Accelerator :- https://www.hpe.com/psnow/doc/c03756976

[Straeder]

Hi Rabindra,

Off course the answer you have is is correct. But in my opinion, it is outdated. 

We use 'previously loved hardware'. That means that we use perfectly fine robust and reliable enterprise grade hardware by HPE. Some might call it refurbished. We give it a second life. 

Since the introduction of the gen9 series, a lot has happenend on innovation. TB's of super fast storage is becoming affordable. Making it even more interesting to extend the life-cycle of the gen9, or gen10 with a few years. Where other may see these perfectly fine machines as e-waste, we can use them for another 5 years. 

What we experience now with the gen9's is the HPE, by design, seems to prevent the extended life-cycle of the hardware. You call it qualified/certified. From a quality point of view, I understand. HPE must guarantee that some hardware can be used off-the-shelf without any issues. But this qualified/certified hardware is outdated, overpriced and useless in terms of what years of innovation has produced. 

We have tested a lot. We have benchmarked a lot. We use a lot of different hardware and disks in our experiments. All with the same outcome: the DL380 gen9 deliberately limits hardware performance. 

Nonetheless, we are still big fans of the gen9 series. The hardware is magnificent and outperforms a lot of current servers on the market. We will continue to use 'previously loved hardware'.

Thanks for your insight. Signing off.

[rabindra11sharm]

Dear  Straeder

You are right, HPE gen 9 servers are too robust and reliable to other gens server. I also loved it, but fact is now the hardwares of this gen are realy costly for its irragular availablity in market. So its time to move on current genaration servers with very high performance scale. 

any ways Can you pleased to let me know, on which PCIe slot, you want to bifurcate?? 

[Straeder]

@rabindra11sharm Moving on the a new generation while the gen9 is still good, has a high impact on the environment and carbon footprint. That does not make sense. For our purposes the gen9 is perfect. We do not need higher performance. Except on the PCIe slots.

We tried all three slots on the primary riser. With bifurcation on and off. The benchmarks are so consistent that we expect a intended blockage of throughput. The performance of the slots seems intentionally put at a low speed if non-certified hardware is used.

Yesterday we received another DL380 gen9. Executed the same tests with a multitude of controller cards and disks. Same outcome...

[rabindra11sharm]

Dear  Straeder

I have arrange a document for you, Please go through it once, it may help you to understand the issue, or finding any resolution. have a look on page no 5.

[Straeder]

Hi All,

Our assumption that the performance of the PCIe x16 slots in the HP DL380 gen9 is capped by firmware is correct. The performance is capped at 1GB/s when BIOS settings are default. 

When the server is set to 'Static High Performance Mode', the full capacity of the PCIe slots is enabled. We now have 2 Samsung 970 EVO Plus NVMe disks on an Asus Hyper V2 M.2 controller running 7GB/s Read-Speed in RAID0.

Hope this threads is helpful. 

Kind Regards,

Straeder