topic Re: SSD drives insanely slow on Proliant DL380 Gen 9 in ProLiant Servers (ML,DL,SL)

SSD drives insanely slow on Proliant DL380 Gen 9

ThomasCameron — Mon, 28 Aug 2023 04:14:51 GMT

Howdy! I have three used, surplus (no warranty remaining) servers in my home lab. I use them for education/learning. They have two 2.5" ports in the back, and 12 LFF ports in the front. I figured that using cheap SSDs would allow them to boot quickly and give them a little more life.

Unfortunately, what I've found is that the SSDs are RIDICULOUSLY slow. I installed Red Hat Enterprise Linux on them, and did a couple of performance tests. I copied 16GB to a file using:

[root@neuromancer images]# dd if=/dev/zero of=bigfile bs=16M count=1024 oflag=direct status=progress
17179869184 bytes (17 GB, 16 GiB) copied, 676 s, 25.4 MB/s
1024+0 records in
1024+0 records out
17179869184 bytes (17 GB, 16 GiB) copied, 676.334 s, 25.4 MB/s

It averaged 25.4 MB/s. The advertised speed of the SSD is 540 MB/s. I did a large file (16GB) so that I could make sure it wasn't just writing to the 4GB cache on the Smart array controller. I chose oflag=direct so that it would use direct I/O to test the actual throughput of the drive instead of just writing to operating system cache memory.

Anyone know why it's so horrifically slow? I've got 6 of these SSDs in three servers. The results are the same across all 6 of them. I genuinely think that I should slap some plain old 2.5" non-SSD laptop drives in this thing and see if they're faster!

Query: SSD drives insanely slow on Proliant DL380 Gen 9

support_s — Sat, 26 Aug 2023 00:06:30 GMT

System recommended content:

1. HPE ProLiant DL380 Gen9 Server - Support Guides

2. HPE ProLiant DL380 Gen9 Server - System Manageability

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

Re: Query: SSD drives insanely slow on Proliant DL380 Gen 9

ThomasCameron — Sat, 26 Aug 2023 19:40:22 GMT

@support_s - is this supposed to be an answer? Or helpful in any way? I'm already familiar with the documentation. I didn't ask for a link to documentation. I asked why the SSD drives are so slow.

Literally every time I post a question on here, you post a link to the docs like that's supposed to be some sort of magic that makes everything better. It doesn't help at all.

Re: SSD drives insanely slow on Proliant DL380 Gen 9

Suman_1978 — Sun, 27 Aug 2023 15:21:44 GMT

Hi,

Speed may depend on various factors like:
SSD compatibility wth the Smart Array controller and the server.
SSD Firmware and Smart Array firmware.

Here is the Server Quickspecs.
Please check if the SSD is listed as compatible with this server model.

Thank You!
I work with HPE but opinions expressed here are mine.
HPE Tech Tips videos on How To and Troubleshooting topics

Re: SSD drives insanely slow on Proliant DL380 Gen 9

NathanAngus — Wed, 27 Dec 2023 02:00:27 GMT

Hi,

Just wondering if you were able to find a solution to this issue? I see to be having the same issue with consumer SATA SSD's.

Re: SSD drives insanely slow on Proliant DL380 Gen 9

ThomasCameron — Wed, 27 Dec 2023 15:05:51 GMT

Nope. No answer. In my case, it's not a huge deal because it's "just" the boot drive, and the array of spinning disks is the one I really care about. It's frustrating, but doesn't really affect the workload.

I will say, I won't be buying Proliants in the future. Firmware updates behind a paywall, no support without paying more per year than I paid for the used servers, and their staff posts links to the documentation as the "answer" for darned near every question, which is not helpful at all. I miss the days when Compaq ran the show. HPE is terrible. I used to swear by Proliants. Now I mostly just swear AT them.

Re: SSD drives insanely slow on Proliant DL380 Gen 9

Pierpa — Thu, 30 May 2024 16:01:18 GMT

I'm having a similar issue on DL380 GEN9 with 2 4TB samsung evo 870 QVO dicrectly connected to motherboard without passing form smart array (i want to try CEPHfs on them).

I found this discussion in proxmox forum:

https://forum.proxmox.com/threads/very-poor-disk-speed-with-hp-dl380-gen9-40mb-s-write-speed.119635/

where they talk about: " disk writecache is disabled by default in hp bios" if ssd (like mine) has no PLP (Power Loss Protection), but they attach ssd disks to smart array not may case (Ceph fs doest not love hardware raid).

Still trying to find a solution.....

Any ideas?

Thanks in advance,

Pier Paolo

Re: SSD drives insanely slow on Proliant DL380 Gen 9

ThomasCameron — Thu, 30 May 2024 16:13:43 GMT

Unfortunately not. I've just learned to live with it. In my case, it's not a huge deal since it's just the OS boot drive. But it's frustrating knowing that the drive is rated for MUCH faster throughput than it's delivering.

I will try to enable write caching and see if that gives me better performance. I'll report back if so.

Re: SSD drives insanely slow on Proliant DL380 Gen 9

ThomasCameron — Fri, 31 May 2024 14:32:50 GMT

Sadly, enabling write cache did not really help. It's faster, but still less than 10% of the supported speed:

[root@neuromancer ~]# cd /var/tmp/
[root@neuromancer tmp]# dd if=/dev/zero of=testfile bs=16M count=1024 oflag=direct status=progress
17179869184 bytes (17 GB, 16 GiB) copied, 383 s, 44.9 MB/s
1024+0 records in
1024+0 records out
17179869184 bytes (17 GB, 16 GiB) copied, 382.549 s, 44.9 MB/s

Re: SSD drives insanely slow on Proliant DL380 Gen 9

ThomasCameron — Fri, 31 May 2024 15:07:32 GMT

Interestingly, I tried it on a second server, and it DID make it faster:

[root@molly ~]# cd /var/tmp/
[root@molly tmp]# dd if=/dev/zero of=bigfile bs=16M count=1024 status=progress oflag=direct
16861102080 bytes (17 GB, 16 GiB) copied, 40 s, 421 MB/s
1024+0 records in
1024+0 records out
17179869184 bytes (17 GB, 16 GiB) copied, 40.785 s, 421 MB/s
[root@molly tmp]#

It may be the drive version on neuromancer. It could just be a crappy drive.

Re: SSD drives insanely slow on Proliant DL380 Gen 9

ThomasCameron — Mon, 16 Sep 2024 09:25:35 GMT

Just tested it on a third server. It's much faster. I think that the server called neuromancer must just have a crappy drive or something. On molly and armitage, enabling write caching DID improve performance significantly. On neuromancer, it did not:

[root@armitage ~]# cd /var/tmp/
[root@armitage tmp]# dd if=/dev/zero of=bigfile bs=16M count=1024 status=progress oflag=direct
16995319808 bytes (17 GB, 16 GiB) copied, 36 s, 472 MB/s
1024+0 records in
1024+0 records out
17179869184 bytes (17 GB, 16 GiB) copied, 36.4001 s, 472 MB/s
[root@armitage tmp]#

Re: SSD drives insanely slow on Proliant DL380 Gen 9

Pierpa — Sun, 02 Jun 2024 13:02:23 GMT

Thanks @ThomasCameron for your report. Just some question if you can help me:

Did you connect your sata through a smart array RAID controller or directly to sata connector on motherboard (like i did)?
Where you find settings for enabling write caching? I can´t find it on bios, is something related to smart array?

Thanks in advance,

Pier Paolo

Re: SSD drives insanely slow on Proliant DL380 Gen 9

ThomasCameron — Mon, 16 Sep 2024 09:34:45 GMT

I'm connected to the P840 RAID controller.
The array controller menu shows up when I hit F9 to go into system settings. It opens a second GUI where I can change the smart array contoller settings.