P440ar & battery: TRIM not supported w/RAID? Battery-backed write cache in HBA mode?

[MLHJ]

Hi,

We recently bought two DL380 Gen9's (768345-425) which include the HP Smart Array P440ar Controller and HP Smart Storage Battery. Yesterday I was adding SSD's to the machines and noticed in Linux that the controller does not support TRIM (in HW RAID mode, I'm using RAID1).

Three questions:
1) Am I correct that in RAID mode this controller does not support TRIM/DISCARD?

2) If I instead use HBA mode instead of RAID (I will then use software raid, Linux MD) can I use TRIM?

3) In HBA mode, does the battery-backed 2Gb write cache work? Or is this part completely bypassed in HBA mode?

Thanks in advance.

[parnassus]

Interesting questions!

I think I can give you a straight answer to question 2: if P440ar's HBA mode "offers" disks control directly to overlying Linux OS (with recent Kernels, since 3.8.x) the mdraid (at least for RAID 1) implementation does support TRIM/Discard (With TRIM implemented through "discard" option in Linux's fstab or via the "fstrim" userspace command manually executed or driven by a dedicated cron job).

This happens on normal workstations with SATA in AHCI mode...if P440ar's HBA feature is transparent enough it should work that way too.

Other two questions are really interesting...

[Torsten.]

Hopefully this helps:

https://www.hpe.com/h20195/v2/GetPDF.aspx/4AA4-8144ENW.pdf

[MLHJ]

@parnassus: Indeed, re question #2 that's what I'd expect as well. I guess I'll know soon enough since I will be doing a clean install. Still don't know about the other questions, though. Question #3 is rather hard to test I think.

@Torsten.: Yes, I've read this PDF as well. It has some interesting information about "Smart path". But.. that's also the only subject it covers. Since it was the only PDF that talked about SSD's w/RAID I was hoping it would contain more information as well. But it doesn't mention TRIM/DISCARD at all - something quite crucial for SSD's. It's unfortunately not a generic "SSDs with HP ...." article.. it doesn't answer any one of the three questions :)

[parnassus]

The document @Torsten. linked doesn't contain the HP SmartSSD Wear Gauge concept which, conversely, the HPE Solid State Drives (SSDs) datasheet does.

It's the only reference of the word "wearing"...related to SSDs.

The HP SmartSSD Wear Gauge, AFAIK, is a Smart Array feature that reports, through a summary, a brief overview of the following counters:

• Total Solid State Drives with Wearout Status
• Total Smart Array Solid State Drives
• Total Non Smart Array Solid State Drives
• Total Solid State SAS Drives
• Total Solid State NVMe Drives
• Total Solid State Drives

and, when running a report, the user can either view a graphic representation of the report with SSD usage and estimated lifetime information...or can generate a report without a graphical display, with the option of saving the report (Reference here).

The HP SmartSSD Wear Gauge is basically a SSD wearing monitoring and reporting tool embedded into Smart Array Controllers.

So the question about TRIM/Discard is still on the air...and it is quite interesting...I've read that (some) HP SSDs are rated for (an endurance of) up to 2.5 random writes of the entire SSD's capacity per day (using the acronym DPWP or Drive Writes Per Day)...continuously for 5 years [*]...so, maybe, they didn't implemented TRIM/Discard mechanism in Hardware...or did they (without advertising it so loudly[**])? It's impossible they ddin't.

[*] Reference: "Breakthrough performance for application workload demands" HP Solution Brief (June 2015) about HP 6G SATA Value Endurance (VE) SSDs <-- the word "wearing" or "trim" is absent...

[**] On the datasheet linked above please note the sentence:

"HPE firmware optimizes our qualified SSD performance, wear leveling, and over-provisioning."

...so it's likely they only don't call it explicitly "TRIM Support" (at OS's filesystem level) because they implemented it at SSD's Firmware level.

[MLHJ]

Strange. I've enabled HBA mode but TRIM/DISCARD isn't working under Linux (Ubuntu 16 LTS)

With "hdparm" I can see the two drives sda & sdb support TRIM (which is correct), but a subsequent "blkdiscard" or "fstrim" fails saying "Operation not supported".

This can't be true. It must be possible to get TRIM support, right?

@parnassus: that's useful to know too, thanks!

[parnassus]

You said you installed Ubuntu 16.04 LTS...what's about disk partitoning (and filesystems) used on your SSDs? did you followed the guided partitioning with the LVM option (and, maybe, mdraid on top of LVM) or did you selected guided partitioning with direct creation of partitions on disk without LVM (eventually with mdraid)?

What the lsblk -D command reports on your sda (or sdb)?

[MLHJ]

No software raid, no LVM, so no layers in-between that may lack TRIM support.

hdparm -I shows:
           *    Data Set Management TRIM supported (limit 8 blocks)

..which means the drive supports TRIM.

While cat /sys/block/sdb/queue/discard_granularity shows 0.

..which means I cannot use TRIM.

root@ubuntu:~# blkdiscard /dev/sdb
blkdiscard: /dev/sdb: BLKDISCARD ioctl failed: Operation not supported

I guess I'll have to physically take the RAID controller out of the loop by rerouting the x4 SATA cables to the onboard controller.

I'm really surprised. I would never have thought with direct access it would not support TRIM commands.

[parnassus]

Probably the device access provided by the Smart Array P440ar isn't so "direct" as you expect even if the HBA setting was involved.

The lsblk -D command should report something like (here, as example, the sda is a Samsung 830 SSD - so non HPE - partitioned using EXT4 filesystems on separated /, /boot and /home mountpoints plus the classic Swap partition):

lsblk -D
NAME   DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sda           0      512B       2G         0
├─sda1        0      512B       2G         0
├─sda2        0      512B       2G         0
├─sda3        0      512B       2G         0
└─sda4        0      512B       2G         0
sr0           0        0B       0B         0

Note how the DISC-GRAN (Discard Granularity) parameter is non Zero (512) as it is reported - for the same device  - through the discard_granularity setting:

cat /sys/block/sda/queue/discard_granularity
512

I'm not testing this on a HPE ProLiant...that's just a Linux Workstation I have.

[MLHJ]

Yes, exactly, it seems not as direct as claimed. These are Samsung 850's by the way (both on sda and sdb)

root@ubuntu:~# lsblk -D
NAME   DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sda           0        0B       0B         0
├─sda1        0        0B       0B         0
├─sda2        0        0B       0B         0
└─sda3        0        0B       0B         0
sdb           0        0B       0B         0
sr0           0        0B       0B         0

[parnassus · Accepted solution]

The mistery is that the claim is official, at least reading the HPE Smart Array Controllers User Guide (March, 2016) on page 19:

"HBA mode

HPE Smart Array controllers support HBA mode. In HBA mode, all physical drives are presented directly to the operating system and the hardware RAID engine is disabled."

so, the consequence of that sentence is that it should work...or, if it doesn't in your particular case...it probably means that that sentence is not accurate enough (since your SSDs and your OS both support TRIM).

I've the impression that that "HBA mode" role of a Smart Array Controller doesn't act exactly as a real SAS/SATA HBA Card (e.g. like the Avago 9300 family).

An interesting reading could be (it's clearly OT because other brands are invoved but it's interesting because it shed some light on HBA modes role of typical RAID controller cards when the user need to have the most transparent control between the physical disk and the OS, as example when the usage of OpenZFS is planned on a Server) this one...the RAID nature of a Controller Card can be problematic if you look for pure HBA.

[Jimmy Vance]

I'll try a P440ar later this afternoon.  I just tried using an H240 controller in HBA mode and the output of the above cat command shows 512, lsblk also shows 512

[MLHJ]

Thanks, much appreciated :)
Just to let you know TRIM is working after I took out the x4 SATA links from the RAID card and put them into the onboard. Which is a bit of a ($200) waste since the RAID card is unused now and I no longer have the 2GB write cache etc. But at least it works.

[Jimmy Vance]

Will report back with the P440ar results shortly, but wanted to state that the cache module is only enabled when the Smart Array is in RAID mode.

[Jimmy Vance]

With both the H240 and the P440ar in HBA mode, the drive reported TRIM support. Drive is an older HP branded Samsung 100GB enterprise SSD.  Testing was done using a DL360 Gen9.

 cat /sys/block/sda/queue/discard_granularity
512

lsblk -D
NAME   DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sda           0      512B       2G         1
├─sda1        0      512B       2G         1
├─sda2        0      512B       2G         1
├─sda3        0      512B       2G         1
├─sda4        0      512B       2G         1
└─sda5        0      512B       2G         1

 fstrim -v /
/: 48.6 GiB (52176195584 bytes) trimmed
 fstrim -v /home
/home: 34.7 GiB (37199839232 bytes) trimmed
 fstrim -v /boot
/boot: 372.5 MiB (390631424 bytes) trimmed

[parnassus]

So could it be "only" an issue really related to using a non HP branded SSD (the Samsung 850, so not HP Firmware) on the P440ar in HBA mode?

[Jimmy Vance]

I'm not sure. I don't think I have access to any non-branded drives to test with.

[MLHJ]

@Jimmy Vance: Thanks for cleaing up that the cache module only works in RAID mode! Then it doesn't matter whether I use the onboard SATA or use the P440ar (in HBA mode).

So it works with HP drives in HBA mode, that's odd. Another difference could be the Linux kernel version perhaps? Then again, I'm running 4.4.0 (default on Ubuntu 16 LTS) which isn't that old. What else? P440ar firmware version? The server is only a few weeks old, so.. I wouldn't expect any old firmware on it.

[parnassus]

IMHO it shouldn't be a Linux 4.4 Kernel's fault (TRIM is supported [*] since about Linux Kernel 3.8, so many years ago...and Server class OS distributions generally backport newer Kernel features to their Kernel lines so watching the Kernel version itself is not so relevant as a parameter to judge the inclusion/exclusion of a feature)...provided that you haven't tested TRIM against a filesystem that doesn't natively support it (EXT3, as example), thing I believe you didn't since you know what you're doing.

Can you test with another Linux distribution: as example, CentOS 7 or Fedora Server? ...just to take the OS layer out of this equation.

[*] potentially OT but could be interesting to read this (which seems related to all Samsung 8xx SSDs).