NVMe SSD low write performance in hardware raid

[KarolisS]

Hello,

Hardware RAID from nvme ssd’s writes performance is lower comparing to single nvme ssd in JBOD mode.

Hardware RAID5 (4 disks) performance:
Random read + write: READ 1460MB/s 22.7k IOPS, WRITE 1462MB/s 22.6k IOPS
Hardware RAID6 (4 disks) performance:
Random read + write: READ 1695MB/s 25.9k IOPS, WRITE 1697MB/s 25.6k IOPS

If disks are not in raid but in JBOD mode, connected to the same RAID controller performance (single disk):
Random read + write: READ 3148MB/s 47.2k IOPS, WRITE 3149MB/s 46.9k IOPS

Why hardware RAID from nvme SSD’s has 2x lower write performance comparing to single disk in JBOD mode connected to the same hardware RAID controller? Even considering additional parity operations RAID5 and 6 needs to do, shouldn't write performance be at least the same as single disk in JBOD? 
All details about hardware specifications, RAID configuration and linux settings below.
Am I missing some configuration settings to get better performance?
Is it worth changing disks sector size from 512 to 4k?

Server:
ProLiant DL325 Gen10 Plus v2
AMD EPYC 7713P 2.0GHz 64-core
16x HPE 64GB (1x64GB) Dual Rank x4 DDR4-3200 RAM
4x HPE 7.68TB NVMe Gen4 High Performance Read Intensive SFF BC U.3 PM1733a SSD
RAID controller - HPE MR416i-a Gen10 Plus x16 Lanes 4GB Cache NVMe/SAS 12G Controller
All 4 disks are connected with 4 lanes to MR416i-a.

Server is running on Rocky 9.3 (kernel 5.14) with latest firmware versions.
IO scheduler set to none
current_clocksource set to tsc

Performance tests done with fio 3.35.

RAID5 from 4 disks.
Raid settigs: iopolicy=direct, wrcache=wt, rdcache=nora, Strip Size = 64 KB
Raid performance:
Sequential reads: 14.9GB/s, 228k IOPS
Sequential writes: 1781MB/s, 26.9k IOPS
Randon read: 14.9GB/s, 228k IOPS
Random write: 1459MB/s, 22.3k IOPS
Random read + write: READ 1460MB/s 22.7k IOPS, WRITE 1462MB/s 22.6k IOPS

If disks are not in raid (JBOD mode) performance is:
Sequential reads: 7362MB/s, 112k IOPS
Sequential writes: 4100MB/s, 62.8k IOPS
Randon read: 7363MB/s, 112k IOPS
Randon write: 4053MB/s, 62.5k IOPS
Random read + write: READ 3148MB/s 47.2k IOPS, WRITE 3149MB/s 46.9k IOPS

Disk info:
Sector size (logical/physical): 512 bytes / 512 bytes
Drive /c0/e252/s1 Device attributes :
===================================
Manufacturer Id = NVMe
Model = VO007680KYDNA
NAND Vendor = NA
Firmware Revision = HPK4
Raw size = 6.985 TB [0x37e3e92b0 Sectors]
Coerced size = 6.985 TB [0x37e2e0000 Sectors]
Non Coerced size = 6.985 TB [0x37e2e92b0 Sectors]
Device Speed = 16.0GT/s
Link Speed = 16.0GT/s
Write Cache = Disabled
Logical Sector Size = 512B
Physical Sector Size = 512B
Connector Name = Port 1I x4
Port Number = Port 1I x4
Box = 1
Max lane width = 4

Raid info:
---------------------------------------------------------------
DG/VD TYPE State Access Consist Cache Cac sCC Size Name
---------------------------------------------------------------
0/239 RAID5 Optl RW No NRWTD - ON 20.957 TiB
---------------------------------------------------------------

VD239 Properties :
================
Strip Size = 64 KB
Number of Blocks = 45005537280
VD has Emulated Drive = No
Span Depth = 1
Number of Drives Per Span = 4
Write Cache(initial setting) = WriteThrough
Disk Cache Policy = Disk's Default
Write Cache Status = Disabled
Encryption = None
Data Integrity = None
Active Operations = None
Exposed to OS = Yes
OS Drive Name = /dev/sde
Creation Date = 22-12-2023
Creation Time = 09:27:13 AM
Emulation type = default
Cachebypass size = Cachebypass-64k
Cachebypass Mode = Cachebypass Intelligent
Is LD Ready for OS Requests = Yes
SCSI NAA Id = 600062b2179cb5802d1812f1959fa55f
Unmap Enabled = No

DG=Arrays | Slot=Drive Bay No|VD=Virtual Drive/Logical Drive|EID=Enclosure Device ID

Fio test config:
ioengine=libaio
direct=1
iodepth=128
group_reporting
time_based
runtime=600
bs=64k
numjobs=16
random_generator=tausworthe64

[SydUmar_1]

Hello KarolisS,
Could you please confirm the MR416a-a controller firmware that is installed? Do we have the version 52.26.3-5250 installed? 
Latest firmware link - https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_57d8f895b87341ae83a035981f 

[SydUmar_1]

Hello KarolisS,

Request you to share an update regarding this query.