M.2 not working with iLO6 module in HPE ProLiant MicroServer Gen11 Server

[BennyInc]

I want to setup my new ProLiant MicroServer Gen11, and installed the iLO6-module (P65741-B21) with an M.2 NVMe as described in the user guide.

The iLO module works fine, but the NVMe cannot be found. It won't be listed as a device during Debian installation, and the iLO device inventory shows the M.2 Slot 20 as "Empty Drive Bay 1" / "Populated: No". The dmesg output does not show any signs of an nvm being found, nor anything related to incorrect RAID settings. I also tested Ubuntu without any success.

The NVMe is a "Kingston FURY Renegade PCIe 4.0 NVMe M.2 SSD", Datasheet: https://www.kingston.com/datasheets/sfyr_en.pdf

I have tested the NVMe in a separate enclosure and it works as expected. I have also reseated the M.2 again to make sure the contacts are proper. I have also updated iLO and ROM to the latest version. Finally, I also tested the following 3 BIOS options individually, hoping they might be related to and resolve the issue:

• System Options -> Boot Time Optimizations -> UEFI POST Discovery Mode? Auto -> Force Full
• Storage Options -> NVM Express Options -> Embedded NVM ROM? Enabled -> Disabled
• PCIe Device Configuration -> Advanced PCIe -> NVME PCIe Resource Padding? Disabled -> Enabled
  

Any ideas what else I can check? Any other settings that might block my system from detecting the NVMe?

[shiva_jr]

Hi @BennyInc ,
Following docs may help you. 
1. System board components to verify the connectivity.
2. M2.SSD option 
3. Microserver quickspecs page no. 21

Regards,
Shiva_JR
Please mark as 'Accepted solution' and give 'Kudos' if my post worked.

[BennyInc]

Thanks for your update, @shiva_jr .

I have installed the iLO module as documented here: https://support.hpe.com/hpesc/public/docDisplay?docId=sd00003930en_us&page=GUID-AB8CF752-7EEC-4930-B133-AE65BE4C54BE.html
The iLO module is working correctly, I have been using it to remotely manage the server.

I then installed the M.2 as documented here: https://support.hpe.com/hpesc/public/docDisplay?docId=sd00003930en_us&page=GUID-14E4370F-61F4-4F0F-AD96-DF7C14CF9FB9.html
The M.2 is never recognized in the BIOS, unfortunately. I reseated it again after successfully testing it in an M.2 PCIe USB-C enclosure, and I'm sure connectivity is fine. A photo showing the setup:
Photo showing the M.2 installed together with the iLO module.

Your third link, the QuickSpecs, again reiterates that the iLO kit is required to install the NVMe drive, which I did. The drive itself is not the same one as linked in the QuickSpecs, but the Kingston one follows the same specs (M.2 2280, PCIe 4.0, TLC NAND), while having an even lower idle power consumption. So unless HPE locks out other vendors' M.2 explicitely, which is not documented anywhere, I would expect mine to work as well.

[Sham82]

Hello @  BennyInc

Thank you for your Post.

Since you have followed the user guide.
we would suggest to use HPE recommended NVME drives

HPE M.2 Drives
HPE 480GB NVMe Gen4 Mainstream Performance Read Intensive M.2 PM9A3 SSD P69543-B21
Notes: To support this low power NVMe M.2 drive, the HPE ML30 Gen11 iLO/NIC/M.2/COM Port Kit
(P65741-B21) must be installed.

Regards
HPE

[BennyInc]

When buying the server and iLO module, I was under the impression that not only HPE NVMe drives would be supported, as the various documents say it "supports NVMe SSDs in 2280 or 22110 form factor".
M.2 NVMe is a standard which, like other standards, works across hardware vendors. I also wouldn't expect having to use an HPE certified ethernet-cable just to attach my server to my non-HPE-switch.

I would expect HPE to be highly interested in properly supporting standards like M.2 NVMe, and investigate possible incompatibilities thoroughly.

[Sham82]

Hello,

We highly recommend using hardware specified in the official QuickSpecs for HPE servers, as these components have been tested and verified to ensure optimal performance and compatibility. Unfortunately, we cannot provide support or comments on non-HPE hardware, as it falls outside the scope of our testing.

Regards
HPE

[BennyInc]

I don't need support or comments on the NVMe drive, as I am confident it to be fine. I need support on the iLO 6 module, which claims to support M.2 NVMe drives but doesn't. If HPE is not confident that the M.2 NVMe standard is fulfilled, maybe the documents should read "supports HPE NVMe SSDs in 2280 or 22110 form factor" instead.

[BunnyPon]

While I can't afford a shiny toy like you, I do have an a similar ILO/M.2 card in the junk ha8000v, so I thought I'd go looking.

Documentation suggests that it should be a Gen3 or possible Gen4 M.2 NVMe and the HP prices are frightening. It totally would not surprise me if they made it require only HPE unobtainium drives to work. (That and that JP prices are usury.)

Do you have any other nvme drives you can test with? THe only thing I can think of is your nvme many not be backward compatible enough.

[BennyInc]

I don't have any other NVMe drives to test with, unfortunately.

HPE does not provide much in specifications for the ILO 6 module. All I found initially was, that it supports M.2 NVMe in 2280 or 22110 size. Once I got it, I could confirm from the pins that it is indeed Key M, so that's what I bought. I'm not sure what else could go wrong compatibility wise?

The HP NVMe also does not have too much spec available, but it is PCIe 4.0 (like mine) and TLC NAND (like mine). Power-wise the Kingston even consumes less power in idle, so I wouldn't expect any issues there.

Within iLO, the "M.2 Slot 20", which is listed as "Empty Drive Bay 1" shows a bus width of x1 in the details, whereas the other PCI-E Slots show x16 or x4 as expected. I wonder whether the iLO module seriously only supports 4x1 for the drives? I'm also not sure whether this would create any issues (like mine) when connecting a 4x4 NVMe? The spec pages for the P69543-B21 (one example drive) don't show anything about bus width, so I'm not sure whether the HPE NVMe drives are 4x4 or 4x1.

[Sham82]

Hello,

For further assistance we would request to please log a HPE support case.

Regards
HPE

[BennyInc · Accepted solution]

I did open a support case of course, but got no help there.

In the meantime I removed the iLO module in order to return it to the supplier and get a new one, suspecting a defect on the module. After removing the module, I decided to give it one last try, and reinstalled the iLO module. With that, the NVMe suddenly was recognized and is working fine now.

In Summary: Reseating the iLO module has resolved the issue.

Now I just feel a little cheated by the non-existing M.2 specs for the iLO module. My NVMe is a PCIe 4.0 device with a bus width of 4x. On the iLO module it only runs at PCIe 3.0 speeds however, with a bus width of 1x. With some more specs available I would have spent less money on a PCIe 3.0 NVMe instead. Maybe this information can be helpful for others however, so Ill just leave this here.

root@hulk:~# lspci
...
03:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. KC3000/FURY Renegade NVMe SSD E18 (rev 01)

root@hulk:~# lspci -vv -s 03:00.0
...
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
...
LnkSta: Speed 8GT/s (downgraded), Width x1 (downgraded)
...