MicroServer Gen10 Plus v2 with M2 SSD: NMI_HARDWARE_FAILURE and no solution yet

[WiFriendly]

Hi HPE!

I am in trouble and need support from you. It looks like I am out of ideas, as are other third-party internet board members.

This is the original thread on Reddit. And this is the second thread on the STH board. 

Long story short, I got two of the HP MS Gen10 V2 Plus and can't run M2 SSDs smoothly by using the Lenovo PCIe to M2 riser card, P/N 4XH0L08579. The same hell as described at ServeTheHome: NMI_HARDWARE_FAILURE at the OS reboot (and sometimes at other stages too).

According to the advice on the STH board that helped other MS Gen10+ owners, I already went from the Samsung 970 series MZ-V7S1T0B/AM to the Intel 670p series SSDPEKNU010TZX1 and... got the same error again. So, changing the SSD to Intel didn't help for some reason. Another piece of advice that helped other users was slowing down the PCIe bus or/and PCIe slot to Gen2.0. It doesn't help me either!

HP SPP 2024.04 was installed in full, but there were no changes. The Windows Update was not helpful either.

It looks like the issue is related to the PCIe and NVMe, because everything is fine when the SSD is removed from the raiser card. The card as a whole doesn't cause any issues (because, from what I know, it's straight-through and, roughly speaking, extends the PCIe lines to the M2 SSD). It should also be said that the MS works just fine when the OS is installed on a SATA device or VROC RAID.

It looks like no one at HPE has tested the M2 SSD with these devices via adapter cards, so they haven't fixed this problem in a few years, which is mentioned on many Internet boards.

Is there any hope that this problem will be addressed and an updated version of the BIOS will be released that will solve these problems?

[TQ12]

Hello

This Lenovo card is a dumb NVMe PCIe adapter card, which can only be used to connect NVMe drives if the BIOS of the computer supports bifurcation, as far as I can rememeber.
However, HPE has removed this option from the microserver BIOS and it is no longer available.
There are also one corresponding topic here in the forum.
Here:
https://community.hpe.com/t5/proliant-servers-netservers/microserver-gen10-plus-pcie-bifurcation-support-issue/td-p/7092453/page/4
But there is still a way to use NVMe disks in the microservers. You need a card similar to the one mentioned by Lenovo, but it must be equipped with an PCI interface/switch like the Asmedia ASM2824.
Here is a example that should work:
https://www.delock.de/produkt/90305/merkmale.html?f=s
or
https://a.co/d/jd8gdW1
You also can boot from both of them.
The NMVe vendor does not matter.
Maybe useful:
https://www.servethehome.com/hpe-proliant-microserver-gen10-plus-ultimate-customization-guide/3/

Cheers

[WiFriendly]

@TQ12Thanks for the response. I have read about wiped-out bifurcation in HP MS Gen10 (although it is still presented as a setting in the latest BIOS version, which was released quite recently!). But it all looks like I do not need it because I run only 1 M2 SSD with straight-through lines to the PCIe bus. If I connect only one device, I don't think that I need a bifurcation-capable BIOS.

[ManBha]

Hello,

We have not tested the compatibility with Lenovo PCIe to M2 riser card. Request you to check the status with HPE riser card.

Thanks.

[WiFriendly]

HI @ManBha!

What card are you talking about? HP 759505-001? It looks like a 'dumb' card too, because I can't see any controller IO chip.

And based on the information we have from other HPE answers, HP MS Gen10 doesn't support bifurcation. As a result, 2 NVMe slots will not work even in theory.

[ManBha]

Hello,

Yes, we would recommend you to use HPE supported configuration and parts.

https://support.hpe.com/hpesc/public/docDisplay?docId=a50004293enw&docLocale=en_US

Thanks.

[WiFriendly]

There is no HPE-recomended M2 SSD riser board at all. It looks like they hate M2, and they love to watch their clients suffer.

P.S.
In the future, when you give advice, we could at least delve a little into the essence of the problem.

[ManBha]

Hello,

I understand there is no M2 SSD riser board and hence we recommend to use only tested or recommended configuration. 
We will share your feedback with our team.

Thanks.

[WiFriendly]

Stll NMI_HARDWARE_FAILURE BSODs all the time, even when the ASMedia-based NVME card is used (QNAP QM2-2P-384A).
SSD models don't matter, because I have tested tons of them from different vendors and different series.

Case opened with the HPE: 5382013385. A lot of the ASH was collected, as well as the Windows memory dumps, and posted to the case too.

Please investigate and give this issue the highest priority!

[Moderator edit: Erased the confidential info.]

[WiFriendly]

A warning to everyone.

Avoid the HPE Microserver. After 2 months of experiments, spending thousands of dollars, and uploading many logs with errors and proofs, an XPE technical support employee was unable to carefully read the case in 3 days and responded with random advice. Moreover, most of all, he is interested in my phone number and not in analyzing the logs.

Avoid this HPE Microserver! This is now a Chinese product with Chinese-level technical support.

[KluthR]

@ManBha 

To summarize:

• Microserver Gen10 Plus v2 does NOT support BiFurcation due to Intel CPU incapabilities?
• One is good to go, if only ONE NVMe is in use?

Is that correct?

[TQ12]

---

@KluthR  schrieb:

To summarize:

• Microserver Gen10 Plus v2 does NOT support BiFurcation due to Intel CPU incapabilities?
• One is good to go, if only ONE NVMe is in use?

Is that correct?

---

Basically yes.
Bifurcation was supported but later for unknown reasons removed from the bios by HPE.
Now it is officially unsupported.
One NVMe should work most of the time. It depends a little on the riser cards used. It's sometimes a little trial and error.

Cheers

[KluthR]

Well, the riser just connects the pins, no logic on it. With just one nvme I thought it should work. Will try.

https://www.amazon.de/gp/product/B07KWWDV88

[WiFriendly]

One NVMe should work most of the time. It depends a little on the riser cards used. It's sometimes a little trial and error.

No, you are wrong. Any cards, active, passive, $20 or $400, from China, Taiwan, the US, or Mars, do not work. Namely, they only work until the benchmark is launched for 5–15 minutes. Then the result is always the same, BSOD.

[WiFriendly]

Well, the riser just connects the pins, no logic on it. With just one nvme I thought it should work. Will try.

https://www.amazon.de/gp/product/B07KWWDV88

This card will give BSOD on Windows reboot if you use a real monitor (instead of an RDP session) and will hard reboot after running any disk test (for the NVME/PCIe one) for more than 8 minutes.

At the same time, HDD tests, RAM tests, CPU tests, or GPU tests can go on for as long as you like.

[KluthR]

However, my linked riser card is just doing fine. The one nvme is working even under load. So, for my use case perfect.

[WiFriendly]

What do you call load? Real server load, like DB transactions and file copying? You won't see any crashes, then. Run the ATTO benchmark or CrystalDiskMark for 15 minutes, selecting the NVME disk as the target. And you will see what I am talking about.

[WiFriendly]

Normal server operation, namely, synchronizing files over the Internet, also leads to spontaneous reboots and problems with the RAID5 array. It just takes 12-24 hours to PCIe errors appear, not 5-15 minutes like with the benchmark. The benchmark simply allows you to save a lot of time.

And the result is still the same. The server with the NVME disk is not yet viable.

[WiFriendly]

Hi there!

After six months and over $5,000 in hardware investment, I'm not giving up.

I was able to make the server work reliably with SSD. But for this, I had to refuse NVME altogether and roll back 10 years to M2 SSD. I used the Dell BOSS S1 controller, product number 7HYY4 (07HYY4), and Intel M.2 SSD D3-S4510 disks, P/N SSDSCKKB480G801, in RAID-1 mode. I got read and write speeds 7 times lower than when using NVME disks. But there are no spontaneous reboots and NMI_HARDWARE_FAILURE errors anymore.

The latest BIOS (2.20) and SPP are installed, BTW.

But this is, of course, not a solution, but a half-measure that throws us back to 2014.

Therefore, I still hope for a BIOS fix from HP, because there are simply no other options left.