Disabling RMRDS / RMRR / HP Shared Memory features on Microserver Gen8

[kiler129]

I relize this is not a new problem, but it's unsolvable without at least a minimal HPE help. I know HPE may not want to officially want to engage in support for MS G8 as we have G10 & G10+ but seeing as iLO4 updates are still realsed I'm writing here in a hope that someone with insider's knowledge can help.

Cutting to the chase:
- PCI passthru is impossible on Linux-based systems for any device while working perfectly in ESXi even for built-in Smart Array controller
- Linux throws "DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor." as iLO reports all device's memory as protected
- HPE published advisory recommending connrep disabling RMRR for physical PCI slots but unfortunately while Microserver Gen8 is affected, the method doesn't work for it, and it only applies to 3rd-party devices in a slot and not to e.g. built-in ethernets or P120i

Workarounds:
- Community tried hacking Linux kernel removing RMRDS / RMRR check which appears to work, but only HPE knows what implications this may have (i.e. we cannot be sure if iLO is just reserving these regions for monitoring tools "just-in-case" or actively doing something with them)
- The advisory specifies "fix" for physical PCI slots, but for example P120i doesn't have such designation. I tried putting 0xFF in NVRAM 0x26A and 0x26B in a hope that engineers maybe used different undocummented masks  for internal PCIe but it obviously doesn't work.
- Final workaround: using ESXi. As it's propriatary we don't know what is it doing that makes PCI passthru work (maybe it's just YOLOing the DMA and ignoring RMRDS / RMRR all together?)

Help needed:
While I relize this is a community forum I know people from HPE and isnide contacts are reading and responding. I hope that HPE can just provide an access/information about a flag which can disable "HP Shared Memory" / RMRDS / RMRR for internal devices as it's definitelly safe to do so if ESXi is doing this.

[Moderator edit: Removed the broken link. Please refer to https://support.hpe.com/]

[Chandra_S]

Hello,

Good day,

Pelase elevate case with us , need more analysis so pelase captuure latest logs and elevate to HPE..

Regards

[kiler129]

Hello Chandra,

It's great to see an HPE engineer here!

Unfortunaely I'm unable to open the case as this server is not currently being used in an enterprise, and thus I don't have a support plan attached to it. However I think, giving the fact how many of these (and other affected ones mentioned in the support document) are still in good use outisde of the US (where a rotation of hardware is much slower), HPE could release the fix for everybody. From the scope of the change needed I hope this fix requires a very small engineering effort.

I actually create a Linux kernel patch to go around that problem and went on a deep dive to describe the technical details of the issue: https://github.com/kiler129/relax-intel-rmrr

Now I can say with full certanity: HPE needs to just exclude these memory addresses from RMRR in the bios. Literally just a one flag in the NVRAM "don't include any PCI/PCIe devices in RMRDS". This will fix issues with literally all Linux distributions (especially these starting to use Linux >=5.4 as it patched a lot of workarounds) and allow for PCI/PCIe passthrough (including e.g. P120i soldered on the motherboard).

Regards,
Greg

[erlon]

Hi folks,

For those coming across this same error, I worked it around by disabling the 'HP Shared Memory Features' on the device controller settings. After that, I was able to attach DPDK devices to my OVS bridge.

Bios prints here:

https://ibb.co/xhYHg2n
https://ibb.co/gSGmHh8
https://ibb.co/1vTHnG2
https://ibb.co/YW1HTdG

Erlon

[HPfan1]

Your screenshots do not look like HP Microserver Gen 8.

Can anyone confirm this working for HP Microserver Gen 8. I tested it and it did not work.

[flowmotion]

Hello , i found the option for Shared Memory communication in Bios

in Bios main Screen hit "CTRL+A" the menuelement "Service options" get visible. There you find it