Device is ineligible for IOMMU domain attach due to platform RMRR requirement

[pharidos]

Hi Folks

Is PCI passthrough supported on DL380 Gen 9 server? I have a 2 port 10 gig X540-AT2 and a 2 port 10G 82599ES  linecard and I am not able to pass through these devices to a quest

I get the error message "Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor." in /var/log /messages on trying to passthrough any of the above mentioned 10G port

Regards

Prem

Logs

====

pharidos@uks9:~$ lspci -vvv -s 05:00.0 | grep driver
Kernel driver in use: pci-stub
pharidos@uks9:~$

pharidos@uks9:~$ sudo qemu-system-x86_64 -enable-kvm -nographic -smp 2 -m 4G -cdrom /home/pharidos/ubuntu15.04.serial.iso -device pci-assign,host=0000:05:00.0,id=eth0
qemu-system-x86_64: -device pci-assign,host=0000:05:00.0,id=eth0: Failed to assign device "eth0": Operation not permitted
qemu-system-x86_64: -device pci-assign,host=0000:05:00.0,id=eth0: Device 'kvm-pci-assign' could not be initialized
pharidos@uks9:~$
pharidos@uks9:~$ sudo tail -n1 /var/log/messages
Jun 2 14:41:46 uks9 kernel: [533287.588971] pci-stub 0000:05:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.

[pharidos]

More info: Same issue is seen on trying to passthrough the devices using "vfio-pci" option instead of "pci-assign"

pharidos@uks9:~$ lspci -vvv -s 05:00.0 | grep driver
Kernel driver in use: vfio-pci

pharidos@uks9:~$
pharidos@uks9:~$ sudo qemu-system-x86_64 -enable-kvm -nographic -smp 2 -m 4G -cdrom /home/pharidos/ubuntu15.04.serial.iso -device vfio-pci,host=0000:05:00
.0,id=eth0
qemu-system-x86_64: -device vfio-pci,host=0000:05:00.0,id=eth0: vfio: failed to set iommu for container: Operation not permitted
qemu-system-x86_64: -device vfio-pci,host=0000:05:00.0,id=eth0: vfio: failed to setup container for group 20
qemu-system-x86_64: -device vfio-pci,host=0000:05:00.0,id=eth0: vfio: failed to get group 20
qemu-system-x86_64: -device vfio-pci,host=0000:05:00.0,id=eth0: Device initialization failed
qemu-system-x86_64: -device vfio-pci,host=0000:05:00.0,id=eth0: Device 'vfio-pci' could not be initialized
pharidos@uks9:~$ 

pharidos@uks9:~$ sudo tail -n1 /var/log/messages
Jun 2 14:55:34 uks9 kernel: [534115.142496] vfio-pci 0000:05:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.
pharidos@uks9:~$ 

[Suman_1978]

Hi,

I guess this issue could be with the Adapters or OS/Kernel you are using rather than a server.

Check this HP advisory for example.

[Moderator edit: Removed the broken link]

Check this Bug report from RedHat.  Check comment 10.

You can refer to this document on VFIO.

http://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html

Thank You!
I am a HP employee
__________________________________________
Was the post useful? Click on the white KUDOS! Thumb below to say Thank You!

Useful Links for ProLiant Servers

Self Solve Options for ProLiant Servers

[aw___]

As the error message indicates, the problem is with the platform, ie. the server and the firmware running on it.  Current Linux kernels no longer allow assignment of devices associated with VT-d Reserved Memory Reporting Regions (RMRRs).  These regions are used by some platform vendors for, among other things, side-band communication between I/O devices and the management controller.  Unfortunately the existence of these regions is fundamentally incompatible with PCI device assignment.  PCI device assignment requires full control of the device's I/O virtual address space, yet the RMRR imposes a platform requirement to reserve and identity map a vendor defined range of memory.  Honoring the RMRR would require the reserved region be imposed on the VM and prevent hot-plug, ignoring the RMRR risks that the device continues to do DMA to the RMRR defined memory, putting the integrity of the VM at risk.

It's my understand that HP has BIOS utilities that can disable these RMRRs on a per-slot basis.  This issue is not related to PCIe ACS isolation issues of the specific adapter or the resulting IOMMU groups.

[aw___]

Here's a Red Hat technical  paper for anyone that may be seeing this issue on Red Hat kernels or anywhere else: https://access.redhat.com/articles/1434873

[pharidos]

This is definitely a HP issue. The same adapter I am using (INtel 82599ES and Intel X540-AT2) works fine on other servers with the same OS.

Please have a look at what Redhat has reported under https://access.redhat.com/articles/1434873

[moarph]

Hello,

I have the same problem. I applied the HP advisory.

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c04781229&sp4ts.oid=5249566

but no luck.

Am I correct: kernelflag has to be iommu=on andf nothing more?

Slot number in this advisory is the same slot number as shwon in the iLO device infroation tab?

If above questions are to be answered with yes, I have no idea how to get pci passthrough working with HP servers

[Moderator edit: The above link is no longer valid. Please refer to support.hpe.com]

[Jimmy Vance]

---

@pharidos wrote:

This is definitely a HP issue. The same adapter I am using (INtel 82599ES and Intel X540-AT2) works fine on other servers with the same OS.

Please have a look at what Redhat has reported under https://access.redhat.com/articles/1434873

---

This advisory explains how to configure the server for HPE and 3rd party adpaters

Advisory: Linux - KVM Virtual Machine Does Not Start and Displays "Device is ineligible for IOMMU domain attach due to platform RMRR requirement" Message on HP ProLiant Systems with an Intel Processor
http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c04781229&sp4ts.oid=5249566

[Moderator edit: Removed the broken link. Please refer to https://support.hpe.com/]

[Jimmy Vance]

---

@moarph wrote:

Hello,

I have the same problem. I applied the HP advisory.

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c04781229&sp4ts.oid=5249566

but no luck.

Am I correct: kernelflag has to be iommu=on andf nothing more?

Slot number in this advisory is the same slot number as shwon in the iLO device infroation tab?

If above questions are to be answered with yes, I have no idea how to get pci passthrough working with HP servers

---

More information please

Whate model server server and NIC are you working with and what section of th advisory did you apply? 

Have you opened a support case with HPE?

[homer1990]

I know this thread is really old, but the advisory is not visible anymore. Can I find this somewhere else?